Data engineer building large-scale ingestion, streaming and training-data pipelines — turning telemetry and imagery from a 100-node edge fleet into clean, queryable, model-ready data.
Projects
Leopard Detection System — Data & Streaming Pipeline
Built the ingestion, streaming and training-data pipelines behind a 100-node edge fleet.
- Telemetry ingestion at scale: streamed detections and device metrics from 100 edge nodes through a pub/sub topic — partitions, consumer offsets and lag monitoring.
- Training-data pipeline: built the ETL for 50k+ labelled images — cleaning, dedup, augmentation and train / val / test split — feeding model training.
- Pipeline optimization: removed data-loading bottlenecks (parallel workers, pinned memory) to cut training time and cost.
- Observability data: aggregated metrics and logs across the fleet into time-series stores and dashboards (Prometheus / Grafana) for fleet-wide analytics.
Analytics & Query
- SQL analytics over detection history — species counts, hotspots, trends by zone and time.
- Built heatmap & trend dashboards for rangers from the aggregated data.
- Packaged pipelines in Docker on Linux for reproducible, portable runs.
Advanced AI Ecosystem Training
by DeepManthan AI Production AI Ecosystem Program
Intensive program on building & operating production AI systems — containers, orchestration, distributed systems, networking and deep learning.
Containerisation — Docker, images, isolation, OTA rollouts
Orchestration — Kubernetes, self-healing, scaling
Distributed Systems — Raft consensus, consistent hashing
Messaging — pub/sub, topics, partitions, offsets
Concurrency — multi-threading, producer–consumer
Computer Vision — CNNs, convolution, detection
Networking — TCP/IP, SSH, private overlays
Linux & Ops — processes, daemons, systemd
Impact Highlights
100
nodes streamed via pub/sub
50k+
images in the training pipeline
Real-time
consumer-lag monitoring
↓ time
faster training after tuning