AI/ML Training Pipeline

Leopard Detection Model · Optimised GPU Training on AI Server
UTC+05:30 00:00:00
Training Ready · GPU Online
Training Data Flow · Live Epoch Simulation
DATA FLOW → DISK LOAD DECODE XFER GPU SAVE
💾
Dataset Disk
Leopard images
Labels / Annotations
~50,000 images
STORAGE
⚙️
DataLoader
batch_size = 32
num_workers = 4
pinned_memory = True
Parallel disk reads
CPU · RAM
🔄
Decode · Augment
JPEG → RGB Tensor
Resize · Normalize
Flip · Rotate · Crop
0.0 → 1.0
CPU · PREPROCESS
⚡
CPU → GPU Copy
Pinned Memory
non_blocking=True
Async PCIe transfer
CPU keeps loading
PCIE · ASYNC
🚀
GPU Training Core
▶ Forward Pass — predict
▶ Loss — measure error
▶ Backprop — gradients
▶ Optimizer — update
GPU · VRAM
📦
Checkpoint
model.pt saved
Log loss / accuracy
Epoch 0 / 50
SAVE · LOG
📷
Edge Device
NVIDIA Jetson
Runs trained model
Leopard detection
On-site · Low power
DEPLOYMENT
batch_size
32
Images processed by GPU at once. Larger = faster, but needs more VRAM. Out of Memory? Reduce this first.
num_workers
4
CPU threads reading images from disk in parallel. Too few = GPU waits idle. Think of workers as librarians fetching books simultaneously.
pinned_memory
True
Reserves fast CPU RAM for GPU transfers — like a dedicated highway lane instead of shared road. Makes PCIe transfer significantly faster.
non_blocking
True
CPU doesn't wait for GPU transfer to finish — immediately loads the next batch. GPU and CPU run simultaneously = maximum efficiency.
Forward Pass
→ ŷ
Model sees the image, runs through all neural network layers and makes a prediction: "Is this a leopard? Where is it?"
Loss Function
ℒ
Measures how wrong the prediction was. Lower loss = smarter model. Goal of training is to reduce this number every epoch.
Backpropagation
∇
Maths that traces the error backwards through every layer, calculating how much each weight contributed to the mistake (gradient).
Weight Update
w←
Optimizer (Adam/SGD) nudges each weight slightly to reduce loss. After thousands of iterations, the model learns to detect leopards accurately.
Training Run Summary
— loss
Current Epoch—
GPU Utilisation—
Batches / sec—
Est. time remaining—
Model accuracy—
GPU Resource Monitor
GPU Utilisation0%
VRAM Usage0%
CPU Dataloader0%
PCIe Bandwidth0%
Common Bottlenecks · Student Guide
BottleneckSymptom
CPU DataLoaderGPU idles, low utilisation
Too few workers<50% GPU usage
Heavy augmentationSlow epochs, CPU maxed
Large batch_sizeOut-of-Memory (OOM)
Many negativesSlow convergence
training.log · ai-server-01
[SYSTEM] AI Training Server online · NVIDIA GPU ready · CUDA initialised · Press "Run Training Epoch" to start.