AI/ML Training Pipeline

Leopard Detection Model · Optimised GPU Training on AI Server
UTC+05:30 00:00:00
Training Ready · GPU Online
Training Data Flow · Live Epoch Simulation
DATA FLOW → DISK LOAD DECODE XFER GPU SAVE
💾
Dataset Disk
Leopard images
Labels / Annotations
~50,000 images
STORAGE
⚙️
DataLoader
batch_size = 32
num_workers = 4
pinned_memory = True
Parallel disk reads
CPU · RAM
🔄
Decode · Augment
JPEG → RGB Tensor
Resize · Normalize
Flip · Rotate · Crop
0.0 → 1.0
CPU · PREPROCESS
CPU → GPU Copy
Pinned Memory
non_blocking=True
Async PCIe transfer
CPU keeps loading
PCIE · ASYNC
🚀
GPU Training Core
▶ Forward Pass — predict
▶ Loss — measure error
▶ Backprop — gradients
▶ Optimizer — update
GPU · VRAM
📦
Checkpoint
model.pt saved
Log loss / accuracy
Epoch 0 / 50
SAVE · LOG
📷
Edge Device
NVIDIA Jetson
Runs trained model
Leopard detection
On-site · Low power
DEPLOYMENT
batch_size
32
Images processed by GPU at once. Larger = faster, but needs more VRAM. Out of Memory? Reduce this first.
num_workers
4
CPU threads reading images from disk in parallel. Too few = GPU waits idle. Think of workers as librarians fetching books simultaneously.
pinned_memory
True
Reserves fast CPU RAM for GPU transfers — like a dedicated highway lane instead of shared road. Makes PCIe transfer significantly faster.
non_blocking
True
CPU doesn't wait for GPU transfer to finish — immediately loads the next batch. GPU and CPU run simultaneously = maximum efficiency.
Forward Pass
→ ŷ
Model sees the image, runs through all neural network layers and makes a prediction: "Is this a leopard? Where is it?"
Loss Function
Measures how wrong the prediction was. Lower loss = smarter model. Goal of training is to reduce this number every epoch.
Backpropagation
Maths that traces the error backwards through every layer, calculating how much each weight contributed to the mistake (gradient).
Weight Update
w←
Optimizer (Adam/SGD) nudges each weight slightly to reduce loss. After thousands of iterations, the model learns to detect leopards accurately.
Training Run Summary
loss
Current Epoch
GPU Utilisation
Batches / sec
Est. time remaining
Model accuracy
GPU Resource Monitor
GPU Utilisation0%
VRAM Usage0%
CPU Dataloader0%
PCIe Bandwidth0%
Common Bottlenecks · Student Guide
BottleneckSymptom
CPU DataLoaderGPU idles, low utilisation
Too few workers<50% GPU usage
Heavy augmentationSlow epochs, CPU maxed
Large batch_sizeOut-of-Memory (OOM)
Many negativesSlow convergence
training.log · ai-server-01
[SYSTEM] AI Training Server online · NVIDIA GPU ready · CUDA initialised · Press "Run Training Epoch" to start.