Real-Time Edge Computer Vision: Deploying YOLO on Embedded Systems

Overview

Deploying deep convolutional neural networks onto resource-constrained edge hardware—such as the Raspberry Pi 4/5, NVIDIA Jetson Nano, or microcontroller companion boards—demands careful trade-offs between inference latency, thermal limits, and detection accuracy.

During the development of our award-winning Autonomous Road Inspection Robot, we engineered a real-time computer vision pipeline capable of detecting surface anomalies, potholes, and cracks while concurrently transmitting GPS telemetry and navigation coordinates.

flowchart TD
    A[Camera Feed 30 FPS] --> B[OpenCV Preprocessing & Resizing]
    B --> C[ONNX Runtime / TensorRT INT8 Engine]
    C --> D[Non-Maximum Suppression (NMS)]
    D --> E[Classify Anomalies & Bounding Boxes]
    E --> F[GPS Geotagging & Telemetry Stream]
    E --> G[Visual Feedback Overlay]

---

1. Selecting and Fine-Tuning the YOLO Architecture

For real-time edge processing, standard YOLOv8x or YOLOv11x models are computationally prohibitive. Instead, YOLOv8n (Nano) or YOLOv8s (Small) provides the ideal architectural balance.

Training Strategy:

• Custom Dataset Augmentation: Applied Mosaic, MixUp, HSV color jitter, and random perspective transformations to simulate varied sunlight and road asphalt textures.
• Anchor & Loss Optimization: Tuned Complete IoU (CIoU) and Distribution Focal Loss (DFL) parameters for accurate localization of irregularly shaped cracks.

from ultralytics import YOLO

# Load lightweight YOLO backbone
model = YOLO("yolov8n.pt")

# Fine-tune on custom road anomaly dataset
results = model.train(
    data="road_surface_data.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
    device="cuda",
    plots=True
)

---

2. Exporting and Quantizing with TensorRT & ONNX

Running models directly in PyTorch incurs substantial runtime overhead. Exporting the trained checkpoint to ONNX and performing INT8 Post-Training Quantization (PTQ) reduces memory bandwidth consumption by up to 75% and accelerates matrix computations on edge tensor cores.

# Export trained model to ONNX with dynamic batching
yolo export model=best.pt format=onnx dynamic=False simplify=True imgsz=640

Quantization Benefits:

Format	Precision	Model Size	Raspberry Pi 5 FPS	Jetson Nano FPS
PyTorch (.pt)	FP32	14.2 MB	6.4 FPS	14.1 FPS
ONNX (.onnx)	FP32	12.1 MB	11.2 FPS	21.0 FPS
ONNX Runtime / OpenVINO	INT8	3.8 MB	28.6 FPS	34.2 FPS

---

3. Asynchronous Threading Pipeline in Python & OpenCV

In a single-threaded architecture, frame capture from the CSI/USB camera blocks detection inference. Decoupling frame acquisition and inference into distinct worker threads ensures constant 30 FPS camera ingest without dropped frames:

import cv2
import threading
import time

class VideoStreamWidget:
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        self.capture.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
        self.capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
        self.status, self.frame = self.capture.read()
        self.stopped = False
        
        # Start background capture thread
        self.thread = threading.Thread(target=self.update, args=(), daemon=True)
        self.thread.start()

    def update(self):
        while not self.stopped:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(0.01)

    def read(self):
        return self.frame

    def stop(self):
        self.stopped = True

---

Key Takeaways

1. Model selection matters: Start with Nano backbones before attempting pruning on heavier variants.
2. Quantization is essential: INT8 execution cuts power draw and unlocks real-time throughput on low-wattage boards.
3. Decouple I/O: Threading cameras and inference engines prevents frame lag and maintains steady robotic feedback loops.