Skip to content

About

Low-Latency Vision-Language Perception Pipeline with TensorRT INT4 Quantization & ROS2 Robotics Telemetry

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

▓█████ ▓█████▄   ▄████ ▓█████     ██▒   █▓ ██▓     ███▄ ▄███▓
▓█   ▀ ▒██▀ ██▌ ██▒ ▀█▒▓█   ▀    ▓██░   █▒▓██▒    ▓██▒▀█▀ ██▒
▒███   ░██   █▌▒██░▄▄▄░▒███       ▓██  █▒░▒██░    ▓██    ▓██░
▒▓█  ▄ ░▓█▄   ▌░▓█  ██▓▒▓█  ▄      ▒██ █░░▒██░    ▒██    ▒██
░▒████▒░▒████▓ ░▒▓███▀▒░▒████▒      ▒▀█░  ░██████▒▒██▒   ░██▒
░░ ▒░ ░ ▒▒▓  ▒  ░▒   ▒ ░░ ▒░ ░       ░ ▐░  ░ ▒░▓  ░░ ▒░   ░  ░
 ░ ░  ░ ░ ▒  ▒   ░   ░  ░ ░  ░       ░ ░░  ░ ░ ▒  ░░  ░      ░
   ░    ░ ░  ░ ░ ░   ░    ░            ░░    ░ ░   ░      ░
   ░  ░   ░          ░    ░  ░          ░      ░  ░       ░
        ░                              ░

SYS_05 // MULTI-MODAL VISION-LANGUAGE EDGE PERCEPTION

[SYSTEM ID]       SYS_05
[DEPLOYMENT]      NVIDIA Jetson AGX Orin 64GB (Autonomous Robotics)
[QUANTIZATION]    4-Bit AWQ (INT4) TensorRT Acceleration
[SENSOR FUSION]   ROS2 (Robot Operating System) Transform Pub/Sub
[FRONTEND STACK]  Vite + React 19 + TypeScript (Bun)
[BACKEND STACK]   FastAPI + GStreamer + OpenCV + Python 3.11 (uv)
[COMMAND RUNNER]  just (Justfile)
[PERF METRICS]    60 FPS Real-Time (<18ms) // 75.4% Memory Cut // <25W Power

⚡ ARCHITECTURAL SPECIFICATIONS

Ultra-efficient edge vision-language reasoning engine engineered for autonomous robotics and situational awareness. Features 4-bit AWQ weight quantization, TensorRT acceleration, open-vocabulary object grounding, and temporal video comprehension within a 25W embedded power envelope.

 ┌────────────────────────────────────────────────────────────────────────┐
 │                      GStreamer CSI Camera Stream                       │
 │                    (640x480 @ 60 FPS Video Pipeline)                   │
 └──────────────────────────────────┬─────────────────────────────────────┘
                                    │
 ┌──────────────────────────────────▼─────────────────────────────────────┐
 │            TensorRT AWQ Engine (4-Bit INT4 Quantization)               │
 │           - 75.4% Memory Reduction vs FP16 Baseline                    │
 │           - Sub-16ms End-to-End Perception Forward Pass                │
 └──────────────────┬─────────────────────────────────┬───────────────────┘
                    │                                 │
 ┌──────────────────▼──────────────┐   ┌──────────────▼───────────────────┐
 │   Open-Vocabulary Grounding     │   │     Spatial Object Tracker       │
 │   - Text Embeddings Match       │   │     - Kalman Filter Centroid     │
 │   - Zero-Shot Category Predict  │   │     - 3D Velocity & Trajectory   │
 └──────────────────┬──────────────┘   └──────────────┬───────────────────┘
                    │                                 │
                    └─────────────────┬───────────────┘
                                      │
 ┌────────────────────────────────────▼───────────────────────────────────┐
 │                     ROS2 Middleware Publisher                          │
 │         Topic: /edge/perception/tracked_objects (60 Hz)                │
 │         Msg:   geometry_msgs/TransformStamped                          │
 └────────────────────────────────────────────────────────────────────────┘

🛠 INDEPENDENT EXECUTION WITH just

Frontend and backend run independently or concurrently:

# Display all recipes
just

# Install backend (uv) and frontend (bun) dependencies
just install

# Run backend independently on http://localhost:8005
just backend

# Run frontend independently on http://localhost:5175
just frontend

# Run both backend and frontend concurrently
just dev

# Run test suite
just test

# Execute open-vocabulary spatial detection via CLI
just detect prompt="locate robotic arm components"

# Inspect Jetson hardware telemetry via CLI
just telemetry

📦 CONTAINERIZATION (DOCKER & DOCKER-COMPOSE)

# Build multi-stage image
just docker-build

# Launch perception container
just docker-up

# Verify health
curl -s http://localhost:8005/health | jq .

🌐 API SPECIFICATION

Verb Path Description
GET /health Hardware socket and TensorRT runner health
GET /api/v1/telemetry Jetson power draw, FPS, and ROS2 stats
POST /api/v1/detect Execute natural language spatial grounding
GET /api/v1/stream/mjpeg Live MJPEG video stream with bounding boxes

About

Low-Latency Vision-Language Perception Pipeline with TensorRT INT4 Quantization & ROS2 Robotics Telemetry

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages