Watch the Live Action Demo
See the 3DoF physical arm solve pathing, execute suction picks, and communicate telemetry in real-time.
Core Capabilities
An engineered pipeline coordinating hardware and software layers.
On-Device Computer Vision
Uses YOLO26x optimized and compiled to CoreML, running directly on the team's iPhones to locate and classify objects locally.
Central Brain
Coordinates kinematics, validates safety envelopes, persistent logs, registers camera status, and routes operator demands.
ESP32-S3 Edge Controller
Runs native WebSocket server to digest joint angles and operate physical stepper motors and vacuums with low latency.
Digital Twin Telemetry
Renders physical movements live on a Three.js simulator showing current robotic poses and active detections.
Autonomous Sequence
Fully autonomous loop: triggers detection, solves inverse kinematics, verifies constraints, and executes picking.
Download PickPilot Vision
Transform your iPhone into a high-precision spatial detection sensor. Runs YOLO/CoreML locally and hosts a local REST server.
Scan to install on iOS device
v1.0.0 • iOS 16.0+Operational Workflow
How PickPilot links perception to execution in a closed loop.
1. 1. Capture & Object Detection
The iOS Vision service captures camera feed, executes YOLO inference, estimates depth, and returns structured 3D spatial points.
2. 2. Path Planning & Kinematics
Next.js resolves inverse kinematics (IK) for the joint angles, performs safety boundary checks, and logs execution stages.
3. 3. Direct Hardware Execution
Embedded ESP32-S3 parses commands over WebSockets, adjusts steppers, toggles suction, and streams logs back to Mission Control.
System Architecture
How the modules interact within the local network topology.
iOS Vision App (YOLO/CoreML REST Service)
REST Endpoint / YOLO Client
Next.js Backend (Central Coordinator & database)
Next.js / Trajectory IK Solver
ESP32-S3 Firmware (WebSocket Server)
WebSocket Server / Stepper motor drivers
Mission Control (Next.js operator board)
Next.js operator interface client
Target Use Cases
Flexible robotics platform designed for multiple scopes.
Logistics & Smart Sorting
Simulating high-frequency packaging sorting and conveyor picking setups under industrial configurations.
Robotics System Integration
Testing control schemes, limits, network topology, and sensor options without costly industrial machinery.
Academic & Lab Research
An accessible platform for experimenting with machine learning, path planning, and embedded control models.