Projects — Jonathan Tao

Driverless Perception Pipeline

Ouster OS1 LiDAR on a test stand facing a course of orange cones at night, with a laptop running the pipeline — Bench testing the OS1-128 against a cone course at night.

C++, ROS 2, Ouster OS1-128, PointNet Transformer, ONNX Runtime, TensorRT, Jetson AGX Orin

Built a LiDAR-only ROS 2 perception pipeline for an Ouster OS1-128. It deskews each scan from odometry and IMU data, transforms the cloud into the vehicle frame, and removes ego and out-of-range returns before ground segmentation.

Replaced the vendored Patchwork++ with an in-house ground segmenter. It bins the cloud into concentric zones, fits a PCA plane to each patch from low seed points, rejects patches that are not upright, and keeps sparse distant patches as obstacles so far cones survive. Parallel binning and allocation-free patch processing removed about 72,000 heap allocations per frame and cut p95 latency from 3.91 ms to 0.95 ms.

Range-adaptive curved-voxel clustering groups obstacle returns into size-gated cone candidates. For each candidate, the C++ runtime samples up to 64 XYZ returns and canonicalizes them into the candidate box frame. A shared 6→64→128 PointNet MLP produces point features, then masked maximum, mean, and standard deviation pooling creates a 384-value shape descriptor. The graph appends twelve box features and five position features before projecting each candidate to a 128-value token.

Flowchart from masked XYZ point rows through PointNet pooling and candidate token projection — PointNet turns each masked XYZ cluster into one 128-value candidate token.

All candidate tokens from one scan enter 12 post-LayerNorm transformer blocks in one frame-wide inference call. Each block uses eight-head dense self-attention and a 128→512→128 feed-forward network. A learned per-head bias encodes the planar distance and bearing between every candidate pair, while a mission token supplies acceleration, skidpad, or track context without hard-coded rules.

Flowchart of geometry-biased self-attention, residual connections, normalization, and token MLP — One post-LayerNorm transformer block: geometry-biased attention, residual updates, and a per-token MLP.

A single output head scores each candidate as cone or non-cone from geometry alone, with temperature calibration baked into the graph. The acceptance threshold maximizes precision at 0.95 target recall. The composable ROS 2 node executes the dynamic-candidate graph through ONNX Runtime with TensorRT and CPU providers on an NVIDIA Jetson AGX Orin, then republishes each source cluster with its cone probability and original timestamp.

Flowchart from a final candidate token through the output MLP and calibrated sigmoid to cone probability — The final token feeds one output MLP and a temperature-scaled sigmoid for cone probability.

Composable C++ nodes and intra-process transport limit point-cloud copies. Post-processing merges accepted clusters within 0.4 m, confirms candidates across distinct odometry-associated views with log-odds evidence, and publishes centroid positions with range- and count-calibrated uncertainty.

Recorded-scene replay: raw OS1-128 cloud, size-filtered clusters, ground-truth cones, and final cone detections.

On a 630-frame real-scene replay, the complete detector reached 99.90% precision, 84.50% recall, and 8.78 cm localization RMSE. Moving inference from ONNX CUDA to TensorRT on the Orin cut p50 classifier-stage latency from 3.05 ms to 1.43 ms with no accuracy loss, at 2.09 ms p95.

Foxglove view of final perception cone detections with measured distances between cones — Final perception cones in Foxglove, with distances measured across the detected course.

Driverless Controls

Simulated track drive in Foxglove: racing line colored by planned speed, steering and acceleration commands, and planned, estimated, and true speed.

C++, ROS 2, Sparse Newton QP, Unscented Kalman Filter, Foxglove, EUFS Sim

Built the planning and control stack that turns the cone map into steering and acceleration commands. Lane detection grows track boundaries from the cone SLAM map and publishes a corridor of center-line points and widths. A path planner node turns each corridor into a racing line and speed profile, and a separate path follower node tracks it at 200 Hz, so a path solve can never delay a command.

Racing line

The planner moves each center-line point c_i along its fixed left normal n_i by a scalar offset d_i, bounded by the corridor width minus the car's half-width b and a safety margin m:

p_{i} = c_{i} + d_{i} n_{i}, - r_{i}^{R} \leq d_{i} \leq r_{i}^{L}

r_{i} = \min (d_{\max}, \max (0, w_{i} - b - m))

The objective is the lap time of the same speed profile the car follows:

T = \sum_{i} \frac{2 ‖ p_{i + 1} - p_{i} ‖}{v_{i} + v_{i + 1}}

Optimizer

Rather than optimize T directly, the planner minimizes a convex surrogate: squared second differences, which approximate curvature, weighted by inverse planned speed so the solve straightens the slow corners that cost the most time:

E (d) = \sum_{i} w_{i} {‖ p_{i - 1} - 2 p_{i} + p_{i + 1} ‖}^{2}

w_{i} = \frac{1}{\max (v_{i}, v_{\min})}

Because each point moves along a fixed normal, E is a box-constrained quadratic program with a banded Hessian. It is solved with projected Newton passes: compute the gradient, freeze offsets that sit on a bound and are pushed outward by adding a stiff diagonal penalty, solve for the Newton step with a sparse LDL^T factorization that costs linear time on the banded system, then project the step onto the bounds and halve it until E falls. The condition number of the Hessian grows with the fourth power of the point count, which is why the earlier Gauss-Seidel sweeps converged slowly and Newton steps do not.

An outer loop reweights from the best line's speed profile and solves again, accepting a new line only if the exact lap time T drops at a full, half, quarter, or eighth step, and stopping at the first rejection. The solver reads no clock and has fixed pass limits, so identical corridors give identical lines, and changed-input solves stay under a 40 ms p99 budget. Identical corridors skip the solve entirely.

Speed profile

Corner speed comes from curvature and lateral grip (11.7 m/s²), then is clamped to the 3 to 21.6 m/s speed range:

v_{i} = \sqrt{\frac{a_{lat, \max}}{\max (κ_{i}, κ_{\min})}}

Backward and forward passes then lower each limit so the car can brake into the next corner and accelerate out of the last one. The longitudinal acceleration available at each step shrinks inside a friction ellipse as lateral acceleration rises. Closed tracks run both passes twice so the limits carry across the start/finish line.

v_{i} \leftarrow \min (v_{i}, \sqrt{v_{i + 1}^{2} + 2 a Δ s_{i}})

a = a_{long, \max} \sqrt{1 - {(\frac{v_{j}^{2} κ_{j}}{a_{lat, \max}})}^{2}}

The corridor becomes a planned path and speed profile, which the steering and acceleration controllers track from the estimated state.

Steering

Pure pursuit picks the first path vertex past a speed-scaled lookahead L_d = clamp(0.5 m + 0.2 s · v, 0.5 m, 4 m) and steers along the circle through it, where x_t, y_t is the target in the vehicle frame. On its own, pure pursuit cuts corners at entry and exit, because the arc to the target does not match the path's curvature there. The planner publishes signed path curvature, so the controller adds it as feedforward and keeps only the difference between pure pursuit from the car and from the nearest path point as the position correction:

κ_{pp} = \frac{2 y_{t}}{x_{t}^{2} + y_{t}^{2}}

κ = κ_{path} + κ_{pp} - κ_{ref}

On the path, κ_pp = κ_ref and the car follows the path curvature exactly. The demanded curvature becomes a kinematic steering angle through the wheelbase L, plus proportional feedback on the gap between the demanded yaw rate vκ and the measured yaw rate. The angle is then clamped to the steering limit and rate-limited to the actuator's 2.35 rad/s.

δ = \arctan (L κ) + g_{yaw} (v κ - \dot{ψ})

Speed control

The acceleration command is two feedforward terms plus PID on the speed error e to the profile speed at the nearest vertex, clamped to ±15 m/s²:

a = g_{resist} v^{2} + g_{plan} a_{plan}

+ K_{p} e + K_{i} I + K_{d} \dot{e}

a_{plan} = \frac{v_{t}^{2} - v_{n}^{2}}{2 ‖ p_{t} - p_{n} ‖}

The resistance term adds back the acceleration lost to drag and rolling resistance, which grows with speed; without it the loop could only cover that loss by holding a speed error, and the integral would saturate before top speed. The planned-acceleration term reads the profile's own v² = v₀² + 2aΔs relation back between the nearest vertex n and the steering target t, so one lookahead serves both axes and every braking and acceleration ramp starts where the plan starts it rather than after an error opens. The PID loop (K_p = 5, K_i = 0.2, K_d = 0) only covers what the feedforward misses, and its anti-windup stores the clamped integral only when the full command, feedforward included, is unsaturated.

Safety and validation

The follower commands full braking and holds the last steering angle whenever the pose is older than 0.2 s, the trajectory older than 0.5 s, the car is off the path, or the target is not ahead. When the SLAM lap count reaches the mission's laps, it brakes while still steering along the path, stopping in about 11 m from 18 m/s, inside the 30 m the FSAE Driverless rules allow. Two diagnostics run on every tick without affecting commands: the ratio of achieved to demanded lateral acceleration, |vψ̇| / v²κ, which drops below one when the car runs wide, and the clearance from the car to the nearest mapped cone.

A 200 Hz Unscented Kalman Filter estimates vehicle twist by fusing INS pose, IMU yaw rate and acceleration, and wheel-speed odometry from an in-house adapter node. In simulation, the car tracked the plan with 0.18 m median and 0.35 m p95 tracking error.

For reference, the clip below shows the car driven by a human through a cone course; the controls results above come from simulation.

Human-driven reference run of the car through a parking-lot cone course.

2025–2026 IEM BMS

C, C++, FreeRTOS, STM32H7, Python, CAN, SPI, HIL

Led the migration of Illini Electric Motorsports' battery management system from bare-metal STM32F4 firmware to FreeRTOS on an STM32H733, which supplies the compute for battery modeling. The completed system monitors a 140-cell series pack through twelve LTC6811-2 devices connected by DMA-backed isoSPI.

IEM accumulator with twelve cell-monitoring boards and wiring — The 140-cell accumulator: twelve monitoring boards, eleven with twelve cells and one with eight.

Split cell polling, isolated pack sensing, state control, precharge protection, battery modeling, and CAN communication into scheduled tasks. The battery model estimates per-cell SOC, SOH, and SOP at 200 Hz using fixed-point OCV-seeded Coulomb counting and a 2-RC equivalent-circuit power model, and measured under 1% SOC error against OCV-derived SOC after 22 km of driving.

Flowchart of STM32H7 FreeRTOS tasks, peripheral interfaces, accumulator sensors, and car signals — System overview from accumulator and car inputs through STM32 peripherals and FreeRTOS tasks to the main contactor, BMS fault line, and CAN. Open the diagram for the full-size view.

BMS bench setup with STM32H7 controller, cell monitors, CAN adapter, and power supplies — Bench validation of the H7 controller, cell-monitor chain, isolated ADC, and CAN interface.

Converted the LTC6811-2 and AMC131M03 drivers from blocking SPI with 50 ms HAL delays to DMA, raising cell polling from 1 Hz to 10 Hz and battery-current sampling from 100 Hz to 4 kHz. Integrated an OpenBLT bootloader across the BMS and safety ECUs for CAN-based flashing through Bazel targets, then built Python and C++ validation tools around the firmware. My Python BMS Viewer displays all cell voltages and temperatures, pack state, faults, current, voltage, SOC, SOH, power limits, and CAN health while exposing controlled balancing, charging, and state-estimation commands.

BMS Viewer showing cell readings, temperatures, faults, and CAN diagnostics — The BMS Viewer during pack validation. Open the image for the full diagnostic view.

The migrated system completed integration and ran on the car. The acceleration test below exercised the pack, sensing, safety logic, telemetry, and contactor control together.

On-car acceleration testing with the completed H7/FreeRTOS BMS.

Morphi (Top 7 of 226 HackVoyager Finalist, HackIllinois 2026)

Devpost GitHub

React Native, Python, Modal, vLLM, YOLO Pose, LoRA, Qwen3-VL

Morphi turns a five-second video into coaching for sports, exercise, dance, or music practice. It finds mistakes in the user's motion, pauses playback at each one, draws the current and target joint positions, and narrates a specific correction. A score and session history show whether the next attempt improved.

Morphi's HackIllinois pitch and product demonstration.

I built the backend data and training pipeline. Distributed Modal jobs collected sports, yoga, and workout images, kept clear single-person poses with YOLO, and used Gemini 3 Flash to generate schema-validated corrections tied to detected joints. That process produced more than 2,300 labeled examples for LoRA fine-tuning Qwen3-VL-32B on eight H200 GPUs.

I served the adapted model with vLLM across four H200s and kept GPU containers warm to avoid repeated model startup. The complete path—from pose extraction through coaching JSON, overlays, and voice generation—fell from about three minutes to less than 24 seconds.

Custom CoreXY 3D Printer

Autodesk Inventor, C++, Klipper, Raspberry Pi

Designed the frame and motion system in Autodesk Inventor, fabricated the custom parts, and assembled the printer around salvaged Ender 3 electronics.

A speed test of the completed CoreXY motion system.

Configured Klipper on a Raspberry Pi 4 and tuned the motion system beyond 1,500 mm/s at 7,000 mm/s².

W.A.N.D.E.R. Autonomous Robot

C++, ESP32, Autodesk Inventor

Designed a 3D-printed maze robot and fused gyroscope, wheel-encoder, and distance-sensor data for localization.

The robot follows a programmed route between temporary maze walls.

Implemented mapping and pathfinding in C++ on an ESP32 so the robot could build and traverse an unknown maze without an external computer.