Autonomous City Car

Picture the roads on your Learning Group's city tiles. Wireless driveable cars with controllers already exist in the studio. You mount a camera above the city, perceive cars and lanes from that overhead view, and drive a car through town autonomously — perception, planning or control, then wireless actuation. That is robotics plus AI on the model city: a closed loop from image to motion, with fail-safe stop when the world is unclear.

In this challenge you design and build an autonomous city-car demonstrator: overhead vision → detect and track → command the car wirelessly. Building on your earlier city demonstrator, you raise the bar with a versioned perception pipeline, evaluation metrics, and ESP32 firmware in C++ on FreeRTOS (not Arduino sketches as the delivered stack). You integrate with the city’s shared firmware where drive commands live, make status and controls available to full-stack classmates, and work through research, advice, design, and realisation for your Sprint 1-2 Learning Goal. Follow the city Requirements.

Your Learning Goal title in Portflow

Use: Sprint 1-2 - Learning Goal - Autonomous City Car (adjust the sprint number if needed). See the Learning Goal page for the Portflow template.

Illustration

The animation below shows the idea in simplified form: an overhead camera watches the roads while a car follows a path through the city. Your delivered system must close the loop — perception, control, and wireless drive — with logging and a documented fail-safe stop.

Challenge

Your Learning Group city needs an autonomous drive loop on the model roads: a camera above the tiles perceives the scene; your software detects and tracks the car (and optionally lanes or markers); a control loop sends wireless drive commands; and the system stops safely when confidence is low. Your work ties into shared city conventions for firmware and interfaces — not a disposable laptop-only demo that never talks to the city stack.

You will deliver a working prototype and the documents that justify and explain it. Work in this order:

Understand the problem. Which car(s) and which road stretch will you demo? Where does the camera mount? What does “autonomous” mean for two sprints (follow a route, stay in lane, stop at a marker)? Write clear rules, e.g. “If the car is not detected for N frames, send stop.” Agree how this fits the city plan and which tiles you use.
Research. Compare realistic overhead perception approaches (classical CV, own small model / fine-tune, hybrid), control strategies, and wireless drive protocols already used by the studio cars. Include how you will expose status and commands to full-stack. Cite papers, datasheets, and toolkit docs; plan evaluation metrics against a baseline (e.g. manual drive or open-loop).
Advise. Recommend one architecture: camera host (laptop, Pi, or MCU + co-processor), model or CV pipeline, control loop, wireless protocol, fail-safe stop, and the data/control interface for full-stack. State what you are not building in this period.
Design. Specify hardware (camera mount, wiring, BOM), software (pipeline stages, dataset versioning, training config if ML, control gains, interface messages), and reproducible experiment notes so someone else could rebuild and evaluate your system.
Realise. Build and test in small steps. Prefer separate experiments (camera feed, detection offline, wireless drive only, then closed loop) before you merge into the shared city firmware / city repo on GitLab main. Keep each experiment tiny and logged. Then demonstrate autonomous drive on the physical city tiles. Use simulation or stubs if full-stack is not ready — do not wait on classmates to progress.

Minimum scope (Sprint 1–2)

Your contribution must include:

Demo on the physical city tiles (roads from the LightBurn templates), with a documented camera mount above the city (laptop + USB camera or Pi camera is fine to start).
An overhead perception pipeline that detects and tracks the car(s) and/or lanes/markers in the camera view. Own ML or a classical CV + ML hybrid is expected — not only hard-coded colour thresholds without evaluation.
A control loop that steers/drives the car wirelessly; document the protocol (message format, rates, who owns the radio link).
A fail-safe stop: when perception confidence is low, tracking is lost, or commands cannot be sent, the car stops (or the documented safe command is sent).
Logging of perception outputs (e.g. detections, confidence) and drive commands over time, enough to replay or analyse a demo run.
Evaluation of perception/control quality with explicit metrics (e.g. detection precision/recall or IoU on a held-out set, tracking continuity, success rate of a scripted route, latency) compared to a documented baseline.
Reproducible experiments: dataset version (or labelled frame set), training/config seeds where applicable, and a short training or calibration log.
Integration path into the city’s shared firmware / city GitLab main for the drive or status side, implemented as ESP32 C++ on FreeRTOS (ESP-IDF or PlatformIO + ESP-IDF; not Arduino as the delivered firmware). Follow city conventions; do not leave the assessed result only as a private notebook that never joins the city repo.
A documented data and control interface for full-stack developers: autonomy status, detections or pose summary, last command, and stop/override (message names, payload fields, and an example). Full-stack may connect later — you still define and demo the interface (Serial, MQTT, HTTP, or another city-agreed protocol), with stubs if needed.
Updates to the city’s shared BOM and wiring / mount diagram for camera, compute, and radio path.

Recommended scope

If you have time, add one or more of:

Lane or marker following in addition to car tracking.
Multi-car awareness (detect more than one vehicle; still drive one).
On-device or edge inference (e.g. TensorFlow Lite, Edge Impulse, OpenCV on Pi) with measured latency.
A replay tool that overlays logged detections and commands on recorded video.
A stub or simulator feed that full-stack can use before hardware is on the desk.

Out of scope for this challenge

These are not part of this challenge:

Full self-driving car industry stacks (HD maps, fleet ops, V2X standards compliance).
Outdoor public roads or street-legal vehicles.
Training huge foundation models from scratch when that is not feasible in two sprints — small custom models, fine-tunes, and classical+ML pipelines are in scope.
Building the city back-end, front-end, or Raspberry Pi stack yourself — that is full-stack work. You do define and expose the embedded/edge interface they connect to.
A separate personal repo that never merges relevant firmware or interface modules into the city’s shared application.
Delivering the assessed embedded path as Arduino .ino sketches — Advanced work uses ESP32 FreeRTOS in C++.

Demo and safety context

Treat the prototype as lab equipment with moving parts:

Default to stop when perception is unsure, tracking is lost, or the link drops; never leave the car driving blind.
Predictable rules. Document exactly when autonomous mode may send throttle/steer vs when it must stop.
Clear demo script. Anyone in your Learning Group can show at least: start → car detected → car follows a short route or stays on the road → lose detection / press stop → car stops; plus a glance at logs or metrics.
Independent progress. You can demo perception offline and drive commands with stubs while classmates finish their parts.

Starting points

Use these concrete first steps early in Sprint 1:

Read the city Requirements and agree your challenge with the Learning Group (no duplicate Advanced challenges that collide on the same car/camera).
Inventory the wireless cars and controllers. Note protocol, range, and who may send commands; sketch the camera mount over the tiles.
Agree on the demo route. Which tiles, which start/stop pose, and what success looks like in two sprints.
Inventory hardware and compute. Camera, host (laptop/Pi/ESP), radio path, mounting. Note gaps early and ask your coach. Start the city’s shared BOM.
Test in small experiments first. Capture a short video; run offline detection; prove wireless drive without vision; only then close the loop. Do not develop everything only inside the city’s main shared firmware.
Confirm toolchain. Deliver on ESP32 with FreeRTOS in C++ (ESP-IDF, or PlatformIO using the ESP-IDF framework). Use FreeRTOS tasks/queues for perception hand-off, control, and radio — not Arduino .ino sketches as the assessed firmware. OpenCV / TensorFlow Lite / Edge Impulse may run on a host or co-processor beside that ESP32 path. See RTOS application development and Operating systems and RTOS.
Draft the full-stack interface with your city (status, detections, stop/override). Agree a simulation path so neither side waits.
Draft your research question. Example: Which overhead perception and control approach best fits the model city given latency, lighting, wireless protocol limits, and measurable route success?
Create your Learning Goal in Portflow. Link the product deliverables you plan to submit. Follow Learning Goal.
Integrate only what works. Prove perception, then drive, then closed loop with fail-safe; move proven modules into the shared city repo in small merges.

Suggested sprint timeline

The Learning Group does not use Scrum, but you still follow the sprint rhythm of the Group Project: two sprints for this challenge (Sprint 1-2).

In the first sprint you focus on research and advice (plus early experiments: camera feed, offline detection, wireless drive proof). In the second sprint you focus on design and realisation of the full advanced system: closed loop, fail-safe, logging, metrics, city integration, and the full-stack interface.

Sprint	Focus	Outcome
1	Research & advise (+ early experiments)	Research report; advisory report; camera mount and route chosen; baseline defined; draft data/control interface; camera + detection and/or wireless drive proven in small experiments
2	Design & realisation (full autonomous loop)	Architecture + wiring/mount + control design; shared BOM updated; closed-loop demo on city tiles with fail-safe stop; perception/command logs; metrics vs baseline; integration toward city GitLab `main`; Portflow evidence linked; reflection

Tools, frameworks, and hardware

Area	Suggestions
City tiles	Roads from the LightBurn templates
Camera	USB webcam or Pi Camera mounted above the city
Compute	Laptop, Raspberry Pi, or MCU + edge co-processor (align with city choices)
Perception	OpenCV; own model / fine-tune; TensorFlow Lite; Edge Impulse; classical + ML hybrid
Drive	Existing studio wireless cars + documented protocol
Firmware / tooling	ESP32 FreeRTOS + C++ (ESP-IDF or PlatformIO + ESP-IDF); optional host tools for vision/ML
Version control	City GitLab project, shared modules on `main`
Documentation	Markdown on the city MkDocs site and in the repo (English); cite sources; version datasets
Full-stack link	Documented messages (Serial, MQTT, HTTP, or city-agreed protocol); stubs/simulation so you do not block each other

Profile knowledge worth revisiting:

Products

Your challenge evidence is a set of products: studio deliverables tied to learning outcomes. Submit them in Portflow and link them from your Learning Goal.

Analyse & Advise

Product	What you deliver	Template
Research	Analysis of overhead perception, control, and wireless drive for the model city; comparison with criteria; evaluation plan vs baseline; recommendation for design	Research product
Advice	Decision document: camera/compute role, perception approach, control loop, fail-safe, full-stack data/control interface, and implementation order (perception → drive → closed loop)	Advisory report

Suggested titles for Portflow

Research: Overhead perception and control for autonomous city car
Advice: Closed-loop autonomous drive architecture for the Learning Group city

The advice report should build on the research report. Do not repeat the full comparison tables.

Design & Realise

Product pages for design and realisation are still being added to the studio catalogue. For this challenge, deliver the following and attach them as evidence (repository links, PDFs, or videos as your coach specifies).

Product	Learning outcome	What you deliver
System design	Design & Realise	Self-contained design document: context, city requirements link-in, block diagram (camera → perception → control → wireless drive), mount/wiring, dataset and training config if ML, control parameters, fail-safe rules, software module overview for city integration, data/control interface for full-stack, contribution to the city BOM, and test/evaluation plan
Prototype realisation	Design & Realise	Working closed-loop prototype on the city tiles, code integrated toward the city’s shared firmware / repo on GitLab `main`, README with build/run steps, interface demo (live or simulated), logs, metrics vs baseline, and validation evidence of autonomous drive with fail-safe stop

Suggested titles for Portflow

Design: Autonomous City Car — perception, control, and interface specification
Realisation: Autonomous City Car — closed-loop prototype and validation

System design — minimum contents

Your design document should let a peer rebuild and evaluate the system without guessing:

Requirements. Perception targets, control goals, fail-safe stop rules; how this fits the city Requirements.
Architecture diagram. Camera, compute, perception, controller, wireless path, power, and the path to full-stack (protocol boundary).
Perception design. Model or CV pipeline; dataset version; training/config; how detections become pose or lane features.
Control design. How commands are computed; rates; gains or planner rules; stop conditions.
State machine. Idle, tracking, driving, lost-track / stop, override — never drive without a documented safe path when unsure.
Pin map / mount and wiring diagram. Camera mount, compute, radio; include power notes.
BOM. Parts list aligned with the city’s shared BOM.
Evaluation plan. Metrics, baseline, held-out frames or scripted route criteria.
Full-stack interface. Messages for status, detections/pose, commands, stop/override; example payloads; stub test path.
Shared firmware / city repo. Where your modules live and which code conventions you follow.
Test plan. e.g. “detection on static frames”, “wireless drive only”, “closed loop short route”, “tracking lost → stop”, “override stop”.

Realisation — minimum evidence

Small experiments (or a short test log) showing you verified parts separately: camera feed, offline detection, wireless drive, then closed loop — before full integration.
Work on the city GitLab project: proven modules merged (or ready to merge) into main — not a dump of unfinished notebooks into main.
Photo or diagram of the physical setup (camera over tiles, car on road), labelled.
Demo script (numbered steps) matching your video or live assessment, covering autonomous drive, fail-safe stop, logging, and the data/control interface.
Metrics vs baseline and versioned dataset / config evidence.
Proof you can progress without waiting on full-stack (simulation, Serial demo, or stub consumer).
Honest known limitations (e.g. lighting sensitivity, latency, single-car only).

Quality indicators

When assessing this challenge as a whole, the following indicators are used. Use them as a self-check before you submit.

Problem fit

The demo route, camera setup, and autonomy rules (including fail-safe stop) are described in plain language.
Scope is explicit: this challenge delivers a closed-loop autonomous drive on the city tiles (perception → control → wireless actuation), integrated with city conventions — not only offline video analysis or only remote control.
The work respects the city Requirements (tiles, shared repo/firmware, BOM/wiring, full-stack interface, no waiting on classmates).

Analyse & Advise products

Research and advice reports meet the research and advisory report quality indicators.
The advice report references the research report and states one clear recommended architecture, including evaluation against a baseline and how status/commands reach full-stack.
Component and approach choices are supported by datasheets, toolkit docs, or comparable evidence, not only opinion. Decisions cite resources.

Design

The design document is self-contained and includes perception, control, fail-safe, and evaluation plans that match the intended system.
Mount/wiring diagram, BOM, and protocol notes match the realised prototype (or differences are documented with reason).
Fail-safe behaviour is designed explicitly (default stop when unsure, lost track, or link loss).
The data/control interface for full-stack is specified with example messages.
Experiments are designed to be reproducible (dataset version, seeds/config, training or calibration log).

Realisation

The prototype demonstrates overhead perception, wireless drive commands, and a closed loop on the physical city tiles, with fail-safe stop.
Behaviour lives in (or is clearly opened as an MR toward) the city’s shared firmware / repo on GitLab main, following city code conventions.
Code and notebooks are readable: meaningful names, constants for thresholds/rates, no unexplained magic numbers.
Perception and commands are logged; quality is reported with metrics vs a baseline.
The full-stack interface is demonstrable (live or with simulation/stubs) without depending on a finished back-end.
Validation evidence covers happy path, lost detection / stop, and at least one edge case (e.g. lighting change, override).
Repository README explains how to build, run, and reproduce the demo as part of the city project.

Professional practice

Learning Goal in Portflow uses the correct naming format and links to all product evidence.
End-of-sprint reflection connects results to your learning question and to robotics/AI practice (perception, control, safety, evaluation, city integration).
Documentation is in English Markdown and usable by other city students.
You did not block on classmates: simulation, stubs, or a local demo kept you moving.

Focus points

Watch out for these common issues while you work:

Arduino as the delivered stack. Keep throwaway experiments short; the assessed ESP32 path is FreeRTOS + C++.
Vision without actuation. Two sprints mean a closed loop on the tiles; offline detection alone is not enough.
Driving without a fail-safe. Always define and test stop-on-lost-track / stop-on-link-loss.
Everything in one mega-script. Develop and test parts in small experiments first; only integrate working pieces into the shared city repo.
Unreproducible ML. Version the dataset and config; record seeds and metrics; keep a baseline comparison.
Waiting on full-stack. Define the interface early and demo with Serial/stubs; do not pause realisation until the Pi dashboard exists.
Ignoring latency. Measure end-to-end delay; a perfect detector that is too slow will miss control deadlines.
Solo stack forever. Experiments are fine for learning; assessed integration still joins the shared city project per the Requirements.
Advice without analysis. Coaches will send back an advice report that invents new comparisons not present in research.
Design–code drift. Update the design document when you change mounts, models, protocols, or interface messages during realisation.

Key points

Core task: overhead perception → plan/control → drive the car wirelessly on the city tiles, with fail-safe stop.
Deliver embedded control as ESP32 FreeRTOS in C++, not Arduino.
Because this is Sprint 1-2, the delivered system is a closed loop with evaluation metrics — not only a camera demo or only remote control.
Test small parts in separate experiments; integrate only what works into the shared city repo/firmware.
Expose status and controls for full-stack; keep a shared BOM and mount/wiring diagram.
Deliver four products: research, advice, system design, and prototype realisation.
Work research → advice → design → realise; Sprint 1 emphasises research & advice (and early proofs); Sprint 2 emphasises full closed-loop design & realisation.
Industry AV stacks and outdoor public roads are out of scope; building the city back-end/front-end is out of scope for you.
Link everything to your Learning Goal in Portflow and reflect at the end of Sprint 1-2.