FIM Challenge++

Picture the city tile behaviour you already built in Sprint 1–2 — a parking gate, traffic lights, waste point, safe crossing, or another FIM demonstrator. Now you upgrade that same city challenge with your own AI system: you collect a dataset, train or carefully fine-tune a model, run inference on or near the embedded setup, and wire predictions into the city behaviour so a rule-based or sensor-only path is replaced or augmented. Building on your earlier city demonstrator, re-implement and extend the firmware as ESP32 C++ on FreeRTOS (not Arduino as the delivered stack), with clean electronics toward a PCB path.

In this challenge you design and build a FIM Challenge++: choose one completed Sprint 1–2 FIM city challenge and add an AI pipeline that changes how the system decides. You evaluate against the baseline behaviour, integrate with the city’s shared firmware, make AI status and predictions available to full-stack classmates, and work through research, advice, design, and realisation for your Sprint 1-2 Learning Goal. Follow the city Requirements.

Your Learning Goal title in Portflow

Use: Sprint 1-2 - Learning Goal - FIM Challenge++ (adjust the sprint number if needed). See the Learning Goal page for the Portflow template.

Illustration

The animation below shows the idea in simplified form: familiar city-tile behaviour with an AI overlay improving a decision. Your delivered system must show a clear baseline FIM behaviour, then the AI-augmented path with metrics and city integration.

Challenge

Your Learning Group city needs an AI upgrade of one FIM city challenge you (or your city) already realised. Choose among existing FIM challenges in this folder, for example:

You collect data, train (or carefully fine-tune) a model, run on-device or edge inference, and integrate predictions so they replace or augment a rule-based part of that challenge (e.g. sensor classification, anomaly detection, vision instead of ultrasonic-only). Your work stays part of one shared embedded firmware for the city — not a parallel personal AI stack that never joins main.

You will deliver a working prototype and the documents that justify and explain it. Work in this order:

Understand the problem. Which FIM challenge is your baseline? What did the rule-based system do well and where does AI help? Write clear rules, e.g. “Model classifies approach occupancy; firmware only turns green when confidence ≥ T.” Agree how the upgrade fits the city plan on the same tile.
Research. Compare realistic data collection, model options (own train vs fine-tune), inference targets (MCU, Pi, Edge Impulse, TensorFlow Lite), and how to evaluate against the FIM baseline. Include how you will expose AI status and predictions to full-stack. Cite sources; plan metrics (accuracy, latency, confusion).
Advise. Recommend one architecture: data plan, model artefact, inference host, how predictions enter the FIM state machine, fail-safe when confidence is low, and the data/control interface for full-stack. State what you are not building in this period.
Design. Specify hardware/firmware changes, dataset versioning, training config, inference module, interface messages, and evaluation plan so someone else could rebuild the upgraded system.
Realise. Build and test in small steps. Prefer separate experiments (dataset collection, offline training/eval, inference-only, then wire into city behaviour) before you merge into the shared city firmware on GitLab main. Keep each experiment tiny and logged. Then demonstrate the upgraded tile behaviour with AI in the loop. Use stubs if full-stack is not ready — do not wait on classmates to progress.

Minimum scope (Sprint 1–2)

Your contribution must include:

A clear baseline: which FIM challenge you upgrade, and what the Sprint 1–2 system did without AI (link the sibling page; document behaviour in your own words).
An AI goal that replaces or augments a concrete rule-based or sensor-only decision in that challenge.
A data collection plan and a versioned dataset you collect (or carefully curated city data with provenance) — enough labelled samples for train/validation/test splits you document.
A trained model artefact (own train or careful fine-tune) with training config and logs; seeds where applicable.
On-device or edge inference wired into the city challenge behaviour on the physical tile (or documented bench stand-in), not only offline notebook predictions.
Evaluation metrics vs baseline (e.g. accuracy/F1, confusion matrix, latency, false-trigger rate of the old ultrasonic path vs the AI path).
Fail-safe behaviour when confidence is low or inference fails (fall back to safe FIM default, e.g. red / closed / idle).
Integration into the city’s one shared embedded firmware on GitLab main, implemented as ESP32 C++ on FreeRTOS (ESP-IDF or PlatformIO + ESP-IDF; not Arduino as the delivered firmware). Follow city conventions.
A documented data and control interface for full-stack developers: AI status, prediction class/score, and optional override or “use baseline” mode (message names, payload fields, and an example). Full-stack may connect later — you still define and demo the interface, with stubs if needed.
Updates to the city’s shared BOM and wiring diagram for any new sensors/camera/compute.

Recommended scope

If you have time, add one or more of:

Continuous capture tooling so the city can grow the dataset between demos.
On-MCU inference (TFLite Micro / Edge Impulse) with a measured latency budget.
A side-by-side demo switch: baseline rules vs AI path on the same tile.
A stub or simulator feed that full-stack can use before hardware is on the desk.
Quantisation or pruning notes if you squeeze the model onto smaller hardware.

Out of scope for this challenge

These are not part of this challenge:

Starting a brand-new city challenge unrelated to a Sprint 1–2 FIM demonstrator (use Autonomous City Car or Robotic System for greenfield Advanced topics).
Training huge foundation models from scratch when that is not feasible — small custom models and careful fine-tunes are in scope.
Cloud-only inference with no edge/on-device path in the city demo.
Building the city back-end, front-end, or Raspberry Pi stack yourself — that is full-stack work. You do define and expose the embedded/AI interface they connect to.
A separate personal firmware or ML repo that never merges relevant modules into the city’s shared application.
Delivering the assessed embedded path as Arduino .ino sketches — Advanced work uses ESP32 FreeRTOS in C++.

Demo and safety context

Treat the prototype as lab equipment with AI in the decision path:

Default to the safe FIM state when the model is unsure or inference fails; never let a bad prediction leave the system in an unsafe actuator state.
Predictable rules. Document confidence thresholds and fallback to baseline behaviour.
Clear demo script. Anyone in your Learning Group can show at least: baseline reminder → AI inference on a live sample → city behaviour updates → low-confidence / fail → safe fallback; plus a glance at metrics.
Independent progress. You can demo inference and messages with stubs while classmates finish their parts.

Starting points

Use these concrete first steps early in Sprint 1:

Read the city Requirements and agree which FIM challenge you upgrade (no duplicate ++ on the same tile without a plan).
Pick the sibling challenge and re-read its minimum behaviour (e.g. Parking Area, Smart Traffic Light System, Smart Waste Collection Point, Safe Crossing).
Write the AI goal in one sentence. What decision improves, and how will you measure it against the baseline?
Inventory hardware and compute. Existing tile sensors plus camera/mic/extra MCU as needed. Note gaps early; update the city’s shared BOM.
Test in small experiments first. Collect a pilot dataset; train offline; measure metrics; run inference alone; only then splice into the FIM state machine. Do not develop everything only inside the city’s main shared firmware.
Confirm toolchain. Deliver the city firmware on ESP32 with FreeRTOS in C++ (ESP-IDF, or PlatformIO using the ESP-IDF framework). Run training offline in Python if needed; run inference on-device or at the edge and hand results into FreeRTOS tasks — not Arduino .ino sketches as the assessed firmware. See RTOS application development and Operating systems and RTOS.
Draft the full-stack interface with your city (AI status, predictions, fallback mode). Agree a simulation path so neither side waits.
Draft your research question. Example: Which dataset, model, and inference target best upgrade our FIM <challenge> given latency, false triggers, and measurable gain over the rule-based baseline?
Create your Learning Goal in Portflow. Link the product deliverables you plan to submit. Follow Learning Goal.
Integrate only what works. Prove offline quality, then edge inference, then city behaviour; move proven modules into the shared firmware in small merges.

Suggested sprint timeline

The Learning Group does not use Scrum, but you still follow the sprint rhythm of the Group Project: two sprints for this challenge (Sprint 1-2).

In the first sprint you focus on research and advice (plus early experiments: data pilot, offline model, baseline metrics). In the second sprint you focus on design and realisation of the full advanced upgrade: edge inference in the FIM loop, fail-safe fallback, city integration, and the full-stack AI interface.

Sprint	Focus	Outcome
1	Research & advise (+ early experiments)	Research report; advisory report; FIM baseline documented; AI goal and data plan; draft AI status interface; pilot dataset and offline metrics
2	Design & realisation (full AI upgrade)	Design with training config + integration points; shared BOM updated; on-device/edge inference in shared firmware on city GitLab `main`; metrics vs baseline; demo with fail-safe fallback; Portflow evidence linked; reflection

Tools, frameworks, and hardware

Area	Suggestions
Baseline FIM tile	Same LightBurn tile as your Sprint 1–2 challenge
Microcontroller / edge	ESP32-S3 / ESP32-C3 with FreeRTOS + C++; optional host for heavier vision/training
AI tooling	Edge Impulse, TensorFlow Lite / TFLite Micro, OpenCV, scikit-learn for small tabular models
Sensors	Existing FIM sensors plus optional camera/mic for the AI path
IDE / tooling	ESP-IDF or PlatformIO + ESP-IDF (FreeRTOS C++); Python for training
Version control	City GitLab project, shared firmware on `main`; versioned dataset artefacts
Documentation	Markdown on the city MkDocs site and in the repo (English); cite sources; training logs
Full-stack link	Documented messages (Serial, MQTT, or city-agreed protocol); stubs/simulation so you do not block each other

Profile knowledge worth revisiting:

Products

Your challenge evidence is a set of products: studio deliverables tied to learning outcomes. Submit them in Portflow and link them from your Learning Goal.

Analyse & Advise

Product	What you deliver	Template
Research	Analysis of AI upgrade options for your chosen FIM challenge; data and model comparison; evaluation plan vs baseline; recommendation for design	Research product
Advice	Decision document: baseline summary, AI goal, dataset plan, model/inference choice, fail-safe fallback, full-stack AI status interface, and implementation order (data → train → infer → integrate)	Advisory report

Suggested titles for Portflow

Research: AI upgrade options for FIM Challenge++ (<chosen challenge>)
Advice: On-device AI architecture for upgrading <chosen challenge>

The advice report should build on the research report. Do not repeat the full comparison tables.

Design & Realise

Product pages for design and realisation are still being added to the studio catalogue. For this challenge, deliver the following and attach them as evidence (repository links, PDFs, or videos as your coach specifies).

Product	Learning outcome	What you deliver
System design	Design & Realise	Self-contained design document: FIM baseline context, AI goal, city requirements link-in, block diagram (sensors/camera → model → FIM state machine), dataset versioning, training config, inference placement, fail-safe rules, software module overview for the shared firmware, AI status/prediction interface for full-stack, contribution to the city BOM, and test/evaluation plan
Prototype realisation	Design & Realise	Working AI-upgraded prototype on your city tile (or documented bench stand-in), model artefact + inference in the city’s shared firmware on GitLab `main`, README with build/train/run steps, interface demo (live or simulated), metrics vs baseline, and validation evidence of AI-driven city behaviour with safe fallback

Suggested titles for Portflow

Design: FIM Challenge++ — AI pipeline and firmware integration for <chosen challenge>
Realisation: FIM Challenge++ — on-device inference prototype and validation

System design — minimum contents

Your design document should let a peer rebuild and evaluate the upgrade without guessing:

Requirements. Baseline FIM behaviour, AI goal, confidence/fallback rules; how this fits the city Requirements.
Architecture diagram. Existing FIM I/O, new AI path, inference host, power, and the path to full-stack (protocol boundary).
Dataset. How labels are defined; split; version id; ethics/privacy notes if people appear in frames.
Model. Architecture, training config, seeds, expected artefact format.
State machine changes. Where predictions enter the FIM logic; what happens on low confidence.
Timing / latency budget. Inference rate vs control loop needs.
Pin map and wiring diagram. Any new camera/sensor/compute wiring.
BOM. Parts list aligned with the city’s shared BOM.
Evaluation plan. Metrics vs baseline; confusion matrix or equivalent; latency.
Full-stack interface. AI status, predictions, fallback mode; example payloads; stub test path.
Shared firmware. Where your modules live and which code conventions you follow.
Test plan. e.g. “offline metrics”, “edge inference only”, “integrated happy path”, “low confidence → safe fallback”, “override to baseline”.

Realisation — minimum evidence

Small experiments (or a short test log) showing dataset pilot, offline eval, inference-only, then integration — before merging everything.
Work on the city GitLab project: proven AI modules merged (or ready to merge) into main — not a dump of unfinished notebooks into main.
Photo or diagram of the upgraded build on the tile, labelled.
Demo script (numbered steps) matching your video or live assessment, covering baseline contrast, AI path, fallback, and the data/control interface.
Versioned dataset, model artefact, training logs, and metrics vs baseline.
Proof you can progress without waiting on full-stack (simulation, Serial demo, or stub consumer).
Honest known limitations (e.g. class imbalance, lighting, domain shift from training desk to city tile).

Quality indicators

When assessing this challenge as a whole, the following indicators are used. Use them as a self-check before you submit.

Problem fit

The baseline FIM challenge, AI goal, and fallback rules are described in plain language.
Scope is explicit: this challenge upgrades one Sprint 1–2 FIM demonstrator with your own AI pipeline in the city behaviour — not a greenfield challenge and not offline-only ML.
The work respects the city Requirements (tile, shared firmware, BOM/wiring, full-stack interface, no waiting on classmates).

Analyse & Advise products

Research and advice reports meet the research and advisory report quality indicators.
The advice report references the research report and states one clear recommended AI architecture, including evaluation against the FIM baseline and how predictions reach full-stack.
Model and data choices are supported by evidence, not only opinion. Decisions cite resources.

Design

The design document is self-contained and includes dataset versioning, training config, inference placement, and FIM state-machine integration that match the intended behaviour.
Pin map, wiring diagram, and BOM match the realised upgrade (or differences are documented with reason).
Fail-safe behaviour is designed explicitly (low confidence / inference fail → safe FIM default).
The AI status/prediction interface for full-stack is specified with example messages.
Experiments are designed to be reproducible (dataset version, seeds/config, training log).

Realisation

The prototype demonstrates on-device or edge inference changing the upgraded FIM city behaviour on the physical setup.
A trained model artefact and metrics vs baseline are part of the evidence.
Behaviour lives in the city’s shared firmware on GitLab main (or is clearly opened as an integration MR), following city code conventions.
Firmware/ML code is readable: meaningful names, constants for thresholds, no unexplained magic numbers.
The full-stack interface is demonstrable (live or with simulation/stubs) without depending on a finished back-end.
Validation evidence covers AI happy path, low-confidence fallback, and at least one edge case (e.g. novel sample, lighting change).
Repository README explains how to rebuild the dataset path, train or load the model, flash/run, and demo as part of the city project.

Professional practice

Learning Goal in Portflow uses the correct naming format and links to all product evidence.
End-of-sprint reflection connects results to your learning question and to embedded+AI practice (data, evaluation, safety, city integration).
Documentation is in English Markdown and usable by other city students on their tiles.
You did not block on classmates: simulation, stubs, or a local demo kept you moving.

Focus points

Watch out for these common issues while you work:

Arduino as the delivered stack. Keep throwaway experiments short; the assessed ESP32 path is FreeRTOS + C++.
AI without integration. Two sprints mean inference in the city behaviour, not only a notebook accuracy plot.
No baseline comparison. Always measure the upgrade against the FIM rule-based path.
Everything in main firmware. Develop and test parts in small experiments first; only integrate working pieces into the shared city firmware.
Unsafe predictions. Always define confidence thresholds and safe fallback.
Waiting on full-stack. Define the AI interface early and demo with Serial/stubs; do not pause realisation until the Pi dashboard exists.
Unreproducible training. Version the dataset and config; keep seeds and logs.
Solo stack forever. Notebooks are fine for learning; assessed integration still joins the shared city firmware per the Requirements.
Advice without analysis. Coaches will send back an advice report that invents new comparisons not present in research.
Design–code drift. Update the design document when you change labels, thresholds, models, or interface messages during realisation.

Key points

Core task: upgrade one FIM city challenge with your own AI (data → train → edge inference → city behaviour).
Deliver embedded integration as ESP32 FreeRTOS in C++, not Arduino.
Because this is Sprint 1-2, the delivered system shows metrics vs baseline and fail-safe fallback — not only offline ML.
Test small parts in separate experiments; integrate only what works into one shared firmware.
Expose AI status and predictions for full-stack; keep a shared BOM and wiring diagram.
Deliver four products: research, advice, system design, and prototype realisation.
Work research → advice → design → realise; Sprint 1 emphasises research & advice (and data/model proofs); Sprint 2 emphasises integrated design & realisation.
Greenfield Advanced topics belong in other Advanced challenges; building the city back-end/front-end is out of scope for you.
Link everything to your Learning Goal in Portflow and reflect at the end of Sprint 1-2.