Robotic System

Picture a work area on your Learning Group's city tile — for example a waste point where items must move from container X to container Y. You build a multi-joint robot that senses objects or positions, sequences motion through several joints, and completes pick-and-place with an end-effector. At Advanced level, at least one intelligent perception or learning component supports the task (object detection, pose estimation, or grasp success) — not only open-loop servo sweeps. That is embedded robotics: sensing, kinematics-in-the-small, sequenced control, and safety limits on a city demonstrator.

In this challenge you design and build a robotic pick-and-place system for your Learning Group city tile: sense → plan a short sequence → pick from source → place at destination. Building on your earlier city demonstrator, you package the work like professional practice: PCB-minded electronics, ESP32 firmware in C++ on FreeRTOS (not Arduino sketches as the delivered stack), and — where you use ML — versioned data and evaluation. You integrate into the city’s shared firmware, make job requests and status available to full-stack classmates, and work through research, advice, design, and realisation for your Sprint 1-2 Learning Goal. Follow the city Requirements.

Your Learning Goal title in Portflow

Use: Sprint 1-2 - Learning Goal - Robotic System (adjust the sprint number if needed). See the Learning Goal page for the Portflow template.

Illustration

The animation below shows the idea in simplified form: an arm moves an item from bin A to bin B. Your delivered system must include multiple joints, an end-effector, sensing, a sequenced state machine, and at least one intelligent perception or learning component.

Challenge

Your Learning Group city needs a multi-joint pick-and-place robot on a city tile (or a clear bench stand-in tied to that tile). Objects move from a source to a destination under sequenced control, with sensing and safety limits. At least one AI/ML or intelligent perception component supports detection, pose, or grasp decisions. Your work is part of one shared embedded firmware for the city — not a separate personal project.

You will deliver a working prototype and the documents that justify and explain it. Work in this order:

Understand the problem. What is picked and where is it placed (e.g. waste from bin X to bin Y)? How many joints (at least 2–3 DOF)? What sensors or camera fixtures mark positions? What are the safety limits? Write clear rules, e.g. “Arm only moves when an object is confirmed at the source.” Agree which tile template you use and how this challenge fits the city plan.
Research. Compare realistic ways to drive joints (servos/steppers), sense objects/positions, and add intelligent perception (detection, pose, or grasp success). Include how you will expose job requests and status to full-stack. Cite datasheets and toolkit docs; plan evaluation against a baseline (e.g. open-loop timed sweep).
Advise. Recommend one architecture: MCU role in shared firmware, joint actuators, end-effector, sensors/camera, perception approach, safety limits, and the data/control interface for full-stack. State what you are not building in this period.
Design. Specify hardware (wiring, pin map, BOM) and software (simple kinematics or joint targets, state machine, perception module, interface messages) so someone else could rebuild your system and integrate it into the shared firmware.
Realise. Build and test in small steps. Prefer separate experiments (one joint, end-effector, sensor/camera, perception offline, then sequenced pick-and-place) before you merge working pieces into the shared city firmware on GitLab main. Keep each test tiny. Then demonstrate: pick from source → place at destination with sensing and safety limits. Use simulation or stubs if full-stack is not ready — do not wait on classmates to progress.

Minimum scope (Sprint 1–2)

Your contribution must include:

Ownership of one city tile based on a LightBurn template, with the robot placed on that tile (or a clear bench stand-in while the tile is cut). Prefer a tile with room for source and destination fixtures.
Integration into the city’s one shared embedded firmware (follow the city’s code conventions; do not ship a parallel personal firmware stack).
A microcontroller that runs that shared firmware with your robotic behaviour included as ESP32 C++ on FreeRTOS (ESP-IDF or PlatformIO + ESP-IDF; not Arduino as the delivered firmware). Use FreeRTOS tasks for motion, sensing, and safety monitoring.
A multi-joint arm with at least 2–3 DOF (kit servos/steppers are fine as a demonstrator). Document simple kinematics or joint-angle targets for pick and place poses.
An end-effector (gripper, electromagnet, or equivalent) that can hold and release the demo object.
Sensing of objects and/or positions (ultrasonic, IR, camera, buttons as fixtures — document which).
At least one intelligent perception or learning component (e.g. object detection, pose estimation, or grasp-success classification) — not only open-loop timed servo sweeps. If you train or fine-tune a model, version the dataset and report metrics.
Firmware with a state machine that sequences: idle → approach source → grasp → move → place → release → home (or your documented equivalent), with safety limits (joint limits, timeout, stop on fault).
A reproducible demo: pick from source → place at destination on the city setup.
A documented data and control interface for full-stack developers: job request (e.g. start pick-and-place), status (state, joint summary, perception result), and abort/stop (message names, payload fields, and an example). Full-stack may connect later — you still define and demo the interface, with stubs if needed.
Updates to the city’s shared BOM and wiring diagram for your hardware.

Recommended scope

If you have time, add one or more of:

Extra DOF or a wrist joint for nicer approach angles.
Vision-guided approach (camera crop → grasp offset) with measured success rate.
Serial debug of joint targets, state, and perception confidence.
A fixture plate so source/destination poses are repeatable in demos.
A stub or simulator feed that full-stack can use before hardware is on the desk.

Out of scope for this challenge

These are not part of this challenge:

Industrial six-axis cells, force-torque cobot certification, or factory MES integration.
Full inverse-kinematics solvers as a research thesis — simple documented targets or basic IK are enough.
Training huge foundation models from scratch when that is not feasible — small models / fine-tunes / classical+ML are in scope.
Building the city back-end, front-end, or Raspberry Pi stack yourself — that is full-stack work. You do define and expose the embedded interface they connect to.
A separate personal firmware repo that never merges into the city’s shared embedded application.
Delivering the assessed embedded path as Arduino .ino sketches — Advanced work uses ESP32 FreeRTOS in C++.

Demo and safety context

Treat the prototype as lab equipment with moving arms:

Default to stop / hold when sensing is unsure, a joint hits a limit, or a job is aborted; never thrash joints blindly.
Predictable rules. Document joint limits, timeouts, and when the gripper may close.
Clear demo script. Anyone in your Learning Group can show at least: object at source → pick → place at destination → home; plus abort/stop; plus a glance at perception status if used.
Independent progress. You can demo the arm and interface with simulation or stubs while classmates finish their parts.

Starting points

Use these concrete first steps early in Sprint 1:

Read the city Requirements and agree your challenge with the Learning Group (no duplicate challenges).
Pick your tile template and sketch source, destination, and arm base placement.
Agree on pick-and-place rules. What object, when may the arm move, what are joint limits? Write rules in plain language.
Inventory hardware. Servos/steppers, gripper, sensors/camera, MCU, power. Note gaps early and ask your coach. Start the city’s shared BOM. See Servo motor and Stepper motor.
Test in small experiments first. One joint sweep, gripper open/close, one sensor or camera crop, offline detection, then a short sequence. Do not develop everything only inside the city’s main shared firmware.
Confirm toolchain. Deliver on ESP32 with FreeRTOS in C++ (ESP-IDF, or PlatformIO using the ESP-IDF framework). Structure joints, gripper, and safety as FreeRTOS tasks — not Arduino .ino sketches as the assessed firmware. See RTOS application development and Operating systems and RTOS.
Draft the full-stack interface with your city (job request, status, abort). Agree a simulation path so neither side waits.
Draft your research question. Example: Which joint drive, sensing, and perception approach best fits a city-tile pick-and-place given torque, GPIO/power limits, and measurable place success?
Create your Learning Goal in Portflow. Link the product deliverables you plan to submit. Follow Learning Goal.
Integrate only what works. Prove joints, then grasp, then perception, then full sequence; move proven behaviour into the shared firmware in small merges.

Suggested sprint timeline

The Learning Group does not use Scrum, but you still follow the sprint rhythm of the Group Project: two sprints for this challenge (Sprint 1-2).

In the first sprint you focus on research and advice (plus early experiments: joints, gripper, sensing/perception proofs). In the second sprint you focus on design and realisation of the full advanced system: sequenced pick-and-place, safety limits, city integration, and the full-stack interface.

Sprint	Focus	Outcome
1	Research & advise (+ early experiments)	Research report; advisory report; tile and source/destination chosen; DOF and safety limits; draft job/status interface; joints and/or perception proven in small experiments
2	Design & realisation (full robotic system)	Wiring diagram + pin map + state machine; shared BOM updated; pick-and-place sequence in shared firmware on city GitLab `main`; intelligent perception in the loop; demo with safety stop; Portflow evidence linked; reflection

Tools, frameworks, and hardware

Area	Suggestions
City tile	One of the five LightBurn templates
Microcontroller	ESP32-S3 / ESP32-C3 with FreeRTOS + C++
Joints	Servos (e.g. SG90 / MG996) and/or steppers — ≥2–3 DOF
End-effector	Gripper, claw, or electromagnet module
Sensors / vision	Ultrasonic, IR, buttons as fixtures; camera + OpenCV / TFLite / Edge Impulse for intelligent perception
IDE / tooling	ESP-IDF or PlatformIO + ESP-IDF (FreeRTOS C++); not Arduino as the delivered stack
Version control	City GitLab project, shared firmware on `main`
Documentation	Markdown on the city MkDocs site and in the repo (English); cite sources; version datasets if ML
Full-stack link	Documented messages (Serial, MQTT, or city-agreed protocol); stubs/simulation so you do not block each other

Profile knowledge worth revisiting:

Products

Your challenge evidence is a set of products: studio deliverables tied to learning outcomes. Submit them in Portflow and link them from your Learning Goal.

Analyse & Advise

Product	What you deliver	Template
Research	Analysis of multi-joint pick-and-place, sensing, and intelligent perception for the city tile; comparison with criteria; evaluation plan vs baseline; recommendation for design	Research product
Advice	Decision document: MCU role in shared firmware, joint/end-effector choice, sensors/camera, perception approach, safety limits, full-stack job/status interface, and implementation order (joints → grasp → perception → sequence)	Advisory report

Suggested titles for Portflow

Research: Multi-joint pick-and-place sensing and perception for a city robotic system
Advice: Sequenced robotic arm architecture with intelligent perception

The advice report should build on the research report. Do not repeat the full comparison tables.

Design & Realise

Product pages for design and realisation are still being added to the studio catalogue. For this challenge, deliver the following and attach them as evidence (repository links, PDFs, or videos as your coach specifies).

Product	Learning outcome	What you deliver
System design	Design & Realise	Self-contained design document: context, city requirements link-in, block diagram for arm + sensing + perception, circuit/wiring diagram, pin assignment table, state machine for pick-and-place, simple kinematics or joint targets, safety limits, software module overview for the shared firmware, job/status interface for full-stack, contribution to the city BOM, and test/evaluation plan
Prototype realisation	Design & Realise	Working pick-and-place prototype on your city tile (or documented bench stand-in), code integrated in the city’s shared firmware on GitLab `main`, README with build/upload steps, interface demo (live or simulated), and validation evidence showing sequenced motion with intelligent perception and safety stop

Suggested titles for Portflow

Design: Robotic System — multi-joint hardware, perception, and firmware specification
Realisation: Robotic System — pick-and-place prototype and validation

System design — minimum contents

Your design document should let a peer rebuild and integrate the system without guessing:

Requirements. Source/destination rules, DOF, safety limits; how this fits the city Requirements.
Architecture diagram. MCU, joints, end-effector, sensors/camera, perception path, power, and the path to full-stack (protocol boundary).
State machine. Idle, approach, grasp, transfer, place, home, fault/abort — with timeouts.
Kinematics note. Joint angles or simple IK for pick and place poses; workspace sketch.
Timing table. Settling times, grip hold, maximum move duration before fault.
Pin map and wiring diagram. GPIO ↔ joint/sensor; include power distribution for servos/steppers.
BOM. Parts list aligned with the city’s shared BOM.
Perception design. What is detected/estimated; dataset version if trained; metrics vs baseline.
Full-stack interface. Job request, status, abort; example payloads; stub test path.
Shared firmware. Where your modules live in the city codebase and which code conventions you follow.
Test plan. e.g. “single joint limit”, “grasp only”, “full pick-and-place”, “abort mid-move”, “perception miss → safe hold”.

Realisation — minimum evidence

Small test experiments (or a short test log) showing you verified parts separately: joints, gripper, sensing/perception, then full sequence — before integration.
Work on the city GitLab project: proven behaviour merged (or ready to merge) into main as part of the shared firmware — not a dump of unfinished experiments into main.
Photo or diagram of the physical build on the tile (or bench stand-in), with source/destination labelled.
Demo script (numbered steps) matching your video or live assessment, covering pick-and-place, safety stop/abort, perception involvement, and the data/control interface.
Metrics vs baseline for the intelligent component (e.g. detection accuracy, grasp success rate).
Proof you can progress without waiting on full-stack (simulation, Serial demo, or stub consumer).
Honest known limitations (e.g. payload mass, backlash, lighting for vision).

Quality indicators

When assessing this challenge as a whole, the following indicators are used. Use them as a self-check before you submit.

Problem fit

The source/destination, DOF, and safety rules are described in plain language.
Scope is explicit: this challenge delivers a multi-joint pick-and-place system with intelligent perception, integrated into shared firmware — not only an open-loop servo wave demo.
The work respects the city Requirements (tile, shared firmware, BOM/wiring, full-stack interface, no waiting on classmates).

Analyse & Advise products

Research and advice reports meet the research and advisory report quality indicators.
The advice report references the research report and states one clear recommended architecture, including evaluation against a baseline and how jobs/status reach full-stack.
Component and approach choices are supported by datasheets or comparable evidence, not only opinion. Decisions cite resources.

Design

The design document is self-contained and includes a pick-and-place state machine, joint targets or simple kinematics, and safety limits that match the intended firmware behaviour.
Pin map, wiring diagram, and BOM match the realised prototype (or differences are documented with reason).
Fail-safe behaviour is designed explicitly (limits, timeouts, abort → stop/hold).
The job/status interface for full-stack is specified with example messages.
The intelligent perception component is designed with reproducible evaluation (dataset version / config where applicable).

Realisation

The prototype demonstrates ≥2–3 DOF, end-effector grasp/release, and source → destination place on the city setup.
At least one intelligent perception or learning component participates in the demo behaviour.
Behaviour lives in the city’s shared firmware on GitLab main (or is clearly opened as an integration MR), following city code conventions.
Firmware is readable: meaningful names, constants for joint limits/timings, no unexplained magic numbers.
The full-stack interface is demonstrable (live or with simulation/stubs) without depending on a finished back-end.
Validation evidence covers happy path, abort/stop, and at least one edge case (e.g. missing object, perception low confidence).
Repository README explains how to build, flash, and run the demo as part of the city project.

Professional practice

Learning Goal in Portflow uses the correct naming format and links to all product evidence.
End-of-sprint reflection connects results to your learning question and to robotics practice (actuation, sensing, safety, perception, city integration).
Documentation is in English Markdown and usable by other city students on their tiles.
You did not block on classmates: simulation, stubs, or a local demo kept you moving.

Focus points

Watch out for these common issues while you work:

Arduino as the delivered stack. Keep throwaway experiments short; the assessed ESP32 path is FreeRTOS + C++.
Open-loop only. Advanced means at least one intelligent perception or learning component in the behaviour — not only timed sweeps.
No safety limits. Always define and test joint limits, timeouts, and abort → stop.
Everything in main firmware. Develop and test parts in small experiments first; only integrate working pieces into the shared city firmware.
Underpowered servos. Plan power distribution; do not brown-out the MCU when several joints move.
Waiting on full-stack. Define the interface early and demo with Serial/stubs; do not pause realisation until the Pi dashboard exists.
Unreproducible ML. Version data and config; report metrics vs a baseline.
Solo firmware forever. Experiments are fine for learning; the assessed integration still joins the shared city firmware per the Requirements.
Advice without analysis. Coaches will send back an advice report that invents new comparisons not present in research.
Design–code drift. Update the design document when you change pins, poses, thresholds, or interface messages during realisation.

Key points

Core task: multi-joint pick-and-place from source to destination on your city tile, with sensing and intelligent perception.
Deliver embedded control as ESP32 FreeRTOS in C++, not Arduino.
Because this is Sprint 1-2, the delivered system is a sequenced robotic demonstrator with safety limits and evaluation — not only a servo wave.
Test small parts in separate experiments; integrate only what works into one shared firmware.
Expose job requests and status for full-stack; keep a shared BOM and wiring diagram.
Deliver four products: research, advice, system design, and prototype realisation.
Work research → advice → design → realise; Sprint 1 emphasises research & advice (and early proofs); Sprint 2 emphasises full sequence design & realisation.
Industrial cells and factory MES are out of scope; building the city back-end/front-end is out of scope for you.
Link everything to your Learning Goal in Portflow and reflect at the end of Sprint 1-2.