Embedded firmware development for fleets

A scooter in Lisbon reboots twice an hour. A cargo bike in Berlin bricks itself after an OTA. A tracker in a shipping container drains its battery in three days instead of thirty. None of these bugs showed up in the lab. All of them showed up after the tenth thousand unit shipped.

That gap between a working prototype and a fleet that behaves in production is where embedded firmware development for connected devices actually lives. Micro-mobility and IoT teams tend to underestimate it because bring-up looks deceptively similar to the rest of the software stack. It is not.

What is the difference between firmware and embedded software development?

Firmware is the code that runs directly on a microcontroller or SoC with tight coupling to hardware peripherals, power states, and boot sequences. Embedded software development is the broader discipline that includes firmware but also higher level application logic running on embedded Linux, RTOS tasks, and the tooling around them. In practice, firmware development is the layer where a wrong register write can brick a device, and embedded software is where a wrong pointer crashes a process you can restart.

For fleet devices, the distinction matters because the failure blast radius is different. Firmware bugs mean field returns. Application bugs mean a bad afternoon.

What embedded firmware development actually means for connected fleet devices

A connected scooter, e-bike, or asset tracker is a distributed system pretending to be a single product. There is firmware on the main MCU, firmware on the BMS, firmware on the modem, and often firmware on the motor controller. Each has its own update cadence, its own boot loader, and its own way of failing.

The job of senior embedded firmware development in this context is not writing drivers. It is making sure the interactions between those subsystems remain debuggable when a unit is sitting in a rental yard two thousand kilometres away.

That means designing for observability from day one: structured logging over the cellular link, a watchdog strategy that distinguishes hangs from crashes, and a bootloader that can recover from a bad update without a technician.

Firmware vs embedded software: where the line matters at scale

At one device, the line is philosophical. At ten thousand devices, it is operational. Firmware updates are risky, slow to roll back, and expensive when they fail. Application layer updates on an embedded Linux gateway can be canary rolled and reverted in minutes.

A good architecture pushes as much logic as possible up the stack, keeping the firmware layer thin, boring, and stable. The MCU should do the things only it can do: real time control, power management, safety interlocks. Everything else belongs where you can iterate on it.

This is one of the first calls we make on any remote development team engagement in this space. Get the boundary wrong and you spend the next two years shipping firmware releases every sprint.

Board bring-up: the first 90 days of any IoT firmware project

Bring-up is where senior engineers earn their keep. The first three months of an IoT firmware development project set the ceiling for everything that follows.

Week 1 to 3: the boring foundations

• Toolchain locked, reproducible builds, no developer machine specific artifacts.
• JTAG or SWD access documented, not tribal knowledge.
• Power rails measured, not assumed from the schematic.
• A minimal bootloader that can flash itself from a known good source.

Week 4 to 8: peripherals and power

Bring up peripherals one at a time with characterisation, not just a working blink. Measure current draw in every power state. Micro-mobility devices live and die by battery behaviour, and you cannot fix in software what you did not measure in bring-up.

Week 9 to 12: the connectivity stack

Modem integration, TLS, MQTT or whatever your backend expects, and a first pass at the OTA pipeline. If OTA is not working by day 90, the project is already in trouble.

Designing OTA updates that survive 10,000 devices in the field

Over the air updates are the single highest leverage part of any embedded firmware development effort for a fleet. They are also the most commonly underbuilt.

The minimum viable OTA that actually survives contact with a real fleet has five properties:

1. A/B partitions with atomic switch and automatic rollback on boot failure.
2. Signed images verified by the bootloader, not the application.
3. Staged rollout with cohorts, not all-or-nothing pushes.
4. Health checks that gate promotion, measured after the update, not just at boot.
5. A kill switch that can pause a rollout within minutes when telemetry looks wrong.

The ESP-IDF, Zephyr, and Mender documentation all cover the mechanics well. The harder part is the operational discipline: who approves a rollout, who watches the dashboards for the next 24 hours, and what the rollback criteria are in writing before the button is pressed.

Failure modes you only see at fleet scale

Some bugs are statistical. They do not exist at one device or one hundred. They emerge at ten thousand, and they emerge in ways that look nothing like the bug report.

Clock drift induced TLS failures. An RTC that drifts by a few seconds a day will, after months, drift past certificate validity windows on a subset of devices. The failure looks like a connectivity issue. It is not.

Flash wear on write heavy partitions. A logging strategy that seemed fine in QA will chew through flash endurance in fleet units that stay powered for months. You find out when devices start refusing to boot.

Modem firmware regressions from the carrier. The modem is running its own firmware, updated by the carrier, on a schedule you do not control. Fleet-wide connectivity dips that correlate with nothing in your code are often this.

Thermal edge cases. A device shipped to Oslo and a device shipped to Seville see different thermal envelopes. Bugs that hide behind temperature dependent timing show up in one geography and not the other.

The common thread: instrument for these before you ship. Structured telemetry with timestamps, reset reasons, flash health, modem firmware version, and temperature is not optional at fleet scale. It is the difference between debugging and guessing.

Custom firmware development for micro-mobility and IoT: a senior engineer's playbook

The teams that ship reliably in this space share a few habits. If you are staffing a firmware effort, these are worth using as a checklist:

• One engineer owns the bootloader and OTA path end to end. Not a committee.
• Every firmware release has a written rollback plan before it ships.
• Telemetry schema is versioned and reviewed like an API, because it is one.
• Hardware, firmware, and cloud teams share a single incident channel.
• The definition of done for any feature includes what it looks like in the fleet dashboard.

This is the shape of the work when Devspace embeds senior embedded engineers into micro-mobility and IoT teams. The engagements that go well are the ones where the client already understands that firmware is a long game, and wants senior capacity to run it properly rather than a body to write drivers.

The embedded development toolchain we reach for

Stack choices vary by silicon, but a few tools show up in almost every engagement: GCC or LLVM based cross toolchains pinned in CI, Zephyr or FreeRTOS depending on complexity, PlatformIO or CMake for build orchestration, Renode for hardware in the loop simulation where the target is expensive or scarce, and Grafana fronted telemetry for fleet visibility.

The toolchain matters less than the discipline of pinning it, reproducing it in CI, and treating the firmware build as a first class artifact with the same rigour as any backend service.

Embedded firmware development for fleets is not glamorous work. Done well, it is invisible. That is the point.