Hardtech embedded systems pose a challenge that modern web and mobile apps don’t experience — the matrix of firmware versions across the product versions and families.
Discontinuous delivery makes managing multiple firmware versions across diverse product lines, hardware revisions, and customer deployments a persistent challenge in the embedded systems industry. This delicate version matrix can quickly spiral into a complex web, making it difficult to track, reproduce, and maintain firmware builds internally, through the test and cert process, and in the field.
Lifecycle Version Management
One pain point arises when managing firmware deployments in the field. With numerous products and hardware revisions, each potentially requiring slightly different firmware versions, keeping track of who has what and who gets what becomes a logistical nightmare.
Bug Fix Point Releases
Another challenge is the need to recreate a specific firmware version from months or years ago, often required for point releases or bug fixes. The certification house or customers may request an update to address a particular issue, but they cannot adopt the HEAD of the codebase due to certification and testing requirements. Recreating the exact build environment, toolchain, and dependencies from that specific point in time can be a daunting task, leading to wasted effort and potential inconsistencies.
Multiple Processors and Firmware Images in the System
The problem is further compounded when dealing with systems that have multiple processors, each requiring its own firmware image, multiplying the complexity of the version matrix exponentially.
Taming the Version Matrix
Historically, successful hardtech teams and companies use configuration management procedures and tools to handle this version matrix. Build and product managers use tools like Jira to create a manifest and traceability throughout the matrix, rely on the source code and artifact repositories for commit IDs and retrieval, and use golden build servers (or VMs) for production builds.
When done right, this system works, but requires people, procedures, time, and attention to detail. It’s fraught with opportunities for error and holes. If you’ve been in the embedded industry for a while, you probably have some of your own horror stories about version matrix hell.
The good news is that we can borrow some ideas from the modern DevOps practices of the web and mobile application worlds and then reimagine and reapply them to deal with the version matrix problem of hardtech.
Traceability Tools
The no-brainer. Ticketing tools such as Jira and source code control tools like git provide traceability in your process and source code.
Automation
Automate the build itself and attach it to the development workflow. Source code management tools like GitHub, GitLab, and BitBucket allow you to build pipelines around triggers such as PR’s, branch commits, and merges to the main branch. Automating the builds allows you to easily build the images for the entire system, including multiple processor systems. It provides a snapshot, possibly a release snapshot, of build artifacts across the system. The set of commit ID’s associated with that build are important source material for the version matrix.
Suppose libraries or prebuilt packages need to be part of the link or release packaging. In that case, you can use automation to pull the correct versions of these from the proper locations and make them available to your build workflow.
With the traceability tools described above, you can build automation into your workflow to auto-generate and populate as much of the build manifest as possible into a Jira ticket or configuration file.
More automation equals less chance for error and less time to complete.
Containers
Web and mobile application teams use containers for just about everything in their workflow, including the test and operations sides. For hardtech, we can’t use containers for those purposes because we need Harware-in-the-Loop, but we can reimagine and reapply containers for the production build toolchain.
This provides the dual benefit of making the production build machine available to the developers for local builds and allowing build managers to put the container or its Dockerfile under configuration management. Then the build machine and its toolchain can be part of the version matrix helping to ensure you can reproduce old versions in the future.
Delivery Tracking
The web application world uses continuous delivery to an elastic sea of servers and containers. It’s constantly pushing, controlling the scale, and managing rollback and regional updates. It’s in charge.
The discontinuous delivery of hardtech embedded firmware requires a different approach to delivery tracking. First, delivery is a pull. Even if the product development team initiates the new delivery, the manufacturer, test house, or end-user must accept it. This further complicates the version matrix by creating a situation where the field deployed hardware may currently be using a variety of versions.
It’s critical to know what versions of firmware are compatible with what versions of hardware and what has been delivered to whom.
The Single Source of Truth
At the end of the day, you need a single source of truth. Whether you call it a manifest, a release ticket, or a version document, you need to be able to point to a document, file, or database entry that contains everything that describes all of the version and dependency information for a build, it’s package, and any associated release information.
The single source of truth ensures that for any build or release package, whether its for test, certification, or already out in the field, you know exactly what went into it and how to recreate it.
How We’ve Solved the Complex Version Matrix Problem
Our solution, which we call Fuze, combines the DevOps concepts from above with a purpose-built application to create a universal automated build and configuration management executive that streamlines the entire firmware development, build, packaging, release, and delivery process. We build configuration management into the system at the very beginning — the build.
For each build (whether local or in the cloud), we generate the single source of truth, a unique ID value called the FuzeID, that now lives forever and contains all of the metadata required to reproduce it. If that build moves forward through test, release, and delivery, Fuze attaches all of that metadata as well.
With a FuzeID, you know the following:
- The build – who, when, what tools, build command(s)
- Source commitIDs for all repositories
- Pre-built and library dependencies
- Build package contents
- Test results
- Release status – stage, by who, when
- Delivery status – to who, by who, when
For systems with multiple processors and firmware images, Fuze’s hierarchical packaging capabilities can handle the intricate matrix of builds and dependencies with the same automated configuration management, ensuring that you know everything about the release package.
Conclusion
By embracing some modern DevOps concepts, and understanding where current DevOps tools leave gaps, embedded systems teams can effectively tame the complex version matrix problem. They can scale operations to meet growing demands, enhance quality and speed through automated CI/CD pipelines, and redirect their focus towards innovation and complex problem-solving, leaving the repetitive tasks of build, packaging, releasing, and delivering to the automated pipeline.
In the ever-evolving landscape of embedded systems, where complexity is the norm, embracing modern DevOps practices requires some reimagining and reapplication of the current toolset as well as adding your own secret sauce. Solutions like we’ve built with Fuze pave the way for embedded teams to conquer the version matrix challenge, ensuring reliable and market-ready products while fostering a culture of innovation and efficiency.