Embedded software development is hard. The cost of mistakes is high, the skillset is specialized, the software industrial complex of open-source and commercial toolsets largely ignores it, yet it’s at the core of an entire industry that’s growing.

Although most of the DevOps world is aimed at the web and mobile industries, you can and should be integrating best practices from this world, where it makes sense. DevOps, and specifically Continuous Integration (CI) is one of those concepts that greatly improves quality, speed, and lowers the overall effort and cost of your development cycle.

The Importance of CI in Embedded Systems

DevOps engineers typically talk about CI/CD, where CD is “Continuous Delivery” or “Continuous Deployment.” Well, in embedded systems, we don’t typically enjoy the CD piece. In fact, we usually must manage Discontinuous Delivery.

But we can and should use the CI piece.

What is CI?

CI emphasizes integrating new source code often and continuously in small chunks. Each integration, ie, PR, merge to main, merge to release branch, etc, is then built, tested, and analyzed to ensure early detection of integration bugs. To be effective, this process must be 100% automated from start to finish, and the analysis of test results should have voting rights into the PR or merge mechanism.

The Value of CI

CI can dramatically improve embedded software development practices. Some of the key benefits CI bring to embedded developers include:

• Fix bugs quicker: The sooner you can pinpoint the commit that caused the bug, the sooner you can fix it.
• Faster time to market: You know the state of the codebase at all times. Can you take your current base and release it? Does it work for demo purposes? Can we send it to the certification test house?
• Improved quality: More build and testing, all the time.
• Enhanced collaboration: Better alignment across teams through more frequent integration.

How to Build an Automated Embedded CI Pipeline

A CI pipeline is a process that moves code from development, through test, into production-ready. An embedded CI pipeline automates the trigger for building, testing, and analyzing that commit. The developers use it. So does QA, system test, and product management. When done right, it executes fully automated and acts as another member of the team.

Core Operations of a CI Pipeline for Embedded Systems

• Trigger: What triggers the execution?
• Pull: The source code has to go somewhere to build it.
• Build: Build the firmware image(s).
• Analyze/Unit Test: [Optional]: Static analysis and unit/build test.
• Package: Grab the built image(s) and combine it with config files, other images, scripts, and anything else that’s required to be pushed to a fully functional device system.
• Automated Device Test: Push the image package to the devices in the test farm, initialize it all, and let it fly. No hands required.
• Analyze and Report Device Testing Results: Grab the device test results and analyze them for failures. Push the results back to pipeline for voting rights.

Core Components of a CI Pipeline for Embedded Systems

The core components for an embedded CI pipelines support the operations described above.

• Code Repository(s): GitHub, GitLab, BitBucket, etc
• Pipeline Execution Engine/Runner: Jenkins, or the pipeline/runner from your git provider
• Build Machine/Resource: Container, VM, or physical system(s) that contains the cross-compile toolchain and performs the build
• Static Analysis Tool: Clang, Coverity, etc
• Unit Test Tool: Unity, gtest, CppUnite, etc
• Configuration Management for Build and Packaging: Artifact repository plus an executive around it to manage the versioning of firmware packages. Likely homegrown.
• Automated Test Framework: A framework to perform and manage the automated testing across the DUT farm. Likely homegrown.
• Devices Under Test (DUT) Systems: Your devices, plus all of the supporting equipment necessary to program, control, execute, and acquire.
• Results Analysis and Storage Tools: Scripts and other applications for extracting data, saving results, analyzing, and determining pass/fail. Likely homegrown.

Building and Configuring the Embedded CI Pipeline

When implementing the embedded CI pipeline, you’ll need to make several decisions.

1. Trigger: What events should trigger your pipeline? Typical events are PR creation and update, merge to the/a main branch, release branch creation and update, and time-based triggers such as nightly and weekly. Triggering decisions are usually related to and driven by testing decisions due to time and resource constraints. How much testing should we execute for a PR? Ditto for a merge to main, etc.
2. Pipeline: You’ll want to either use the built-in pipeline runners associated with your git provider, or you can stand up an executive like Jenkins in your own infrastructure. Jenkins provides the ultimate in flexibility and customization over the built-in pipeline runners, but it also comes with the management and maintenance overhead.
3. Build: Do you build in the cloud or on a physical golden master server? Do you use a VM or a container? How do you configuration manage the build environment and toolchain? Do you maintain a set of build scripts colocated with the source code or independent?
4. Static Analysis and Unit Test: This step, if you use it, should be associated with the build, rather than the test stage. Does it provide value? If so, how do you interpret the output and configure the rules? What constitutes a violation? What tools do you use?
5. Configuration Management of the Artifacts: When you create the build and packaged artifacts, you’ll want to ensure you have complete traceability to them, and to the constituent parts that comprise them. The pipeline instance itself is ephemeral, therefore, you don’t want to rely on it for the storage of the artifacts. You can use artifact repository tools such as Artifactory to hold the binaries and scripting inside the pipeline to manage it.
6. Automated Test Framework: The most important aspect of your automated test framework is that it is truly 100% automated. No human required. This is a hard problem to solve because no open-source frameworks exist to serve the needs of hardware testing environments, so you’ll almost certainly need to develop it yourself. The automated test framework must command, control, and execute the testing, as well as acquire the results.
7. DUT Systems: The DUT systems are finite resources. They cost money, they take up space, and they’re messy. But they’re also the most important enabler of automated device testing, which is a critical piece to the CI pipeline. What other elements besides your product are necessary or valuable for automated test? How do you program your image into the device?
8. Results Analysis: Your automated testing will likely fall into one or both of two categories: regression testing and device characterization testing. The purpose of results analysis is to determine pass/fail and whether your release falls into the ranges given by the requirements. Python and other scripting languages are your friend here.
9. Merge or Release Decisions: The stages of your pipeline feed the merge and release decision. You must configure the pipeline such that results along the way either block or allow the execution to continue.

Implementing Embedded CI with 4TLAS EmbedScale

4TLAS’s EmbedScale pipeline is a done-for-you solution for incorporating CI into your embedded development workflow.

Why Choose EmbedScale for Embedded CI?

• It solves the build and package configuration management problem: The Fuze build executive inserts configuration management into the workflow right at the beginning. It manages the build environment and toolchain as well as all dependencies and source commitIDs. It keeps a record of everything related to build and allows you to easily and effortless know exactly what you have, what’s in it, and how to recreate it.
• We manage the pipeline. You focus on your code: Why distract your team with developing and managing all the bits of the pipeline? We’ll do that for you and you don’t have to worry or spend cycles on it.
• It solves the automated test framework problem: No open-source automated test frameworks exist for automating your device testing, but EmbedScale has one. The Cloud2Lab Automated Test Framework is a Configuration-as-Code solution that you can plug into your workflow and will 100% automate your device testing.
• It scales with you: As your needs change, EmbedScale can dynamically meet those needs.

Conclusion

CI can significantly enhance the efficiency and effectiveness of your embedded software development. CI practices can help embedded teams deliver high-quality software faster and more reliably. 4TLAS’s EmbedScale pipeline, you can implement the embedded CI model efficiently and quickly.