Automated testing is the backbone of Continuous Integration, the centerpiece of modern software development and DevOps. In the embedded systems world, discontinuous delivery forces a rigorous emphasis on pre-release testing.Even more so than in the Continuous Delivery world of web applications.
What’s the cost of a mistake? You get one shot to make the release. It better be right.
The good news is that if you apply the right workflow and toolset, you can do everything in your power to test the heck out of your release with automation, leaving the development and test teams to focus on what they get paid to do – product development and test development.
The Two Levels of Testing
Whether you develop for the web or for embedded systems, you’re familiar with the two levels of testing: unit testing and End-to-End testing.
Unit Testing
Unit testing is common between web/mobile applications and embedded applications. You employ unit test, right?
The frameworks may be different, but the purpose is the same. Test the API’s and functions underneath to ensure that they don’t fall over at the basic level. Unit testing is more associated with the integrity of the build itself, rather than the system’s functionality.
In embedded systems, you can use frameworks such CppUTest or Google Test.
End-to-End / Integration / System Testing
True functional testing of the system and all its components and interfaces.
This is where the embedded and web/mobile worlds diverge in toolsets and industry support. End-to-end testing for web and mobile applications often revolves around UI interactions, employing tools like Selenium to automate user actions and elastic cloud instances to scale test volume and speed.
In embedded systems, this level of testing is far more complex due to the hardware-software interplay, requiring custom test rigs, specialized equipment, and custom software tools. But this level of testing is also absolutely critical to the success of your product. You have to know that when you release it, it’s going to work.
The Challenges in Automating Embedded End-to-End Testing
True end-to-end testing of embedded systems requires Hardware-in-the-Loop (HIL), which presents a number ofchallenges for automated testing.
Geography
Your devices under test (DUTs) live in the physical world. They exist on a desk, bench, or rack in some physical location in an office or lab. No matter where you build your firmware image(s), whether locally, on a centralized physical server, or in the cloud, you gotta get that image pushed into a device that has ontological status in the physical world. How do you bridge from where your build took place to the physical geography of the DUTs?
Finite Resources
Hardware is limited. You can’t elasticize your DUT farm. You may only have one or a few devices because of cost or other factors that affect availability. Therefore, how do you best utilize these resources to avoid slowing down the development team or holding up a release?
Heterogeneous DUT Farm
You probably have a bunch of different products, or several versions of the same product, or both. Each of these may or may not get the same build or firmware image. You’ll need to manage both the matching of build to DUT type and the matching of test content to DUT type. How do you do this in automation?
Command, Control, and Acquisition
Your test system might have debuggers, USB switches, power supplies, card readers, scopes, logic analyzers, robotic mechanisms, rPi controllers, and a plethora of other physical equipment or host applications required to exercise your system and acquire the necessary data and results. How do you utilize all of this in a 100% hands-off environment?
Reproducibility
Ensuring that test results are consistent and repeatable across multiple test runs and devices is essential for reliable validation. But end-to-end testing with HIL is messy. DUT states, test system states, and environmental factors all work against reproducible results. How will you control the environment to ensure that subsequent runs produce the same results?
Infrastructure Reliability
Automating a plethora of external equipment and applications introduces risk of test failures outside of the DUT and its test content. How will you keep the team from having to triage each and every failure to determine if its a “real” failure or an infrastructure failure?
How to Fully Automate Your End-to-End Testing with Embedded Systems
Luckily, there are some things you can do to fully automate your end-to-end testing with embedded systems.
A Framework – The Person in the Lab
Start with the premise: what does the person do?
How do they interact with a device under test? What applications and tools do they use? What commands do they type? How do they interpret the results? Where do they store the results? How do they know which test content to execute on which DUT?
The person likely interacts with the DUT like this:
Make your automated test framework do that same thing. A similar concept to shell scripting.
Your QA or system tester already has tools, workflows, and methods. Your development team has already built tools to help them develop and test. White box and black box. Don’t reinvent the wheel. This stuff works. Use it.
Not only does that approach reduce further development and eliminate interfaces and APIs prone to errors, but it also allows you to integrate all of the additional equipment needed by embedded systems testers, such as robot mechanisms, control devices, debuggers, and acquisition devices.
When you architect and build your automation system to stand-in for the person in the lab, you minimize the extra work of developing and debugging non-essential systems, and maximize the reuse of the knowledge, workflow, and tools that already exist for manual testing.
Ideally, your automated test framework is a stand-in for the person in the person in the lab.
Tiered CI Test Workflow
Design a tiered automated test workflow for CI. CI testing aims to quickly find and squash bugs that might make it into the main (or release) branch. The quicker you spot an issue, the easier for development to pinpoint the bug.
But that doesn’t mean you should run 100% of the tests in all scenarios. If every PR requires 24 hours of testing before you can merge, your development team will have a lot of time to scroll through TikTok. You can be smart about where you take on risk without slowing down the development team.
Make a technical assessment and ask yourself the following types of questions:
- What set of tests can we run in less than 10 minutes to give us at least an 80% confidence that this PR is good?
- What set of tests can we run in less than 30 minutes when we merge to the main branch to give us at least 95% confidence that the merge didn’t break the main branch?
- What set of tests can we run overnight in a few hours that gives us 100% confidence in the integrity of the main branch?
- What set of tests should we run over the weekend for 24 hours or more that gives us confidence in the reliability of our main branch?
- What set of tests must be run on each release?
DevOps
Although the universe of open-source and commercial toolsets for implementing DevOps is limited for some of the specific challenges of embedded systems development, you can and should still use the concepts.
Source code control for everything, including test content, control scripting, and results analysis. Keep the test content colocated with the firmware source code. Update test content along with firmware updates because these are tightly coupled. Separate the control scripting and results analysis from the firmware source code because these are loosely coupled.
Use pipelines to initiate automated test runs and gather and analyze results. Use the pipeline reporting mechanisms to indicate pass or fail and tie this status into the overall PR, merge build, or release status. Give the test results the power they deserve in the workflow.
Build self-monitoring into your physical infrastructure and hook it into notification channels such as Slack/Teams or even text messages. Use time-based and event-based mechanisms to perform hardware resets and reboots.
EmbedScale with Cloud2Lab Automated Test Framework
At 4TLAS, we’ve taken our years of experience in embedded systems development, automation, testing, and DevOps skills and built a system that helps you automate your end-to-end testing by addressing the areas described above. The EmbedScale DevOps pipeline offers a solution for automating end-to-end testing in embedded systems through the Cloud2Lab automated test framework. Cloud2Lab extends the capabilities of the EmbedScale pipeline by integrating an enterprise-level framework for automating testing activities across a heterogeneous farm of virtual and physical devices. This includes managing and deploying test content, coordinating and parallelizing test case execution, infrastructure monitoring, and publishing results to ensure products meet and exceed quality standards.
By bridging the gap between cloud-based development and lab-based testing, Cloud2Lab facilitates a fully automated CI pipeline. Cloud2Lab allows for scalable and tiered test automation, managing a diverse range of devices and versions with ease, and acts as the Person-in-the-Lab. This approach minimizes the need for custom code, instead relying on configuration, thus enabling a seamless transition to automated testing for embedded systems.
Conclusion
Automated testing in embedded systems presents unique challenges due to hardware dependencies, real-time constraints, and specialized environments. But that doesn’t mean you can’t move towards fully automated end-to-end testing. By following a few key architectural and workflow guidelines, you can move more and more HIL testing from manual to automated. More testing means less risk. To achieve more testing, put the machines to work and keep the people focused on what they do best and where they add the most value.
With the introduction of solutions like 4TLAS’s EmbedScale pipeline with Cloud2Lab, you can quickly and effortlessly overcome the hurdles of rolling your own. The future of automated testing with embedded systems lies in adopting these advanced solutions, ensuring efficiency, scalability, and unparalleled quality.