DevOps is a term that means different things to different companies or even different people and groups within the same company.

Are they the IT department, the build fixers, those cloud weirdos, the automation gurus? Some even argue whether DevOps is a team of people or a concept.

DevOps generally creates some confusion, but even more so in the embedded world.

In this article, we’ll look at what DevOps means to an embedded device development team and why the concepts still work to make your team and its products better by enhancing how you work.

Formal Definitions

Let’s start with a couple formal definitions of DevOps.

“DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.”

– AWS Definition

“DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while also delivering features, fixes, and updates frequently in close alignment with business objectives.”

– Wikipedia Definition

The Challenges of DevOps in an Embedded Development Team

Embedded software development presents a unique set of challenges to traditional DevOps. These challenges range from technical to workflow to culture.

Discontinuous Delivery

Starting with the backend of the DeOps workflow, the first challenge comprises 50% of the word itself — Ops. There is rarely standard “Ops,” i.e., a production server farm, cloud, or otherwise, that hosts the application code for thousands or millions of users that is constantly and continuously updated.

Embedded systems employ discontinuous delivery.

Many times, the firmware image, or images, is a line item on the manufacturing parts list. Like a resistor. It’s burned in at the factory, and that’s the end. Even with field-upgradeable units, the firmware upgrade is usually a pull from the user, not a push from the provider.

“A firmware upgrade exists. Would you like to install it?”

Discontinuous delivery requires a special emphasis on both configuration management of the build image(s) and integration/end-to-end testing of the marriage of the hardware device and the intended release code.

Continuous Integration and Automated Test

The test farm presents the next challenge.

Each Pull Request or merge to master creates the build package and then must push it to a physical device for test to implement continuous integration. Software emulation with a scalable, elastic farm of containers doesn’t exist with embedded systems. You gotta push that image into a physical device that occupies physical space in a rack, lab, or on a desk.

It has existential and geographical status. It’s finite. It’s got an IP address, or is physically attached to a host machine via USB. It’s non-elastic. And it’s custom.

The traditional DevOps community toolset doesn’t support this type of continuous integration. Therefore all tools and utilities are custom developed.

The lack of easy scaling requires an emphasis on automated and efficient test resource allocation and test time completion.

Which brings us to the build.

Build Toolchains

Cross-compile toolchains have been problematic from the dawn of embedded development because the target processor and OS (if one exists) are rarely the same as the host system. Embedded target builds are subject to the intricacies and nuances of the system on which it is built.

Long-time embedded developers have created a skillset around installing and maintaining these various toolchains purely out of necessity. But even with skilled maintainers of personal systems, every organization experiences the heterogeneous environment problems of “it won’t build for me” and “it won’t build for production.”

Maintaining and configuration managing the embedded build toolchain is an organizational and technical challenge throughout the industry, and its no different for us.

Security and Safety Regulations

Many embedded products are subject to a set of security and/or safety regulations based on their industry.

Safety regulations tend to require adherence to a backend process of formal review and test, sometimes externally. Security regulations tend to require adherence to protocols such as encryption, location (people, systems, tools, and data), and access control.

Any standard or regulation that adds and enforces external or artificial workflow protocols works against the core DevOps concept of speed.

Workflow and Culture

Although Agile and iterative development have made their way into the embedded world, the culture doesn’t automatically embrace modern software development concepts such as DevOps.

Embedded developers are experts in registers, the stack, memory maps, and know GDB commands by heart, but many don’t understand (trust?) the cloud, virtualization, and automation. Similarly, many web and application developers who work in the cloud all day do not understand what the heck is happening in the hardware (what hardware?).

Embedded developers are wilderness explorers, mountain bikers, self-sufficient, and tune their development machines as an Executive Chef cares for his knives. They are unimpressed by your fancy new processes and terminology.

In an embedded device company, effective DevOps requires marrying the two worlds — finding the nuggets in modern software methodology that provide value in the embedded space. This marriage necessitates change, and change is rarely welcomed. The challenge requires a combination of technical skills in DevOps and embedded development, people skills of influence and persuasion, and self-awareness to evaluate the usefulness of implemented changes continuously.

DevOps to Increase Speed, Quality, and Scale

DevOps aims to increase the speed and quality of deliveries and increase the organization’s scale through efficiencies and automation.

That’s still the goal of DevOps in an embedded software environment — make the company and its products better.

We do so by applying standard DevOps techniques to a non-standard application. We also develop and employ some ambitious proprietary and commercial-grade tools and workflows because of the specific challenges in the embedded space.

We have a philosophy, “make the right thing to do the easy thing to do.” We do so by marrying human psychology with modern DevOps.

What does Continuous Delivery Mean?

Even though the external delivery process is discontinuous, we employ the modern DevOps concept of CD internally. In our case, the delivery endpoint is the test farm rather than a customer or external entity. We treat each CI build as if it will be deliverable.

As such, the organization has the opportunity to decide if any particular build can be deployed to a customer, vendor, or partner. This decision process, sometimes fully automated, sometimes not, determines which release makes it to QA and ultimately externally, but is only possible because each build could be deliverable.

To make each build delivery-ready, we start the automated configuration management at the build itself.

Continuous Integration and Automated Test

To treat all CI builds as potentially deliverable, we must include a full “build => package => test” cycle. This requires efficient handling of cloud and physical resources to minimize time and increase speed. Therefore, we first decide what development activity receives CI. All commits? All branch activity? PR’s only?

The CI pipeline itself is built upon a cloud foundation that most DevOps engineers would find familiar. We utilize a standard toolset for an elastic and orchestrated farm of containers behind the standard CI executor.

The test farm contains an array of custom physical devices into which we must push the code. Pushing and testing occur almost continuously as per the CI decision. To support this workflow, we’ve built a generic automated test framework that not only manages the device under test (DUT) farm, but bridges the “Cloud to the Lab,” and does so in a secure environment. Build and packaging occur in the cloud, and then we push the code from the lab to the correct target systems.

The framework, which we call Cloud2Lab ATF, stands-in for the person in the lab. It takes the position of, “If I was a person performing manual testing, what would I do and how would I do it? What tools would I use?”

Build Configuration Management, Tracking and Recreating Previous Builds

Embedded software deliveries typically employ a “hardware-like” versioning and serialization process. The software package(s) is often given a part number and included in the Bill of Materials (BOM) and the Product Lifecycle Management (PLM) tool. Because, like hardware, most embedded firmware is static, and knowing who has what version and what that version includes is critical to a healthy operations workflow.

Traditionally, organizations use (semi)manual Configuration Management (CM) processes to keep this straight, and it must start at the build. The build manager manually creates a golden-standard build on the special build machine and packages it correctly for the intended target. Then the associated CM metadata such as commit ID’s, library versions, etc are added along with test results and documentation. This is, by definition, a process that disconnects the everyday work of develop, build, and test from the process of release and configuration management.

We’ve turned this on its head by ensuring that every build produced by our CI system is potentially deliverable. To do so, we have created a tool that automates the configuration management of the entire build and package workflow starting at the build and uses the production build environment (ie, containers) on the developers’ local systems as well as in the CI system.

Codenamed Fuze, this tool is a build executive that wraps and automates build execution, dependency management, release management, and all required metadata attachments into a “single source of truth” ID value.

A Fuze-built package is 100% traceable and reproducible on any system, not just the golden build machine, because all machines, even a developer laptop, can build for production. The included constituent parts and dependencies are also 100% traceable, allowing complete forensic analysis of deployed versions.

With a FuzeID, you know the following:

• The build – who, when, what tools, build command(s)
• Source commitIDs
• Dependencies/Libraries
• Build package contents
• Test results
• Release status – stage, by who, when
• Delivery status – to who, by who, when

Making the Right Thing the Easy Thing

If there was one single implementation principle that we employ to meet the goals of speed, quality, and scale, its “make the right thing the easy thing.” Whether we’re talking about people or systems, our goal is to focus on what is the right thing to do and attempt to make it the easy thing to do.

Most of the time, people choose the easy thing to do, regardless of whether it’s right. That’s not judgment; it’s reality. So we endeavor to marry the two. When successful, your development is happy, the development speed improves, the product quality improves at all times, and the organization can scale accordingly.

Summary

At 4TLAS, we specialize in implementing DevOps for embedded product development teams. We have developed and hardened over years of production, the EmbedScale pipeline, which includes both Fuze and Cloud2Lab. EmbedScale productizes the concepts discussed above.

You can hook into it, and get the benefits right away.

DevOps in an embedded device company is, at its core, simply a term that means enhancing what you make by enhancing how you work. That’s no different from DevOps in any company. Embedded systems provide some unique challenges, but DevOps can be applied with the proper mindset and skills and produce excellent results.

DevOps is like the grease in the system — making the right thing to do the easy thing to do.