I’ve had the opportunity to work with all sorts of products — both software and physical. Today, I want to take you on a journey through the world of physical product testing, offering insights and methods that are different from those commonly associated with software testing.

Let me start by telling you a story about James. James is a software quality professional who also enjoys videography. Due to his passion for capturing moments, he decided to invest in a $4,000 Canon R6 MKII mirrorless camera. Now, what might be his expectations for this significant investment? You might think of flawless image quality, durability, ready-to-use out-of-the-box functionality, appropriate accessories, and user-friendly ergonomics. James’s camera will also need to comply with a range of regulations, which is an essential part of product development.

The development of a physical product such as a camera involves cross-functional engineering spanning optics, electronics, software, and mechanics.

There are three questions to answer when thinking about physical product testing:

What could go wrong?

What needs to be tested?

How can we ensure product quality?

The most important thing to test comes right at the start of any physical product development, which is to test for product-market fit. While “product-market fit” may not fit into the scope of generally what is considered “testing” it is still testing.

Developing something that fulfils consumer needs is the top priority. Testing your product concept with potential buyers to see if they’re willing to pay for it is the most crucial step, and a step can happen before any product is made. This ‘consumer testing’ might not fit the traditional idea of testing, but it may make or break your company if it is wrong.

There are many areas of engineering in physical product testing. This blog post will just skim over the top of these areas to give you a taste of what testing looks like in these areas.

Architecture

Would you consider architectural and design reviews part of the normal software testing activities? In the hardware world, it’s very much part of testing and how to actually get quality in at the early stages.

As with all projects, the later changes are made, the more expensive they tend to be because changes can have a lot of knock-on effects. This is doubly so with physical products because making changes can require new physical prototypes to be made which is both super costly, time time-consuming.

To save as much time and money as possible, getting it right from the outset is the cheapest path to take, and the best shot at that is by careful design and review.

The cost of change

Mechanical

Usually one of the first stages of physical product design is to start with mechanical. Starting with mechanical has the advantage of having a thing that can look and feel like the finished product. This is super handy when doing market testing.

CAD software is fantastic these days for being able to simulate products while they are still in the digital realm. Mechanical properties can be tested and designs can be inspected for manufacturability and component placement. Even draft PCBs can be imported into the software to make sure holes line up and nothing interferes.

The other part of mechanical testing is tactile. Seeing and handling the product allows for design improvements to be intuitively understood. 3D printing helps a lot with making prototypes. 3D prints can tell you a lot about a design and allow for some hands-on testing early on in the development process.

Sometimes mechanical tolerances need to be tight. I once worked on a project that required extreme precision. The machined part prototype was fine but when it was power coated, the microns-thick coating through the dimensions off.

Electronics

Electronic design is typically in two stages: Schematic design, and routing. It can sometimes include a third stage, simulation.

Schematic design is where the components are chosen and “plumbed” together. You can think of an Integrated Circuit (black square boxes you see on circuit boards) as a library that does something specific. ICs take certain inputs and provide certain outputs, but has to be used as the designer intended. There are typically a lot of calculations that need to be done at the schematic stage to ensure limits and thresholds are not exceeded.

Testing a schematic design is also mostly done by design review, where someone else checks the calculations and choice of components.

Electronics

Once schematics have been reviewed, it is onto the routing which is essentially connecting up the wires. A typical PCB board can be up to 6 layers thick and have traces (wires) running through all 6 layers. Traces can’t just go anywhere, rules need to be followed to ensure proper operation of the design. A design review is also conducted on the routing.

Routing

Once the PCB is sent to the PCB manufacturer the engineers at that company will also check the design for manufacturability and ensure all tolerances for the manufacturing process are met.

This is a design review looks like for a PCB

Design Review of a PCB

Once a PCB is sent to the manufacturer for assembly, that manufacturer will also do a review of the bill of materials to ensure that the components can be sourced and to identify alternative parts where required.

Typically for prototypes, the PCB has to be manufactured by hand using a pair of tweezers and a magnifying glass.

There is a lot of room for human error.

This is what a development cycle looks like for electronics.

Bring-up

Bring up is like software debugging on steroids. During bring up engineers are trying to power the PCB on for the first time. Some parts of the board can be tested using multimeters and oscilloscopes but others require functioning firmware before they will power and perform anything.

During bring up engineers are trying to

  • Validate that the firmware code is correct while
  • Testing it on a board that probably had manufacturing defects and is
  • Using a design that has not been validated

Needless to say, the bring-up stage can be difficult. Often soldering rework is required, while not damaging any of the other components on the board.

Once a full bring-up has been achieved, the development of functionality can be started.

Firmware

So, let’s jump into firmware. If you thought that was hard, let’s start on firmware. So, a regular dev loop for firmware is to change some code. Then compile it, which will create a binary file. Next, the file needs to be programmed onto the device using a dedicated programmer. Once the device has rebooted, the developer needs to get the device into the state ready to test the functionality of interest, which may not be an easy thing to do. For instance, a camera might need to take a photo under specific conditions. Once the test has been performed, go around the loop again.

Much of this testing is manual. Automating this type of thing is super hard to do and creating an automation rig is a a project in and of itself.

Firmware development cycle

Another challenge with firmware development is that there are few debug tools. “print” statements are the go-to tool, but not too many, because too much logging can max out the device and mess with the main program.

A lot of firmware is written in a language called C. C is not a language that was built with unit testing in mind. It’s incredibly hard to write unit tests in C. One of the other challenges with C is there isn’t any package manager, such as npm or pip. Finding snippets of code on the internet is usually the main source of “library” code. C is also dangerous because it lacks memory management features. It is easy to create buffer overruns or memory leaks which higher-level languages don’t let happen.

Lessons from firmware: Unit test what you can as that is much easier than end-to-end testing and (for the software folk) be thankful you work in software.

Software

Some hardware needs to talk to other devices, which might be a phone app, desktop app, or might be a cloud server, or the browser. In the web software world, REST and HTTP are used most of the time. In the hardware world, there are many more options with their own trade-offs. For example Serial UART, USB (many versions) Bluetooth (many versions), Wi-Fi (many versions), Cellular (many versions), LPWAN (many versions). Each with a unique set of trade-offs and limitations.

All of the regular software development practices apply to this area of product development, except that talking to low-level protocols and working at the binary level is the norm, not the exception.

One of the other uses for software is to create test rigs for the PCB. These are complex pieces of software and equipment in their own right but are designed to program and test each board as it comes off the production line.

Production

Unlike software, which can be replicated exactly every time, each physical product is slightly unique and therefore needs to be tested independently.

Production testing starts at the pick and place machine. Sometimes components are mis-labelled from the manufacturer and this mis-labelling can have a massive impact on the finished product and may result in thousands of boards needing to be reworked.

Pick and place machines can automatically test some components to ensure that they are what they are supposed to be.

PCB manufacturers can also automatically inspect boards using cameras and X-ray machines to check for soldering defects.

Finally, the boards can go through a programming and functionality testing process.

There is much more that could be said about physical product testing but I’ll leave it there for now.

Back to James

So, coming back to James. James now has a new appreciation for engineering and the countless tests that went into developing his camera. As per his expectation, the camera captures video flawlessly. But, true story, something got missed during testing. Something that made James’s $4, 000 investment a waste of money.

The R6 MKII has a habit of overheating after about 5 minutes of 4K video capture. Once overheated it takes 10 minutes to cool off before you can power it back on again. makes the camera useless for capturing wedding ceremonies because it keeps shutting down. James only found this out after his first commission for the new camera.

Needless to say, the bride and groom were not happy.