How good is the software?

I get pleasure from good quality things. Quality is a vague term, but, to me, it is some combination of good design for usability, functionality and aesthetics, along with reliability and durability. Some of these factors can be assessed very quickly; others take time. For example, you need to live in a house for quite a while – many months at least – before you can be sure that it is well designed; assessing its durability may take decades. You can quickly decide whether a car looks nice and drives well and soon decide whether it is economic on fuel; its reliability becomes apparent over a much longer time. Software can be the most difficult to assess …

Using some software can quickly show whether is has the functionality that you are after [though some software can make features hard to find]. At the same time, you find out how easy or enjoyable it is to use. Its reliability is only proven over time. It is a great disappointment when some software, that you really liked and seemed a great fit to your needs, suddenly loses some data or misbehaves in some other way.

It seems to me that, for the most part, we should be able to rely on manufacturers and vendors to assess the reliability of their products and show us the proof of that. Up to a point with cars and houses this is possible. Software is more difficult. It can be thought of as a machine, where every bit is a moving part. How reliable would you expect a machine with a million moving parts to be? A developer can take steps to maximize reliability:

• Develop the code using recognized methodologies that encourage reliable code [MISRA C is an example that interests me].
• Include self-testing functionality. This is commonly done to verify that the hardware is all working, but software self-checking is valuable. Monitoring of stack boundaries, for example.
• Design comprehensive testing procedures to exercise every possible part of the software’s functionality.

The last of these is tough. I have been pondering what it means to suppliers of embedded software tools and runtime components:

• An assembler has a very specific range of functionality and a testing procedure may be developed that exercises it to almost 100%. Macro programming is the only area where testing is necessarily limited.
• Linkers are more flexible; testing can stretch them in many ways, but there remains some possibility for unforeseen “corner cases”.
• A compiler is another matter. There are numerous test suites available, but there are an infinite number of possible programs that may be written and, hence, the same number of opportunities for finding a flaw in a compiler.
• The kernel of a real-time operating system [RTOS] can have its basic functionality tested to something like 100%, but there is also stress testing and performance measurement to consider.
• Other components of an RTOS, like networking, are readily testable in well-recognized ways using industry-standard test suites.

Many vendors rely on beta testing – having customers do the final test out in the “real world”. This can be very effective, but needs careful management or it can be damaging to the vendor’s reputation. The customers must be completely aware of the beta status of the software and what that implies.

Other companies are more creative. One of the first RTOS products on the market was VRTX from Hunter & Ready [later Ready Systems, who were acquired by Microtec Research, who were acquired by Mentor Graphics, who were acquired by Siemens …]. They were very confident about their product being bug free. They ran an advertising campaign called “A Bug for a Bug”, where they offered a VW Beetle car as a reward for finding a bug in the kernel. I have no idea whether there were lots of strings attached or if they actually ever gave away a car, but it was a very memorable program.

Being realistic, the very best thing that a software vendor can do is ensure that the customer, who finds a problem, can get help. Technical Support is seen as an unglamorous part of a business, but its quality is critical. Years back, there was a compiler vendor [I will not name them – they were later acquired by a semiconductor company, so never heard from again] who offered quite good quality documentation. At the bottom of many pages were Latin phrases praising the glory of God. I have total respect for other people’s religious beliefs, even if I do not share them. But this is not the business of software manuals. We always used to say “They have faith; we have Technical Support.”