Producing a quality product is what every product and engineering org strives for, but what does that look like in practice? What are the important aspects to track when holding ourselves accountable to our users. As a product led company with a goal to enable better quality practices across a company, we hold this topic close to our heart.
We approach evaluating quality with a two-step process:
- Define quality for our specific application: Software quality is a web of different characteristics related to a software’s design and execution, so pinpointing and prioritizing certain aspect is crucial for bringing clarity to a team during development.
- Measure quality thoroughly and consistently: There are a countless number of ways to measure different aspects of software quality, but the primary goal of the software – providing users a useful tool – should be kept in mind when deciding how to measure the quality of your software.
Functional and Non-Functional Quality
There are two main buckets from which we can begin to define quality, functional and non-functional:
- Functional Requirements are about having a product that will provide users value when it works as intended: if everything is working as it should, what value does the software provide? What are its features and functions?
- Non-functional Requirements relate to how the software works: how often does everything work as it should, and what kinds of things cause it to fail?
Example: If you’re building a SAAS application, you need to make sure that your app can facilitate the tasks your customers might need to complete using your app. Functional characteristics cover how well your app’s functions provide value when they are working correctly. Non-functional requirements cover how well the functions get to shine through – do bugs or a confusing interface prevent users from getting the most out of the app’s functions?
The International Organization for Standardization’s ISO/IEC 25010 software quality model gives a hierarchy of eight characteristics of software quality. The first, Functional Stability, is the only of these characteristics that describes a functional requirement.
Functional Stability describes how well a product’s functions match users’ needs. Does it accurately do all the things a user needs it to do? This can be further broken down into three parts:
- Functional completeness: Degree to which the set of functions covers all the specified tasks and user objectives.
- Functional correctness: Degree to which a product or system provides the correct results with the needed degree of precision.
- Functional appropriateness: Degree to which the functions facilitate the accomplishment of specific tasks and objectives.
Seven characteristics correspond to non-functional requirements. These describe what kinds of things can limit a software product’s ability to do what it is intended to do for its users.
- Performance Efficiency: The “hardware resources needed to perform the different functions of the software system,” whether they are processing speed, storage capacity, or data communication capability.
- Compatibility: Does the software play well with others? Does its performance suffer or does it cause the performance of other software to suffer when they are in the same environment and using the same resources?
- Usability: How easy to use and intuitive is the product for its users? Do its parts do what people assume they do? How accessible is the product for all its users? Does it respond well to common user errors?
- Reliability: The “risk of software failure and the stability of a program when exposed to unexpected conditions”.
- Security: How safe is data and information in the hands of the software product? Does information only get put in front of the right eyes, and does the product ensure that things stay that way?
- Maintainability: How easily can a product be updated, and how well does it hold up as it is updated over time? How often do new updates result in bugs or other problems?
- Portability: How easily can the product be transferred to different usage environments? Does it work using different hardware, operating systems, etc?
Non-Functional Requirements Allow Functional Requirements to Matter
Functional requirements depend on the unique role of a product, but different software products can suffer from similar non-functional problems related to performance issues, poor usability, or compatibility problems. Thus, third party tools (like our own PlayerZero) almost exclusively focus on the non-functional side. A third party tool can’t tell you what your product should do to be helpful to your users, but it can help you identify, fix, or avoid problems with how well your product works as intended.
Non-functional requirements support and enable functional requirements. You can have a great idea for a useful software product (successfully passing functional requirements), but if your software is full of bugs, takes a long time to process, and only works on Windows 10, it might as well be designed to infuriate your users – it fails non-functional requirements. Non-functional requirements allow functional requirements to matter.
Functional Stability relies on non-functional requirements, but the non-functional requirements influence each other in a complicated web of ways. Performance Efficiency might enable Compatibility, which enables Portability. A product that isn’t Reliable might not be Secure simply because its bugs negate attempts to be careful with data, and it might not be Maintainable because its spaghetti code breaks when it is updated. The important thing to keep in mind is that you have to get the fundamentals right before other aspects of your product can become important.
Measuring Software Product Quality
To ensure that your product is moving in the right direction, you have to be able to consistently measure the above characteristics of software quality with accuracy.
So how do you measure them? The ISO 25010 standards list gives suggestions for how to measure each characteristic, but keep in mind that there are tradeoffs to each measurement method and a good way of measuring a characteristic of one project might not work for another project – what makes a developer tool usable might be completely different than what makes a ecommerce app usable, for instance. Let’s take a closer look.
As the most abstract characteristic, Functional Stability is particularly un-amenable to simple one-size-fits-all measurements, since you need to figure out whether your idea is valuable and unique enough to provide market value. Instead, you’ll likely have to rely on qualitative feedback on how well your product facilitates users’ ability to achieve their goals and get results that are precise, accurate, and useful.
If your product is an ecommerce app, its functional stability is the degree to which it covers all the different use cases your customers might need. Does it let users pay how they want? How well-designed is its search engine?
Performance Efficiency can be measured by load testing, stress testing, or measuring response times to estimate how well a product will perform under realistic conditions. Essentially, you need to know whether your product will hold up when it is under a lot of use – for example, can you ecommerce app handle holiday traffic without slowing down or breaking?
Measuring compatibility first requires thinking of what other products your product could be sharing an environment and resources with. You can then measure how efficiently your product can perform its functions while sharing an environment and resources with other products, how much it detrimentally impacts the performance of these other products, and how well products can share information with each other and utilize the information they receive from each other.
Your product is usable if it’s usable for your particular users. Measures like completion rate and satisfaction level can help you determine how usable your product is, but on their own they don’t distinguish between problems with usability itself and problems with other characteristics like Performance Efficiency or Reliability that might affect usability.
Focus groups can give a more detailed picture of how users understand and try to navigate your software, and tools like FullStory can let you look into how users navigate an application. You can learn whether users are having problems finding your checkout button or using your search options or using some other feature of your app.
Since we want to know how reliable our product is for our users, we can use load testing to see how it will hold up in realistic scenarios, and we can use the number of high-priority bugs to measure how often it is critically breaking in ways that directly harm the user experience. If you’re finding a high-priority bug every few days, your product likely isn’t reliable.
The Security of your product is about how well it protects information and data, so measurements of security involve measuring the risks of your product failing to keep information and data safe in any one of a variety of ways. This can be how often data inputs are validated, how often security defects are critical, the proportion of security defects that are discovered by your own team instead of your users, or the time your product’s security defects take to fix.
You can measure maintainability by counting lines of code – the fewer lines of code, the more maintainable a product is. The number of lines of code can be a decent proxy for maintainability that doesn’t require extra work to find, but it doesn’t necessarily capture the quality and consistency of code.
You can also measure maintainability using changes in maintenance cost, frequency of new bugs per feature touch, knowledge transfer time (the time team members take to understand new features well enough to fix issues), and the increase in unplanned work when new features are added. Consider whether these measures are worth the extra effort; the value you get from a measurement should exceed the cost of taking the measurement, and, crucially, the way you measure maintainability shouldn’t incentivize poor practices.
The way in which your product needs to be portable will depend entirely on how your users need to be able to access it. Your product needs to work and work well on whatever devices and setups your users are using to access it. Can it be installed where it needs to be installed? How much time does porting it to a new device take?
Do your app’s features work as intended on whatever devices your target customers use?
Software Quality Metrics in Perspective
Characteristics of software quality are valuable to the extent that they facilitate good user experiences and help users achieve their goals. How you measure characteristics of software quality should capture how they provide value to your users. For instance, how does Portability affect your users? You should measure it – or save resources not measuring it – with how it affects your particular users as your guiding principle.
Keep in mind what can undermine your product’s value, as flaws in one characteristic of software quality can easily bleed into other aspects, and succeeding at the fundamentals can enable your good idea to shine through.