We distinguish between abstract quality dimensions (high-level categories like reliability or maintainability defined by standards such as ISO/IEC 25010 and concrete quality indicators (measurable proxies like test coverage or community engagement). While indicators provide empirical evidence, they cannot fully capture the subjective, context-dependent nature and dimensions of “good” software.
Assessing research software for quality requires balancing “fit for purpose” (the ability of a tool to solve a specific scientific problem) against long-term goals of reusability and sustainability as championed by the FAIR4RS principles. This assessment is further complicated by the non-linear relationship between technical quality and scientific impact, as well as the potential for software to be repurposed for unintended uses, which may shift the relevant quality criteria entirely.
We build upon the nine top-level [ISO software quality dimensions][ISO-IEC-25010] to bridge the gap between the nature or research software quality (e.g. community around research software) and engineering standards.
Quality Dimension Definitions
EVERSE project is working to formally define a number of research software quality dimensions along with a formal schema to be used in machine readable metadata embedded in various resources.
Each of the quality dimensions will be an umbrella for a number of quality indicators, representing a specific software quality aspect that can be measured.
Current research software quality dimensions are described below. This is still work in progress - we expect indicators in particular to be updated in the near future.
Community
Research software development and maintenance increasingly depend on active communities of contributors, users, and stakeholders. A thriving community can compensate for technical limitations, ensure long-term viability,and drive innovation through collaborative development. For research software, community engagement takes on particular importance due to the distributed nature of research collaborations, the need for domain-specific expertise, and the challenges of sustaining software beyond individual research projects.
Community health encompasses the social and organisational structures that enable effective collaboration, knowledge sharing, and decision-making around research software. This includes governance models, contributor engagement patterns, communication channels, and the coordination processes that facilitate participation across diverse stakeholder groups. The significance of community aspects varies by software tier - analysis code may have a single author with minimal community interaction, while research software infrastructure requires structured community governance and broad stakeholder engagement.
Community aspects overlap with sustainability and open source dimensions, as healthy communities directly contribute to long-term software viability and align with open source development practices. Despite this overlap, community merits explicit attention as it captures critical social dynamics that determine whether research software achieves adoption, maintains quality, and evolves to meet changing research needs.
Read more about community dimension on TechRadar.Compatibility
Degree to which a product, system or component can exchange information with other products, systems or components, and/or perform its required functions while sharing the same common environment and resources. This characteristic is composed of the following sub-characteristics:
Co-existence
Degree to which a product can perform its required functions efficiently while sharing a common environment and resources with other products, without detrimental impact on any other product.
Interoperability
Degree to which a system, product or component can exchange information with other products and mutually use the information that has been exchanged.
Read more about compatibility dimension on TechRadar.FAIRness
FAIRness refers to the degree to which research software adheres to the FAIR principles: Findable, Accessible, Interoperable, and Reusable. These principles, adapted for research software, aim to enhance the discoverability, accessibility, interoperability, and reusability of software, thereby maximizing its value and impact in scientific research.
Read more about fairness dimension on TechRadar.Flexibility
Degree to which a product can be adapted to changes in its requirements, contexts of use or system environment. This characteristic is composed of the following sub-characteristics:
Adaptability
Degree to which a product or system can effectively and efficiently be adapted for or transferred to different hardware, software or other operational or usage environments.
Scalability
Degree to which a product can handle growing or shrinking workloads or to adapt its capacity to handle variability.
Installability
Degree of effectiveness and efficiency with which a product or system can be successfully installed and/or uninstalled in a specified environment.
Replaceability
Degree to which a product can replace another specified software product for the same purpose in the same environment.
Read more about flexibility dimension on TechRadar.Functional suitability
This characteristic represents the degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions. This characteristic is composed of the following sub-characteristics:
Functional completeness
Degree to which the set of functions covers all the specified tasks and intended users’ objectives.
Functional correctness
Degree to which a product or system provides accurate results when used by intended users.
Functional appropriateness
Degree to which the functions facilitate the accomplishment of specified tasks and objectives.
Read more about functional suitability dimension on TechRadar.Interaction Capability
Degree to which a product or system can be interacted with by specified users to exchange information via the user interface to complete specific tasks in a variety of contexts of use. This characteristic is composed of the following sub-characteristics:
Appropriateness recognizability
Degree to which users can recognize whether a product or system is appropriate for their needs.
Learnability
Degree to which the functions of a product or system can be learnt to be used by specified users within a specified amount of time.
Operability
Degree to which a product or system has attributes that make it easy to operate and control.
User error protection
Degree to which a system prevents users against operation errors.
User engagement
Degree to which a user interface presents functions and information in an inviting and motivating manner encouraging continued interaction.
Inclusivity
Degree to which a product or system can be used by people of various backgrounds (such as people of various ages, abilities, cultures, ethnicities, languages, genders, economic situations, etc.).
User assistance
Degree to which a product can be used by people with the widest range of characteristics and capabilities to achieve specified goals in a specified context of use.
Self-descriptiveness
Degree to which a product presents appropriate information, where needed by the user, to make its capabilities and use immediately obvious to the user without excessive interactions with a product or other resources (such as user documentation, help desks or other users).
Read more about interaction capability dimension on TechRadar.Maintainability
This characteristic represents the degree of effectiveness and efficiency with which a product or system can be modified to improve it, correct it or adapt it to changes in environment, and in requirements. This characteristic is composed of the following sub-characteristics:
Modularity
Degree to which a system or computer program is composed of discrete components such that a change to one component has minimal impact on other components.
Reusability
Degree to which a product can be used as an asset in more than one system, or in building other assets.
Analysability
Degree of effectiveness and efficiency with which it is possible to assess the impact on a product or system of an intended change to one or more of its parts, to diagnose a product for deficiencies or causes of failures, or to identify parts to be modified.
Modifiability
Degree to which a product or system can be effectively and efficiently modified without introducing defects or degrading existing product quality.
Testability
Degree of effectiveness and efficiency with which test criteria can be established for a system, product or component and tests can be performed to determine whether those criteria have been met.
Read more about maintainability dimension on TechRadar.Open Source Software
Open source software is software with source code that anyone can inspect, modify, and enhance. Research software can be published with or without open access to the source code. Open access to source code aligns better with academic research purposes than closed source software; open source software aligns with the FAIR4RS principles. It allows other researchers to directly verify the methods used to produce the results published in papers. It also makes reproducibility much easier. In addition to these research-driven reasons, publishing research software as open source software can help with long term maintenance in a cost-effective way, since interested developers can easily contribute new functionality or fix bugs. Moreover, by integrating with the greater open source ecosystem, researchers can leverage tools and support communities already available. As such, for most academic communities with limited resources, it is also a good choice from a software engineering perspective
Read more about open source software dimension on TechRadar.Performance Efficiency
This characteristic represents the degree to which a product performs its functions within specified time and throughput parameters and is efficient in the use of resources (such as CPU, memory, storage, network devices, energy, materials…) under specified conditions. This characteristic is composed of the following sub-characteristics:
Time behaviour
Degree to which the response time and throughput rates of a product or system, when performing its functions, meet requirements.
Resource utilization
Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity
Degree to which the maximum limits of a product or system parameter meet requirements.
Read more about performance efficiency dimension on TechRadar.Performance Efficiency
This characteristic represents the degree to which a product performs its functions within specified time and throughput parameters and is efficient in the use of resources (such as CPU, memory, storage, network devices, energy, materials…) under specified conditions. This characteristic is composed of the following sub-characteristics:
Time behaviour
Degree to which the response time and throughput rates of a product or system, when performing its functions, meet requirements.
Resource utilization
Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity
Degree to which the maximum limits of a product or system parameter meet requirements.
Read more about performance efficiency dimension on TechRadar.Performance Efficiency
This characteristic represents the degree to which a product performs its functions within specified time and throughput parameters and is efficient in the use of resources (such as CPU, memory, storage, network devices, energy, materials…) under specified conditions. This characteristic is composed of the following sub-characteristics:
Time behaviour
Degree to which the response time and throughput rates of a product or system, when performing its functions, meet requirements.
Resource utilization
Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity
Degree to which the maximum limits of a product or system parameter meet requirements.
Read more about performance efficiency dimension on TechRadar.Performance Efficiency
This characteristic represents the degree to which a product performs its functions within specified time and throughput parameters and is efficient in the use of resources (such as CPU, memory, storage, network devices, energy, materials…) under specified conditions. This characteristic is composed of the following sub-characteristics:
Time behaviour
Degree to which the response time and throughput rates of a product or system, when performing its functions, meet requirements.
Resource utilization
Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity
Degree to which the maximum limits of a product or system parameter meet requirements.
Read more about performance efficiency dimension on TechRadar.Performance Efficiency
This characteristic represents the degree to which a product performs its functions within specified time and throughput parameters and is efficient in the use of resources (such as CPU, memory, storage, network devices, energy, materials…) under specified conditions. This characteristic is composed of the following sub-characteristics:
Time behaviour
Degree to which the response time and throughput rates of a product or system, when performing its functions, meet requirements.
Resource utilization
Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity
Degree to which the maximum limits of a product or system parameter meet requirements.
Read more about performance efficiency dimension on TechRadar.