Designing a Qualified Onboard Software Production Pipeline using ECSS standards
How can we leverage ECSS understanding in the areas of onboard software engineering and software product assurance to lay out the foundation for a qualified software production pipeline that is compatible with Modelled Based Systems Engineering (MBSE) approaches and assists in qualification of onboard software up to criticality A.
Introduction
This essay will try to describe the managerial and technical challenges of providing product assurance for software products developed for space projects following ECSS standards. A background on the description of critical software products and the requirements imposed by ECSS will be presented, followed by the challenges associated with meeting these requirements. Finally, a design for a software product assurance pipeline will be described. Hence, the aim of this essay will be to present a convincing argument for investing in early software product assurance efforts with progressive scaling as the phases of the space project advance.
Definition of software criticality
Criticality in the space environment is a term used to identify systems or components whose functions could potentially have mission threatening consequences if they were to cease operating due to any internal or external cause. This applies to all the domains of space engineering, including software.
Software criticality for a system or component is assessed based on the criticality of the functions it executes, specifically its highest criticality function (ECSS-Q-ST-40C, 15 February 2017). Function criticality is assessed based on the severity of the potential consequences that could arise from that given function not operating as intended (ECSS-Q-ST-40C, 15 February 2017). Hence software criticality follows the following logic:
Software criticality categories
Software criticality is broken down into four possible categories, as shown in Figure 2. These categories dictate the level of documentation needed to qualify a software product according to the space engineering software standard (ECSS-E-ST-40C, 30 April 2025). This entails that, according to the assigned level of criticality of a software product, one can then do some tailoring of the standard and its requirements, which in the case of low criticality categories such as C and D, can severely reduce the workload required for product assurance. This tailoring is presented in detail in (ECSS-E-ST-40C, 30 April 2025) Annex R Table R-1.
Following this logic of tailoring of the standard it is easy to see that the amount of effort required for software product assurance greatly increases as the software criticality increases, hence why software criticality A is rarely applied and typically reserved for software products that could lead to potential loss of life.
Impact of high criticality software in a project’s life cycle
Taking into consideration the exponential complexity of software product assurance efforts according to criticality categorization. It is also to be expected that there could be potential impacts on projects that contain these critical software products, regarding schedule, availability of domain experts, and overall budget.
A common representation of the impact of quality assurance in a project’s lifecycle can be seen in Figure 3, were the relationship between cost and time when failure is found is represented. Figure 3 clearly illustrates that the sooner a failure is caught the cheaper it is to mitigate that given failure. However, Figure 3 does not take into account costs for product assurance efforts and focuses instead on costs for handling the failure once found.
In the case of software product assurance, and specifically “Category A” software, Figure 3 can be recontextualised to show that not only the costs of a failure increase with time, but also those of implementing quality assurance processes. Furthermore, after a certain threshold the cost no longer is that of a repair or small implementation, but rather that of a full restart of the software project. This is graphically represented in Figure 4 where we can see three plots representing software categories A, B and C, being plotted with regards to cost of software assurance vs project time according to ECSS project phases. This plot is based on the author’s experience in the field of on-board software as well as experiences shared with other industry experts during the ESA software product assurance conference at ESTEC in Q4 2025.
Figure 4 also shows that at a certain point a cost threshold is reached, after which the sunken cost fallacy applies due to the fact that it would be less costly to restart the software project with proper quality assurance strategies, rather than continue investing in trying to rigorously qualify a code base that has no built in qualification tools around it. This is mostly due to the requirements imposed by (ECSS-E-ST-40C, 30 April 2025) on critical software (Categories A, B and C), such as for example 5.8.3.5 part b which lists 100% of code coverage for Category A software in the following areas:
- Source code statement coverage.
- Source code decision coverage.
- Source code modified condition and decision coverage.
Furthermore requirement 5.8.3.5-part e requires that the developer achieves 100% object code coverage for code that cannot be directly traced to source code.
Achieving this kind of code coverage in all these areas in the case of any given code base for on-board space applications, can prove a daunting task, and in most cases gaps in the test coverage will occur. This will leave the development team with no option but to provide justification for these gaps in coverage, for which vast knowledge and understanding of the code base will be required. Taking this into consideration, it is clear that trying to apply 100% test coverage on a code base that has not been developed alongside all the needed tests will be a gargantuan effort, not to mention that when trying to provide justification for any gaps in the test coverage it will most likely prove quite challenging for the developers to remember the very fine details of a design executed during Phase B if they are doing 100% code coverage in Phase D.
As such the premise of this report is to convince the reader that the right time to apply software qualification processes is during “Phase 0” and that the best way to extend it to the following phases is by means of a qualified Continuous Integration (CI) pipeline for regression testing, such as the one proposed in the following section.
Design of a continuous integration pipeline for space applications
The design of the proposed continuous integration pipeline for space applications is based around section 5.8 of (ECSS-E-ST-40C, 30 April 2025), which details the software code verification process and as such defines the requirements for achieving software product assurance. Its rough structure can be seen in Figure 5, which shows the step-by-step flow and the requirements addressed in each of the stages of the pipeline. The pipeline design is based on the understanding that software will be developed following best practices in version control processes, where software development is done in separate branches which can then be merged into the main software branch. Merge requests will trigger the pipeline execution, ensuring that new code added to the application will not affect the previously verified software. This is what is typically referred to as regression testing (IBM, s.f.). The following sections will provide details on each of the individual stages.
Static code analysis
Static code analysis is a testing methodology which can be run on source code prior to its execution. It is widely used in the field of critical software development to check fulfilment of coding standards such as MISRA-C (Kraeling, 2019), find bugs and security vulnerabilities (Miramar, 2025). Some of the issues static code analysis can find prior to code execution are:
- Cyclomatic complexity levels above the recommended maximum of 10 leading to increased chances of programming defects (McCabe, 1976).
- Type safe discrepancies such as type mismatches.
- Dead code or unreachable code.
- Memory leaks, stack overflow, null pointer dereferences, dangling pointers, use-after-free errors etc…
As such implementing static code analysis as one of the stages of the software qualification pipeline will help fulfil the following (ECSS-E-ST-40C, 30 April 2025) 5.8.3.5 requirements:
- Part a.3: By checking compliance with a coding standard such as MISRA-C, static code analysis fulfils the requirement to be in conformance with coding standards.
- Part a.5: By checking for bugs and vulnerabilities present in the source code, static code analysis will provide part of the proof needed for the verification of this requirement.
- Part a.7: Static code analysis will help find out about corner cases that might result in run time errors and otherwise not found until they occur during operation.
- Part f: Measures of code robustness will be greatly complimented by documentation produced by static code analysis tools.
Unit tests
Unit tests are software-based tests that can be written alongside the source code and are meant to target the smallest functional elements of the code, often referred as units. Implementation of unit testing as one of the stages of the software qualification pipeline will help fulfil the following (ECSS-E-ST-40C, 30 April 2025) 5.8.3.5 requirements:
- Part a.7: Unit testing will provide assurance at software function level that all the possible behaviours, inputs and outputs of the function have been tested against, and that as such the tested functions will not produce run time errors.
- Part b: Code coverage, as shown in Figure 6, for the different categories listed by these requirements can only be achieved via unit testing (Hamilton, 2024):
- Source code statement coverage: Achieved by white box testing with unit tests for each of the statements present in the code base.
- Source code decision coverage: Achieved by white box testing with unit tests for each of the boolean expressions of the code base.
- Source code modified condition and decision coverage (MC/DC): Achieved by testing the following elements of a function with unit tests (Hayhurst, 2001):
- Each entry and exit point is invoked.
- Each decision takes every possible outcome.
- Each condition in a decision takes every possible outcome.
- Each condition in a decision is shown to independently affect the outcome of the decision.
Part C: Results from unit testing campaigns can provide all the necessary metrics to support code coverage metrics.
Functional tests
Unlike the earlier two stages, functional tests are project specific and are supposed to test that the software under test is compliant with the project specifications. As such these tests are highly related to requirements and the architectural elements of the software design, providing traceability between them that can be used in the software product assurance process. Implementation of functional tests into the software qualification pipeline will help fulfil the following (ECSS-E-ST-40C, 30 April 2025) 5.8.3.5 requirements:
- Part a.1: By providing direct traceability between requirements, design elements, source code and functional tests, this stage can demonstrate consistency between all these elements.
- Part a.3: Similarly to Part a.1 the achieved traceability from automatic functional testing will also fulfil this requirement.
- Part a.5: By doing functional testing the consistency between the previously tested software units is achieved. This is called integration testing.
- Part a.6: By doing functional testing of safety related functionalities this requirement can be fulfilled.
- Part a.7: Functional tests will further prove that there are no run time errors.
- Part a.8: Furthermore, functional testing can also prove that error handling methods work as expected.
Conclusion and next steps
As shown in the analysis and design presented in this essay, the implementation of an early software qualification pipeline can not only be greatly beneficial to a projects schedule and budget, but in some cases, it is absolutely essential.
As next steps it is expected that a conference proceeding for SPIE 2026 will be prepared, presenting further details on this topic, as well as the implementation of the design presented here.
References
ECSS-E-ST-40C. (30 April 2025). ECSS-E-ST-40C Rev.1 Space engineering software. Noordwijk, The Netherlands: ESA-ESTEC Requirements & Standards Section.
ECSS-Q-ST-40C. (15 February 2017). ECSS-Q-ST-40C Rev.1 Space product assurance. Noordwijk, The Netherlands: ESA-ESTEC Requirements & Standards Division.
ECSS-Q-ST-80C. (30 April 2025). ECSS-Q-ST-80C Rev.2 Software product assurance. Noordwijk, The Netherlands: ESA-ESTEC Requirements & Standards Section.
Hamilton, T. (2024, November 21). Code Coverage Tutorial: Branch, Statement & Decision Testing. Retrieved from GURU99: https://www.guru99.com/code-coverage.html
Hayhurst, K. J. (2001). A Practical Tutorial on Modified Condition/. NASA.
IBM. (n.d.). regression-testing. Retrieved from ibm.com: https://www.ibm.com/think/topics/regression-testing
Kraeling, M. (2019). 11 – Safety-Critical Development. In R. Oshana, & M. Kraeling, Software Engineering for Embedded Systems (pp. 394-426).
McCabe, T. (1976). A Complexity Measure. IEEE, 308 – 320.
Miramar, H. (2025, September 19). What Is Static Code Analysis and How Does It Work. Retrieved from Mergify: https://articles.mergify.com/what-is-static-code-analysis/
SEAC. (2025). ECSS Standards Course. Retrieved from https://seac-business-school.mykajabi.com/products/ecss-standards-2
