Photo Rowan Heuvel on unsplash
The aim of a technology readiness scale
The Technology Readiness Level (TRL) is a method used to assess the maturity of technologies during the acquisition phase of a program. It was developed by NASA in the 1970s to facilitate consistent discussions on technical maturity across different types of technology. TRL is measured on a scale from 1 to 9, with 1 representing the earliest research stage and 9 indicating full maturity and practical implementation.
Each new TRL level signifies:
- More people involved
- Higher probability of reaching production level
- Greater financial investment and attention
The transition from TRL 6 to 7 is often considered critical, as it marks the point where the product starts to be used in real conditions. For instance, when a software company releases its new Computer-Aided Design product to external beta-testers or when an engine manufacturer ships a prototype to the integration division of the target machine.
The reason for this gap is the sudden addition of people with higher expectations and lower tolerance, resulting in increased stress within the product’s community.
This phenomenon can be related to the concept of “Crossing the Chasm” in marketing. The chasm represents a significant gap between the population of “Early Adopters,” who are more willing to accept new products, and the larger “Early Majority” group, which has higher expectations and is more challenging to persuade.
Defining TRL Levels for Scientific Software in HPC
In this section, we will provide a TRL scale specifically adapted to the scientific software commonly used in the Cerfacs business. This TRL scale is derived from the KTH Technology readiness level, which is a part of the KTH Innovation Readiness Level. These levels are based on the original scale outlined in Horizon Programme 2020 of the European Community (Annex G), reproduced here:
- TRL 1 – Basic principles observed
- TRL 2 – Technology concept formulated
- TRL 3 – Experimental proof of concept
- TRL 4 – Technology validated in the lab
- TRL 5 – Technology validated in a relevant environment (industrial environment for Key Enabling Technologies)
- TRL 6 – Technology demonstrated in a relevant environment (industrial environment for Key Enabling Technologies)
- TRL 7 – System prototype demonstrated in an operational environment
- TRL 8 – System complete and qualified
- TRL 9 – Actual system proven in an operational environment (competitive manufacturing for Key Enabling Technologies; or in space)
The following sectionz provides specific definitions of these levels within the context of scientific software in High Performance Computing (HPC) at Cerfacs.
Low TRLs 1-2-3: Preliminary Research
During the low TRL stages, specifically TRLs 1-2-3, the primary figures involved are the supervisor and the author, working together to establish the credibility and initiate research on the topic.
TRL 1: Basic principles observed
At this stage, the idea has been documented as a concept, supported by at least two specialists. Extensive literature review indicates no pre-existing similar ideas. A supervisor, who can champion the idea, has been identified.
TRL 2: Technology concept and/or application formulated
The idea has been further validated through literature review. Early work has demonstrated promising results, attracting potential users. Critical aspects have been identified, and at least one potential solution has been proposed for each. Additionally, at least one potential user has expressed interest in the project. The supervisor (ideally with a potential user) has defined the “Minimum Viable Product” as the primary objective for future efforts, along with the critical skills required to continue the research.
The supervisor has prepared general communication support materials such as documents and slide decks. In addition to presenting early findings, these materials explicitly state:
- The research nature of the project, as engineering work is to be avoided in a research center.
- The intellectual property strategy to be followed.
- The identified contributors, especially if a peer-reviewed paper is expected at TRL 4.
TRL 3: Analytical and experimental proof-of-concept of critical function and/or characteristics
The author, under the guidance of the supervisor, conducts an in-depth literature review. They acquire the necessary skills and demonstrate proficiency by reproducing early works. Furthermore, they test solutions for each critical function, yielding positive and reproducible results.
Academic research with a PhD or a Post-doc belongs mostly to this level. Peer-reviewed scientific papers serve as solid evidence for reaching TRL 3 level.
Photo Matt Ridley on unsplash
Mid TRLs 4-5-6: Advanced Research
During the mid TRL stages (TRLs 4-5-6), the primary individual driving the progress is the author, who takes on the responsibility of advancing the idea to a proof-of-concept capable of withstanding critical investigation.
She must interact with a technical referent in order to develop using the shared practices in the community.
TRL 4: Technology validation in laboratory
The author selects a reception case, which serves as a situation to demonstrate the potential of the idea. The supervisor ensures that the reception case aligns with the minimum viable product.
An initial solution for the reception case is developed. The software codebases are readable, commented, and stored in git repositories, available for sharing upon request.
TRL 5: Technology validation in relevant environment
The author, with the assistance of the community, defines an integration plan aimed at achieving the desired functionality.
The prototype is built until the reception case can be reached on realistic hardware. At this point, the author conducts a typical simulation to evaluate the functionality’s scope.
The software codebases are of “Team Grade,” versioned using TAGS and CHANGELOG, and accompanied by installation and documentation procedures.
TRL 6: Technology demonstration in a relevant environment
The prototype is handed over to an internal alpha tester. He conducts typical simulations to reassess the operational boundaries.
The author verifies that the alpha test was conducted as expected and addresses any issues reported by the internal alpha tester. The critical documentation must be completed at this level.
The technical referent checks that the developments are aligned with the comunity practices.
The supervisor ensures that the working boundaries align with the expected minimum viable product. They identify an external beta tester and establish a sustainable deployment and support strategy in terms of manpower and technology for the external testing phase.
Photo Unsplash, A a giant paper plane at TRL-6
High TRLs 7-8-9: Application Level
At high TRLs, the supervisor triggers short actions in collaboration with:
- the external beta tester, which is usually the earliest customer
- a technical referent, when a technical issue arises.
- a modelling referent, when the simulation is not satisfactory.
TRL 7: Technology prototype demonstration in an operational environment
The prototype is handed over to an external beta tester. She conducts typical simulations and reassesses the operational boundaries.
The supervisor ensures that any issues raised by the external beta tester are addressed by enough manpower. They collaborate with the final user to determine the targeted level of service.
Software codebases are of “Production Grade” : there is a continuous testing coverage that reaches 30% of the code.
Arturo’s Desert Eagle Airplane, from The Register, depicting a giant paper plane trying to reach TRL-7 in 2012.
TRL 8: Actual Technology system completed and qualified through test and demonstration
The final user reports the initial full-scale deployment with satisfactory results. The supervisor ensures that the manpower is adequate for the product’s usage. This means: - gathering short-term teams to take the usual support actions. - escalation to a technical referent when hard technical issues are encountered - escalation to a modelling referent when hard modelling issues are encountered
If necessary, they facilitate the creation of training materials or training events and collect user feedback.
The testing coverage should aim for 80%. The codebase should be treated as “Legacy” code, open for extension and closed for deprecations.
TRL 9: Actual Technology system proven in operational environment
The final user and the supervisor gather information on the product’s life cycle to:
- Identify dangers and threats regarding the current level of service.
- Optimize the balance between manpower and product usage.
- Extract lessons learned for future projects.