Test Automation Maturity Dashboard
Seven projects surveyed in Q1 2026, scored against the Testing Maturity Assessment finalised on 5 June 2026.
7
Projects surveyed
54%
Weighted overall maturity
60%
Avg. automation coverage
86%
GxP regulated projects
Maturity Level Guide
Levels are assigned from observable capabilities: tooling, CI/CD integration and automation coverage. Tower and project ratings below use the same M1 to M5 scale.
Initial & Ad-hoc
Mostly manual testing with isolated automation scripts. Test execution is ad-hoc, with limited repeatability and no defined process.
Structured Automation
Standardised, reusable automated tests with central test management. Repeatable execution, focused mainly on the UI layer.
Integrated & Risk-Based
Risk-based test prioritisation with automation across UI and API layers. Tests run automatically as part of CI/CD pipelines.
AI-Assisted & Optimised
AI-assisted test creation and maintenance, automated test data provisioning, and mocking of dependent services. Broad automation across all layers, fully integrated into CI/CD.
DevTestOps
Quality is embedded across the entire delivery lifecycle with shared ownership. Continuous testing at every stage, including quality feedback from production.
Maturity by Category
Test Strategy & Planning
15% weight5 of 7 projects have a defined or transitioning automation strategy. 5 of 7 have plans to expand automation in the next 6–12 months.
Test Automation Coverage
20% weightAverage automation coverage is 60% across all projects. 5 of 7 projects automate more than half of their test cases.
CI/CD Integration
20% weight2 of 7 projects run automated tests in a full CI/CD pipeline. 2 have no pipeline and 1 is not yet integrated.
Documentation & Traceability
10% weight5 of 7 projects keep fully audit-ready documentation. 4 of 7 have full requirements traceability.
Test Data Management
20% weight2 of 7 projects provision test data automatically. The rest prepare it manually.
Team Capability & Governance
15% weightTeams average 3.3/5 for strategy clarity and 3.4/5 for regular improvement reviews. 5 of 7 projects have dedicated or shared automation maintainers.
Maturity by Tower
Score = mean project score in the tower (1–5). Level = mean project M-level, rounded.
Overall Maturity
Project Breakdown
| Project | Tower | Tool | Automation | CI/CD | GxP | M-Level | Score |
|---|---|---|---|---|---|---|---|
| TOCS Build Project | IT O&L | Playwright | 51–75% | Planned | Yes | M2 | 3.0 |
| VAULTCRM | IT M+S | Tosca | 76–100% | Full | Yes | M3 | 4.0 |
| AH Product Readiness | IT M+S | Tosca | 51–75% | None | No | M2 | 4.0 |
| FRED | GBS MDM | Tosca | 26–50% | None | Yes | M2 | 2.0 |
| GBS | GBS | Tosca | 51–75% | Not integrated | Yes | M2 | 3.5 |
| OMP CLinOps | IT RDM | Tosca | 76–100% | Full | Yes | M3 | 3.0 |
| AMPLIFY | IT RDM | Cypress | 11–25% | Partial | Yes | M2 | 4.0 |
M-Level = maturity level from observable capabilities (tooling, CI/CD, coverage). Score = mean of the strategy clarity and continuous improvement ratings (1–5).
Assessment Framework
18 scored questions across 6 weighted categories. Finalised 5 June 2026; AI question added June 2026.
Assessment Framework
18 scored questions across 6 weighted categories. Finalised 5 June 2026; AI question added June 2026.
1. Scoring Model
Question scores
Each of the 18 questions is answered on a 1–5 scale, where 1 is the lowest and 5 the highest maturity. Some questions use intermediate anchors (2 and 4).
Category %
Category % = average question score within the category ÷ 5. A category averaging 3 of 5 scores 60%.
Overall %
Overall % = Σ (category % × category weight). Current portfolio result: 54%.
2. Categories, Weights & Questions
Test Strategy & Planning
15%- Q1. Test automation strategy maturity
- Q2. Automation expansion roadmap (next 6–12 months)
- Q3. When does testing typically start?
Test Automation Coverage
20%- Q4. Automation coverage (% of test cases automated)
- Q5. Duration of manual regression execution
CI/CD Integration
20%- Q6. CI/CD integration of automated tests
- Q7. Test execution frequency
- Q8. Execution duration of automated portfolio
- Q9. Environment stability (SUT)
Documentation & Traceability
10%- Q10. Traceability between requirements and test cases
- Q11. Test coverage of requirements
- Q12. Documentation maturity / audit-readiness (GxP-weighted)
Test Data Management
20%- Q13. Test data management approach
- Q14. Test data availability and provisioning
Team Capability & Governance
15%- Q15. Accountable owner for testing & automation
- Q16. Skills maturity within the project team
- Q17. Continuous improvement of testing process & assets
- Q18. Use of AI in testing and test automation
3. Maturity Level Mapping
| Level | Title | Overall % range | Efficiency gain |
|---|---|---|---|
| M1 | Initial & Ad-hoc | 0–20% | 1.3x |
| M2 | Structured Automation | 21–40% | 3x |
| M3 | Integrated & Risk-Based | 41–60% | 6xcurrent: 54% |
| M4 | AI-Assisted & Optimised | 61–80% | >10x |
| M5 | DevTestOps | 81–100% | >20x |
Working mapping using equal 20-point bands. The final mapping of overall % to maturity level is being confirmed as part of the assessment rollout.
4. GxP-Aware Scoring (Q12, Documentation)
| Documentation state | GxP score | Non-GxP score |
|---|---|---|
| Minimal or missing documentation | 1 | 3 |
| Partially documented | 3 | 4 |
| Fully documented & audit-ready | 5 | 5 |
Scoring adjusts for regulatory context: non-GxP systems are not penalised to the same degree for lighter documentation, while GxP systems are held to full audit-readiness.
5. Gates & Ceilingsunder definition
Gates stop a single weak area from being averaged away by strong scores elsewhere. Two models are being evaluated: hard gates on individual questions (a low score on a critical question, such as CI/CD integration or test data availability, caps the achievable overall level) or category ceilings (the overall level cannot exceed the weakest weighted category by more than one level). The final gate definitions and minimum category scores per level are part of the next assessment iteration.
6. Non-Scored Elements
Section 1 context: project name, system type, GxP applicability, and automation tooling. Used for segmentation only, no scoring impact.
Q19 (optional): main testing/automation challenges, covering test data, skill gaps, strategy, tooling/integration, maintenance effort, capacity, compliance, and initial costs. Captured as qualitative insight.
Prepared by Automators for Boehringer Ingelheim · Data as of Q1 2026 · Scoring per Testing Maturity Assessment (final, 05.06.2026)