CI/CD processes in a project very much depend on the maturity of processes in the project as a whole and in testing in particular. Therefore, the first step with which we usually begin to improve the existing process is to evaluate the existing solution and build a baseline.
The baseline includes answers to the following questions:
- What is the system architecture?
- Which build server is used?
- What is the principle of versioning?
- How branching and merge strategy is defined?
- How to define the build and release pipeline?
- What is the Project Release creation strategy?
- Are there any automatic code verification utilities?
- Do developers write tests and are there utilities for evaluating code coverage by tests?
- Are there integration tests and on what environment are they performed?
- How many environments are deployed in the project and what is the strategy for their use?
- How does the deployment happen?
- Who has deployment privileges in test environments?
- Who has access to the artifacts within the development, testing, and DevOps environments?
- How is the environment being configured?
- Who has access to the configuration of the environments?
- What APIs of which systems are used in automated tests?
- What are the data warehouses, external resources and who has access to them?
- What third-party systems are used in the project?
- By what principle are testing levels divided?
- How are Test Suites divided?
- What is the test schedule?
- How automated tests are written (technologies, incoming information, outgoing artifacts)?
- How tests are integrated into the pipeline (steps, environments, suites)?
- How is the coverage analysis performed?
- How is auto-testing reporting performed?
- How defects are reflected in autotests?
- How is test data used?
- How is test data created and updated (for functional and non-functional testing)?
- What mocks are used in the testing process?
- How is Hot-fix Testing Strategy defined?
- What are the Quality Gates for releases?
- How often do releases happen?
- How are hot-fixes performed and delivered?
- How is the release delivered to the customer?
- What is written in the Test Plan document?
- What are the Project Acceptance Criteria?
- How is the Traceability Matrix built?
- How third-party system updates impact development and testing?
- How is the integration testing with external systems performed?
- What non-functional tests are carried out and with what frequency?
- How is the management of sensitive data in the project performed?
- What is the average lifetime of defects?
- What is the location of defects?
- How is test coverage by test levels distributed?
After the information is collected and analyzed, we offer 2-3 specific goals, the achievement of which will significantly improve the CI/CD processes in the project. At the same stage, metrics for assessing changes are discussed and approved.
To achieve these goals, we, together with all involved departments, draw up a plan, approve it and proceed to the implementation of the plan on the selected Pilot project: we introduce a new way of working either in a separate team or when developing specific functionality.
Upon completion of the Pilot project, we, first of all, assess whether the goals are achieved, make a retrospective, amend the plan of necessary actions and plan the implementation of the entire project.
At the end of the implementation, we conduct lessons learning and plan a periodic assessment of the project to make sure initial quality objectives are continuously met.