Custom client load testing

docs/automated-testing/performance-testing/load-testing.md

@@ -6,13 +6,13 @@ A load test is designed to determine how a system behaves under expected normal
 
 ## Why Load Testing
 
-The main objective is to prove the system can behave normally under the expected normal load before releasing it to production. The criteria which defines "behave normally" will depend on your target, this may be as simple as "the system remains available", but it could also include meeting a response time SLA or error rate.
+The main objective is to prove the system can behave normally under the expected normal load before releasing it to production. The criteria that define "behave normally" will depend on your target, this may be as simple as "the system remains available", but it could also include meeting a response time SLA or error rate.
 
 Additionally, the results of a load test can also be used as data to help with capacity planning and calculating scalability.
 
 ## Load Testing Design Blocks
 
-There are a number of basic component which are required to carry out a load test.  
+There are a number of basic components that are required to carry out a load test.  
 
 1. In order to have meaningful results the system needs to be tested in a production-like environment with a network and hardware which closely resembles the expected deployment environment.
 
@@ -26,21 +26,49 @@ There are a number of basic component which are required to carry out a load tes
 
 ### Planning
 
-1. **Identify key scenarios to measure** - Gather these scenarios from Product Owner, they should provide a representative sample of real world traffic.
+1. **Identify key scenarios to measure** - Gather these scenarios from Product Owner, they should provide a representative sample of real world traffic.  The key activity of this phase is to agree on and define the load test cases
 2. **Determine expected normal and peak load for the scenarios** - Determine a load level such as concurrent users or requests per second to find the size of the load test you will run.
 3. **Identify success criteria metrics** - These may be on testing side such as response time and error rate, or they may be on the system side such as CPU and memory usage.
-4. **Select the right tool** - Many frameworks exist for load testing so consider if features and limitations are suitable for your needs. (Some popular tools are listed below).
-5. **Observability** - Determine which metrics need to gathered to gain insight into throughput, latency, resource utilization, etc.
-6. **Scalability** - Determine the amount of scale needed by load generator, workload application, CPU, Memory, and network components needed to achieve testing goals. The use of kubernetes on the cloud can be used to make testing infinitely scalable.
+4. **Agree on test matrix** - Which load test cases should be run for which combinations of input parameters.
+5. **Select the right tool** - Many frameworks exist for load testing so consider if features and limitations are suitable for your needs. (Some popular tools are listed below).  This may also include development of a custom load test client, see Preparation phase below.
+6. **Observability** - Determine which metrics need to gathered to gain insight into throughput, latency, resource utilization, etc.
+7. **Scalability** - Determine the amount of scale needed by load generator, workload application, CPU, Memory, and network components needed to achieve testing goals. The use of kubernetes on the cloud can be used to make testing infinitely scalable.
+
+### Preparation
+The key activity is to replace the end user client with a test bench that simulates one or more instances of the original client.  For standard 3rd party tools it may suffice to configure the existing test UI before initiating the load tests.  
+
+If a custom client is used, code development will be required:
+
+1. **Custom development** - Design for minimal impact/overhead.  Be sure to capture  only those features of the production client that are relevant from a load perspective.   Does it matter if the same test is duplicated, or must the workload be unique for each test?  Can all tests be run under the same user context?
+2. **Test environment** - Create test environment that resembles production environment.  This includes the platform as well as external systems, e.g., data sources.
+3. **Security contexts** - Be sure to have all requisite security contexts for the test environment.  Automation like pipelines may require special setup, e.g., OAuth2 client credential flow instead of auth code flow, because interactive login is replaced by non-interactive.  Allow planning leeway in case admin approval is required for new security contexts.
+4. **Test data strategy** - Make sure that output data format (ascii/binary/...) is compatible with whatever analysis tool is used in the analysis phase.  This also includes storage areas (local/cloud/...), which may trigger new security contexts.  Bear in mind that it may be necessary to collect data from sources external to the application to correlate potential performance issues with the application behavior.  This includes platform and network metrics.  Make sure to collect data that covers analysis needs (statistical measures, distributions, graphs, etc.).
+5. **Automation** -  Repeatability is critical.  It must be possible to re-run a given test multiple times to verify consistency and resilience of the application itself and the underlying platform.  Pipelines are recommended whenever possible.
+Evaluate whether load tests should be run as part of the PR strategy.
+6. **Test client debugging** - All test modules should be carefully debugged to ensure that the execution phase progresses smoothly.
+7. **Test client validation** - All test modules should be validated for extreme values of the input parameters.  This reduces the risk of running into unexpected difficulties when stepping through the full test matrix during the execution phase.
 
 ### Execution
 
-It is recommended to use an existing testing framework (see below). These tools will provide a method of both specifying the user activity scenarios and how to execute those at load. It is common to slowly ramp up to your desired load to better replicate real world behavior. Once you have reached your defined workload, maintain this level long enough to see if your system stabilizes. To finish up the test you should also ramp to see record how the system slows down as well.
+It is recommended to use an existing testing framework (see below). These tools will provide a method of both specifying the user activity scenarios and how to execute those at load.  Depending on the situation, it may be advisable to coordinate testing activities with the platform operations team.  
+
+It is common to slowly ramp up to your desired load to better replicate real world behavior. Once you have reached your defined workload, maintain this level long enough to see if your system stabilizes. To finish up the test you should also ramp to see record how the system slows down as well.
 
 You should also consider the origin of your load test traffic. Depending on the scope of the target system you may want to initiate from a different location to better replicate real world traffic such as from a different region.
 
 **Note:** Before starting please be aware of any restrictions on your network such as DDOS protection where you may need to notify a network administrator or apply for an exemption.
 
+**Note:** In general, the preferred approach to load testing would be the usage of a standard test framework such as the ones discussed below.  There are cases, however, where a custom test client may be advantageous.  Examples include batch oriented workloads that can be run under a single security context and the same test data can be re-used for multiple load tests.  In such a scenario it may be beneficial to develop a custom script that can be used interactively as well as non-interactively.
+
+### Analysis
+The analysis phase represents the work that brings all previous activities together:
+* Set aside time to allow for collection of new test data based on the analysis of the load tests.
+* Correlate application metrics and platform metrics to identify potential pitfalls and bottlenecks.
+* Include business stakeholders early in the analysis phase to validate application findings.  Include platform operations to validate platform findings.
+
+### Report writing
+Summarize your findings from the analysis phase.  Be sure to include application and platform enhancement suggestions, if any.
+
 ### Further Testing
 
 After completing your load test you should be set up to continue on to additional related testing such as;
@@ -73,7 +101,6 @@ In the case where a specific workload application is not being provided and the
 - **NGSA** ([Java](https://github.com/retaildevcrews/ngsa-java), [C#](https://github.com/retaildevcrews/ngsa-java)) - Intended for Kubernetes Platform and Monitoring Testing. Built on top of IMDB data store with many CRUD endpoints available. Does not need to have a live database connection.
 - **MockBin** (<https://github.com/Kong/mockbin>) - Allows you to generate custom endpoints to test, mock, and track HTTP requests & responses between libraries, sockets and APIs.
 
-
 ## Conclusion
 
 A load test is critical step to understand if a target system will be reliable under the expected real world traffic.

docs/continuous-delivery/recipes/terraform/terraform-structure-guidelines.md

@@ -57,7 +57,7 @@ For simple Terraform configurations, extensive unit testing might be overkill. I
 ### Key aspects to consider
 
 - **Syntax and validation**: Use `terraform fmt` and `terraform validate` to check the syntax and validate the Terraform configuration during development or in the deployment script / pipeline. This ensures that the configuration is correctly formatted and free of syntax errors.
-- **Deployment and existance**: Terraform providers, like the Azure provider, perform certain checks during the execution of terraform apply. If Terraform successfully applies a configuration, it typically means that the specified resources were created or modified as expected. In your code you can skip this validation and focus on particular resource configurations that are more critical, described in the next points.
+- **Deployment and existence**: Terraform providers, like the Azure provider, perform certain checks during the execution of terraform apply. If Terraform successfully applies a configuration, it typically means that the specified resources were created or modified as expected. In your code you can skip this validation and focus on particular resource configurations that are more critical, described in the next points.
 - **Resource properties that can break the functionality**: The expectation here is that we're not interested in testing each property of a resource, but to identify the ones that could cause an issue in the system if they are changed, such as access or network policies, service principal permissions and others.
 - **Validation of Key Vault contents**: Ensuring the presence of necessary keys, certificates, or secrets in the Azure Key Vault that are stored as part of resource configuration.
 - **Properties that can influence the cost or location**: This can be achieved by asserting the locations, service tiers, storage settings, depending on the properties available for the resources.