[Stress Testing] - Design and document an approach for maintaining performance tests

[noklam]

Description

Design and document an approach for maintaining performance tests to ensure their effectiveness and reliability over time

Tasks

• Analyze the current performance testing framework and identify areas for improvement.
• Develop a comprehensive maintenance plan, including regular updates and reviews.
• Document best practices for maintaining performance tests.
• Create guidelines for identifying and addressing performance test failures.
• Ensure compatibility with new software updates and changes.

The scope of the ticket include:

• Decide how to trigger these tests. Is it on demand, per
• Document how tests should be maintained or extend
• Test should be easy to run locally but also on CI (with Github Action)
  • Idea: Maybe we can use tag or branch name, for example, performance_xxxx_xxxx, if the branch name contains a keyword, that will trigger an additional performance CI automatically.
• (Bonus) How to report the performance benchmark? If it's run periodically, can we have it automatically report as a table? For example, consider some benchmark tools like: [Stress Testing] - Design and document an approach for maintaining performance tests  #4128 (comment)

As a long term goal, It would be really nice to keep a benchmark table like this, so when developer working on individual feature, they can easily compare the result and know if the PR has caused performance issue.

Release	task1 (ConfigLoader)	task2	task3
0.19.1	10s	10s	10s
0.19.2	10s	30s	9s
0.19.3	8s	10s	9s
0.19.4	11s	10s	9s

[astrojuanlu]

In a past life I was using asv for this https://github.com/airspeed-velocity/asv/

[ankatiyar]

My proposal for the performance test setup:

• Add component performance tests setup as described in Add performance testing setup to Kedro #4172.
  • This will contain tests in benchmarks/ directory within the Kedro repo.
  • The results are committed to a separate repository and github pages are built with the results.
  • This tracks the performance of components over time on main branch
• As a second step, we can add workflows that run specific component benchmarks for a specific PR against main to check if any regression is being added. Running benchmark tests for all PRs doesn't make sense and would be computationally expensive so we could add a mechanism to only trigger if needed, like leaving /performance as a comment on your PR to trigger them.
• A whole kedro project, the one that @lrcouto is working on, can be a separate repo with its own benchmarking set up for reporting end to end performance of various commands.

Tagging for opinions @noklam @merelcht @lrcouto @ElenaKhaustova @DimedS

[lrcouto]

I like the idea of triggering the workflows with a PR comment. I'd make it do it automatically for releases. I also like the idea of using github pages to visualize the results.

My only concern is if it would be cumbersome to mantain, since it'd be several separate repos.

[noklam]

The tests are now triggered as a Github Action job:

• pipeline-performance-test.yml
• benchmark-performance.yml

Wiki: https://github.com/kedro-org/kedro/wiki/The-Performance-Testing-Setup