very slow performance due to excessive reconfiguration in modells #1921

juliantaylor · 2022-10-13T13:51:45Z

What happened (please include outputs or screenshots):

Running this simple script in a big cluster takes about 30 seconds to execute:

import kubernetes as k8s
k8s.config.load_kube_config()
apps = k8s.client.AppsV1Api()
print(k8s.__version__)
print(len(apps.list_replica_set_for_all_namespaces().items))

time python3 script:
24.2.0
3661

real	0m32.394s
user	0m17.764s
sys	0m11.235s

as you see it has very high system cpu usage
running this under a profiler shows following interesting things:

    957/1    0.002    0.000   32.177   32.177 {built-in method builtins.exec}
        1    0.173    0.173   32.177   32.177 test.py:1(<module>)
        1    0.000    0.000   31.563   31.563 apps_v1_api.py:3804(list_replica_set_for_all_namespaces)
        1    0.000    0.000   31.563   31.563 apps_v1_api.py:3838(list_replica_set_for_all_namespaces_with_http_info)
        1    0.000    0.000   31.563   31.563 api_client.py:305(call_api)
        1    0.018    0.018   31.563   31.563 api_client.py:120(__call_api)
        1    0.000    0.000   28.090   28.090 api_client.py:244(deserialize)
 780711/1    0.997    0.000   27.281   27.281 api_client.py:266(__deserialize)
 190593/1    0.934    0.000   27.281   27.281 api_client.py:620(__deserialize_model)
  36658/1    0.078    0.000   27.281   27.281 api_client.py:280(<listcomp>)
   190594    0.839    0.000   22.939    0.000 configuration.py:75(__init__)
   190594    0.108    0.000   10.911    0.000 context.py:41(cpu_count)
   190594   10.803    0.000   10.803    0.000 {built-in method posix.cpu_count}
   190597    0.300    0.000    8.327    0.000 configuration.py:253(debug)
   381195    0.216    0.000    7.443    0.000 __init__.py:1448(setLevel)
   381195    4.364    0.000    7.117    0.000 __init__.py:1403(_clear_cache)
...

190594 10.803 0.000 10.803 0.000 {built-in method posix.cpu_count}
10 seconds are spent running multiprocessing.cpu_count which accounts for most of the system usage

381195 0.216 0.000 7.443 0.000 __init__.py:1448(setLevel)
7 seconds are spent configuring logging

looking at what causes this appears to be following line in every model:

        if local_vars_configuration is None:
            local_vars_configuration = Configuration()

This runs the configuration function which sets up logging and calls multiprocessing.cpu_count

commenting the multiprocessing call confirms this, i then runs significantly faster:

real	0m11.964s
user	0m8.162s
sys	0m0.338s

Is there a way to avoid calling Configuration on every model init?

The text was updated successfully, but these errors were encountered:

juliantaylor · 2022-10-13T14:10:35Z

maybe something as simple as this is good?:

--- a/kubernetes/client/api_client.py
+++ b/kubernetes/client/api_client.py
@@ -638,6 +638,7 @@ class ApiClient(object):
                     value = data[klass.attribute_map[attr]]
                     kwargs[attr] = self.__deserialize(value, attr_type)
 
+        kwargs["local_vars_configuration"] = self.configuration
         instance = klass(**kwargs)
 
         if hasattr(instance, 'get_real_child_model'):

the script runs i 8 seconds with it.

yliaog · 2022-10-24T16:33:38Z

python/kubernetes/client/api_client.py

Line 8 in 4dddad8

Generated by: https://openapi-generator.tech

Generated by: https://openapi-generator.tech

any change to the file has to be done in the generator.

yliaog · 2022-10-24T16:42:27Z

/assign @yliaog

juliantaylor · 2022-11-04T14:36:36Z

I assume this file needs to be changed?
https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python-legacy/api_client.mustache

though which version? the code here was generated with 4.3.0 and the latest is 6.2.1 if I interpret #1943 correctly. Is the used version still updated?

what is the best way to get this fixed fast? this issue is quite severe

yliaog · 2022-11-04T15:53:58Z

please submit the fix in the openapi-generator, that is the best way to fix it.

juliantaylor · 2022-11-04T17:29:01Z

too which version? (in addition to master)

yliaog · 2022-11-04T18:24:36Z

to the latest, then backport it to the version this repo is using currently.

juliantaylor · 2022-11-04T18:27:57Z

which is?

The if not passed the models create a new configuration object which configures logging and determines cpu count every time. This causes extreme performance issues when deserializing larger sets of items. See also kubernetes-client/python#1921

juliantaylor · 2022-11-07T09:09:56Z

the generator has been updated on its main branch.
the generator does not support old versions OpenAPITools/openapi-generator#13922 (comment)

I strongly recommend to patch this locally for the current release

k8s-triage-robot · 2023-02-05T09:13:22Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

juliantaylor · 2023-02-05T13:59:01Z

/remove-lifecycle stale

amelim84 · 2023-02-16T16:34:03Z

@yliaog are there any plans to update the project to use the latest/fixed version openapi-generator?
We've been facing the issue described by @juliantaylor and have applied locally a patch with their suggestion, but would be ideal to have it on a release soon 🥺 :)

k8s-triage-robot · 2023-05-17T16:35:09Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2023-06-16T17:06:51Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

juliantaylor · 2023-06-16T17:13:23Z

/remove-lifecycle rotten

juliantaylor · 2023-11-24T14:40:08Z

this problem still exists in the latest version 28.1.0

if you have no plans to update the generator to the fixed version can you please add this patch locally? this is wasting loads of cpu cycles for every user.

k8s-triage-robot · 2024-02-22T15:04:33Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

juliantaylor · 2024-02-22T18:05:15Z

/remove-lifecycle stale

k8s-triage-robot · 2024-05-22T18:52:27Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-06-21T19:06:56Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

juliantaylor · 2024-07-04T12:36:30Z

/remove-lifecycle rotten

gibsondan · 2024-09-22T16:58:48Z

Thanks for fixing this in the OpenApi layer @juliantaylor - adding a +1 that we have applied a gross local change to our repo to apply the same fix and would be thrilled to be able to use the stock api client

zhuoqun-chen · 2024-10-22T18:37:12Z

Has this problem been solved? I also have the same issue when switching context using k8s.config.load_kube_config(), which can even take 5mins at most. Any idea on how to fix this in the user level?

k8s-triage-robot · 2025-01-20T19:19:11Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

juliantaylor · 2025-01-20T19:20:28Z

/remove-lifecycle stale

juliantaylor added the kind/bug Categorizes issue or PR as related to a bug. label Oct 13, 2022

juliantaylor mentioned this issue Oct 24, 2022

1.25 Release #1908

Closed

k8s-ci-robot assigned yliaog Oct 24, 2022

juliantaylor mentioned this issue Nov 4, 2022

[Python] pass api_client configuration to model deserialize OpenAPITools/openapi-generator#13922

Merged

5 tasks

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 5, 2023

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 5, 2023

tomplus mentioned this issue Mar 28, 2023

Slow deepcopy tomplus/kubernetes_asyncio#247

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 17, 2023

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 16, 2023

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 16, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 22, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 22, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 22, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 21, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 4, 2024

aantn mentioned this issue Sep 22, 2024

Extremely slow deserialization due to lock contention and logging module #1867

Open

gibsondan mentioned this issue Sep 22, 2024

Kubernetes api client doesn't pass configuration object through to the models it creates, causing excessive logging calls #2284

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 20, 2025

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 20, 2025

albertobruin linked a pull request Feb 12, 2025 that will close this issue

Avoid unncessary level setting #2346

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

very slow performance due to excessive reconfiguration in modells #1921

very slow performance due to excessive reconfiguration in modells #1921

juliantaylor commented Oct 13, 2022 •

edited

Loading

juliantaylor commented Oct 13, 2022

yliaog commented Oct 24, 2022

yliaog commented Oct 24, 2022

juliantaylor commented Nov 4, 2022

yliaog commented Nov 4, 2022

juliantaylor commented Nov 4, 2022 •

edited

Loading

yliaog commented Nov 4, 2022

juliantaylor commented Nov 4, 2022

juliantaylor commented Nov 7, 2022 •

edited

Loading

k8s-triage-robot commented Feb 5, 2023

juliantaylor commented Feb 5, 2023

amelim84 commented Feb 16, 2023

k8s-triage-robot commented May 17, 2023

k8s-triage-robot commented Jun 16, 2023

juliantaylor commented Jun 16, 2023

juliantaylor commented Nov 24, 2023

k8s-triage-robot commented Feb 22, 2024

juliantaylor commented Feb 22, 2024

k8s-triage-robot commented May 22, 2024

k8s-triage-robot commented Jun 21, 2024

juliantaylor commented Jul 4, 2024

gibsondan commented Sep 22, 2024

zhuoqun-chen commented Oct 22, 2024

k8s-triage-robot commented Jan 20, 2025

juliantaylor commented Jan 20, 2025

very slow performance due to excessive reconfiguration in modells #1921

very slow performance due to excessive reconfiguration in modells #1921

Comments

juliantaylor commented Oct 13, 2022 • edited Loading

juliantaylor commented Oct 13, 2022

yliaog commented Oct 24, 2022

yliaog commented Oct 24, 2022

juliantaylor commented Nov 4, 2022

yliaog commented Nov 4, 2022

juliantaylor commented Nov 4, 2022 • edited Loading

yliaog commented Nov 4, 2022

juliantaylor commented Nov 4, 2022

juliantaylor commented Nov 7, 2022 • edited Loading

k8s-triage-robot commented Feb 5, 2023

juliantaylor commented Feb 5, 2023

amelim84 commented Feb 16, 2023

k8s-triage-robot commented May 17, 2023

k8s-triage-robot commented Jun 16, 2023

juliantaylor commented Jun 16, 2023

juliantaylor commented Nov 24, 2023

k8s-triage-robot commented Feb 22, 2024

juliantaylor commented Feb 22, 2024

k8s-triage-robot commented May 22, 2024

k8s-triage-robot commented Jun 21, 2024

juliantaylor commented Jul 4, 2024

gibsondan commented Sep 22, 2024

zhuoqun-chen commented Oct 22, 2024

k8s-triage-robot commented Jan 20, 2025

juliantaylor commented Jan 20, 2025

juliantaylor commented Oct 13, 2022 •

edited

Loading

juliantaylor commented Nov 4, 2022 •

edited

Loading

juliantaylor commented Nov 7, 2022 •

edited

Loading