bug: Can't push large model to kubernetes (yatai)

[1cadumagalhaes]

Describe the bug

Trying to deploy bento into kubernetes. When I try to push the model I get an error, and the upload speed is very low. I got similar issues with mlflow, and they were solved by setting some environment variables that configure timeouts and upload chunk sizes. I'm using GCS as the backend.

❯ bentoml models push retize-intention:g76w47h6o6qo6nro
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Failed to upload model "retize-intention:g76w47h6o6qo6nro"                                                                                         │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Uploading model "retize-intention:g76w47h6o6qo6nro" ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0% • 4.4/436.0 MB • 129.9 kB/s • 0:55:24
urllib3.exceptions.SSLError: [SYS] unknown error (_ssl.c:2406)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[my.host.com]', port=443): Max retries exceeded with url: /api/v1/model_repositories/retize-intention/models/g76w47h6o6qo6nro/upload (Caused by SSLError(SSLError(5, '[SYS] unknown error (_ssl.c:2406)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/retize/retize-comment-analysis/.venv/bin/bentoml", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml_cli/utils.py", line 362, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml_cli/utils.py", line 333, in wrapper
    return_value = func(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml_cli/utils.py", line 290, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml_cli/models.py", line 317, in push
    cloud_client.push_model(
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml/_internal/cloud/bentocloud.py", line 578, in push_model
    self._do_push_model(
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml/_internal/cloud/bentocloud.py", line 699, in _do_push_model
    raise e
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml/_internal/cloud/bentocloud.py", line 690, in _do_push_model
    yatai_rest_client.upload_model(
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/bentoml/_internal/cloud/client.py", line 377, in upload_model
    resp = self.session.put(
           ^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/requests/sessions.py", line 649, in put
    return self.request("PUT", url, data=data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/retize/retize-comment-analysis/.venv/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='[my.host.com]', port=443): Max retries exceeded with url: /api/v1/model_repositories/retize-intention/models/g76w47h6o6qo6nro/upload (Caused by SSLError(SSLError(5, '[SYS] unknown error (_ssl.c:2406)')))

To reproduce

No response

Expected behavior

No response

Environment

❯ bentoml env
zsh: correct 'env' to '.env' [nyae]? n

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.1.11
python: 3.12.6
platform: macOS-15.3.1-arm64-arm-64bit
uid_gid: 501:20

pip_packages

absl-py==2.1.0
accelerate==1.4.0
aiohappyeyeballs==2.5.0
aiohttp==3.11.13
aiosignal==1.3.2
alembic==1.15.1
altgraph==0.17.4
annotated-types==0.7.0
anyio==4.8.0
appdirs==1.4.4
asgiref==3.8.1
attrs==25.1.0
beautifulsoup4==4.13.3
bentoml==1.1.11
blinker==1.9.0
build==1.2.2.post1
cachetools==5.5.2
cattrs==23.1.2
certifi==2025.1.31
charset-normalizer==3.4.1
circus==0.19.0
click==8.1.8
click-option-group==0.5.6
cloudpickle==3.1.1
coloredlogs==15.0.1
colorlog==6.9.0
contourpy==1.3.1
cycler==0.12.1
databricks-sdk==0.45.0
datasets==3.3.2
deepmerge==2.0
Deprecated==1.2.18
dill==0.3.8
docker==7.1.0
fastapi==0.115.11
filelock==3.17.0
Flask==3.1.0
flatbuffers==25.2.10
fonttools==4.56.0
frozenlist==1.5.0
fs==2.4.16
fsspec==2024.12.0
future==1.0.0
gdown==5.2.0
gitdb==4.0.12
GitPython==3.1.44
google-api-core==2.24.2
google-auth==2.38.0
google-cloud-bigquery==3.30.0
google-cloud-core==2.4.3
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.69.1
graphene==3.4.3
graphql-core==3.2.6
graphql-relay==3.2.0
grpcio==1.71.0
grpcio-status==1.71.0
gunicorn==23.0.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.29.2
humanfriendly==10.0
hyperopt==0.2.7
idna==3.10
importlib-metadata==6.11.0
inflection==0.5.1
itsdangerous==2.2.0
Jinja2==3.1.6
joblib==1.4.2
kiwisolver==1.4.8
lightning-utilities==0.14.0
macholib==1.16.3
Mako==1.3.9
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.10.1
mdurl==0.1.2
mlflow==2.20.3
mlflow-skinny==2.20.3
mpmath==1.3.0
multidict==6.1.0
multiprocess==0.70.16
networkx==3.4.2
nlpaug==1.1.11
nltk==3.9.1
numpy==2.2.3
nvidia-ml-py==11.525.150
onnx==1.17.0
onnx2pytorch==0.5.1
onnxruntime==1.21.0
opentelemetry-api==1.20.0
opentelemetry-instrumentation==0.41b0
opentelemetry-instrumentation-aiohttp-client==0.41b0
opentelemetry-instrumentation-asgi==0.41b0
opentelemetry-sdk==1.20.0
opentelemetry-semantic-conventions==0.41b0
opentelemetry-util-http==0.41b0
optimum==1.24.0
optuna==4.2.1
packaging==24.2
pandas==2.2.3
pathspec==0.12.1
pillow==11.1.0
pip-requirements-parser==32.0.1
pip-tools==7.4.1
prometheus_client==0.21.1
propcache==0.3.0
proto-plus==1.26.1
protobuf==5.29.3
psutil==7.0.0
py4j==0.10.9.9
pyarrow==19.0.1
pyasn1==0.6.1
pyasn1_modules==0.4.1
pydantic==2.10.6
pydantic-settings==2.8.1
pydantic_core==2.27.2
Pygments==2.19.1
pyinstaller==6.12.0
pyinstaller-hooks-contrib==2025.1
pyparsing==3.2.1
pyproject_hooks==1.2.0
pyrsmi==0.2.0
PySocks==1.7.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-json-logger==3.3.0
python-multipart==0.0.20
pytorch-lightning==2.5.0.post0
pytz==2025.1
PyYAML==6.0.2
pyzmq==26.2.1
regex==2024.11.6
requests==2.32.3
rich==13.9.4
rsa==4.9
safetensors==0.5.3
schema==0.7.7
scikit-learn==1.6.1
scipy==1.15.2
sentencepiece==0.2.0
setuptools==76.0.0
simple-di==0.1.5
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
soupsieve==2.6
SQLAlchemy==2.0.38
sqlparse==0.5.3
starlette==0.46.1
sympy==1.13.1
tensorboard==2.19.0
tensorboard-data-server==0.7.2
tensorboardX==2.6.2.2
threadpoolctl==3.5.0
tiktoken==0.9.0
tokenizers==0.21.0
torch==2.6.0
torchmetrics==1.6.2
torchvision==0.21.0
tornado==6.4.2
tqdm==4.67.1
transformers==4.49.0
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
uvicorn==0.34.0
watchfiles==1.0.4
Werkzeug==3.1.3
wheel==0.45.1
wrapt==1.17.2
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0