Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Milvus 2.5.4 Standalone Container is failing to start #40066

Open
1 task done
u-r-v opened this issue Feb 20, 2025 · 4 comments
Open
1 task done

[Bug]: Milvus 2.5.4 Standalone Container is failing to start #40066

u-r-v opened this issue Feb 20, 2025 · 4 comments
Assignees
Labels
kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@u-r-v
Copy link

u-r-v commented Feb 20, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.5.4
- Deployment mode(standalone or cluster):Standalone
- MQ type(rocksmq, pulsar or kafka):  
- SDK version(e.g. pymilvus v2.0.0rc2): 
- OS(Ubuntu or CentOS): Mac OS M2 Sonoma 14.6.1
- CPU/Memory: 12 Core/32GB
- GPU: 
- Others:

Current Behavior

When I follow the instructions provided in https://milvus.io/docs/install_standalone-docker.md and starting the container, getting following error

(Searched existing logs for the same but the resolution provided didn't work and it was for older version of milvus)

`ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.
ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.
2025/02/20 23:16:02 maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined

__  _________ _   ____  ______    

/ |/ / / /| | / / / / / __/
/ /|
/ // // /| |/ / // /\ \
// /////_/__/

Welcome to use Milvus!
Version: v2.5.4
Built: Thu Jan 23 03:41:40 UTC 2025
GitCommit: 6cdabd9
GoVersion: go version go1.22.0 linux/arm64

TotalMem: 6209081344
UsedMem: 41054208

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2025/02/20 23:16:02.975 +00:00] [INFO] [roles/roles.go:344] ["starting running Milvus components"]
[2025/02/20 23:16:02.975 +00:00] [INFO] [roles/roles.go:186] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2025/02/20 23:16:02.980 +00:00] [INFO] [config/manager.go:304] ["Get configuration by items failed"] [error="Read config failed: /milvus/configs/user.yaml: read /milvus/configs/user.yaml: is a directory"]
[2025/02/20 23:16:02.981 +00:00] [WARN] [paramtable/base_table.go:181] ["init baseTable with file failed"] [configFile="[milvus.yaml,_test.yaml,default.yaml,user.yaml]"] [error="failed to load FileSource cause: 5265616420636f6e666967206661696c65643a202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2072656164202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2069732061206469726563746f7279"]
[2025/02/20 23:16:02.982 +00:00] [DEBUG] [runtime/asm_arm64.s:1222] ["start refreshing configurations"] [source=FileSource]
[2025/02/20 23:16:02.982 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]

SIGNAL CATCH BY NON-GO SIGNAL HANDLER
SIGNO: 11; SIGNAME: Segmentation fault; SI_CODE: 1; SI_ADDR: 0x18
BACKTRACE:
github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer.func1
/workspace/source/pkg/util/etcd/etcd_server.go:49 pc=0x2aba124

[2025/02/20 23:16:10.146 +00:00] [DEBUG] [runtime/panic.go:770] ["Closing Natsmq!"]
[2025/02/20 23:16:10.146 +00:00] [INFO] [metrics/thread.go:53] ["thread watcher exit"]
[2025/02/20 23:16:10.146 +00:00] [INFO] [roles/roles.go:320] ["All cleanup done, handleSignals goroutine quit"]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2aba124]
`

Expected Behavior

Container should start.

Steps To Reproduce

#Download the script
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh

# Start the Docker container
 bash standalone_embed.sh start

Milvus Log

No response

Anything else?

No response

@u-r-v u-r-v added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 20, 2025
Copy link

dosubot bot commented Feb 20, 2025

Hey @u-r-v! 👋 I'm here to help you out with any bugs, questions, or even if you want to become a contributor. Let's squash those bugs together!

I found a similar discussion regarding a segmentation fault when running Milvus in a standalone mode on Ubuntu 22.04 in Docker. The issue is still open and unsolved [1].

For your specific issue, the error messages indicate a couple of potential problems:

  1. LD_PRELOAD Error: The error ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded suggests that the specified path for Jemalloc or other libraries is incorrect or the files are missing. Ensure that the path /milvus/lib/ contains the necessary libraries, especially libjemalloc.so.

  2. Configuration File Error: The error Read config failed: /milvus/configs/user.yaml: read /milvus/configs/user.yaml: is a directory indicates that the configuration file path is incorrect or the file is missing. Verify that the configuration files are correctly placed in the /milvus/configs/ directory and that user.yaml is a valid file, not a directory.

  3. Segmentation Fault: This might be related to the configuration or library loading issues. Ensure that all dependencies and configurations are correctly set up. You might want to check the prerequisites and compatibility requirements for running Milvus on Mac OS M2 Sonoma 14.6.1, such as Docker settings and required software versions [2][3].

Addressing these issues might help resolve the startup failure.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot dosubot bot added the severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. label Feb 20, 2025
@u-r-v
Copy link
Author

u-r-v commented Feb 20, 2025

@dosu

  1. LD_PRELOAD Error : This is first step of install not sure how /milvus/lib/ is loaded into.
  2. Configuration File Error: Read config failed: /milvus/configs/user.yaml: read /milvus/configs/user.yaml: is a directory, yes verified it is a file in local directory not sure whether docker is mounting as a directory. Not able to verify since the container is not starting.
  3. Segmentation Fault due to missing prerequisites: I don't have Docker Desktop but have Rancher Desktop. Would it be due to that?

Copy link

dosubot bot commented Feb 21, 2025

  1. LD_PRELOAD Error: The /milvus/lib/ directory is typically set up during the installation process, where necessary libraries are installed into this directory. If this is the first step of the installation, ensure that the installation script is correctly setting up this directory and its contents. You might want to check the installation script to verify that it includes the necessary steps to create and populate /milvus/lib/ with the required libraries [1].

  2. Configuration File Error: If the user.yaml file is being mounted as a directory, it could be due to an incorrect volume mount configuration in your Docker setup. Double-check the Docker Compose file or the command used to start the container to ensure that the file is being mounted correctly. Since the container isn't starting, you might need to inspect the Docker setup outside of the container to verify the mount configuration.

  3. Segmentation Fault and Rancher Desktop: Using Rancher Desktop instead of Docker Desktop could potentially cause issues if there are compatibility problems or differences in how they handle containerization. While specific issues with Rancher Desktop weren't found, it's possible that the segmentation fault could be related to this difference. Ensure that Rancher Desktop is configured to meet the same requirements as Docker Desktop, such as CPU and memory allocation, and verify that it supports all necessary Docker features used by Milvus.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 21, 2025
@Bhagyashreet20
Copy link

facing this issue even in docker runtime. @dosu proposed solutions do not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

3 participants