Needs clarification on QueryNode Disk Usage

[Ald392]

I am using milvus 2.4.0. And the following is QueryNode configuration. I would like to know if there would be significant local disk consumption in this QueryNode when I am not using any Disk Indexes. The reason for that question was that I was getting a low on ephemeral storage error with this config and I couldn't see anything being stored in the localstorage path (/var/lib/milvus/data/) inside the QueryNode. (Verified using "du" and "ls -alrt" commands)

Then I tried enabling mmap and ran my script which did collection creation, insertion, load, search etc. Now I observed lot of directories being created at /var/lib/milvus/data/mmap but most of them were empty or had a directory which was empty. Why is it like that ?

queryNode:
  cache:
    enabled: true
    memoryLimit: 2147483648 # 2 GB, 2 * 1024 *1024 *1024
  dataSync:
    flowGraph:
      maxQueueLength: 16 # Maximum length of task queue in flowgraph
      maxParallelism: 1024 # Maximum number of tasks executed in parallel in the flowgraph
  stats:
    publishInterval: 1000 # Interval for querynode to report node information (milliseconds)
  segcore:
    cgoPoolSizeRatio: 2.0 # cgo pool size ratio to max read concurrency
    knowhereThreadPoolNumRatio: 4
    # Use more threads to make better use of SSD throughput in disk index.
    # This parameter is only useful when enable-disk = true.
    # And this value should be a number greater than 1 and less than 32.
    chunkRows: 1024 # The number of vectors in a chunk.
    growing: # growing a vector index for growing segment to accelerate search
      enableIndex: true
      nlist: 128 # growing segment index nlist
      nprobe: 16 # nprobe to search growing segment, based on your accuracy requirement, must smaller than nlist
  loadMemoryUsageFactor: 1 # The multiply factor of calculating the memory usage while loading segments
  enableDisk: false # enable querynode load disk index, and search on disk index
  maxDiskUsagePercentage: 95
  grouping:
    enabled: true
    maxNQ: 1000
    topKMergeRatio: 20
  scheduler:
    receiveChanSize: 10240
    unsolvedQueueSize: 10240
    # maxReadConcurrentRatio is the concurrency ratio of read task (search task and query task).
    # Max read concurrency would be the value of runtime.NumCPU * maxReadConcurrentRatio.
    # It defaults to 2.0, which means max read concurrency would be the value of runtime.NumCPU * 2.
    # Max read concurrency must greater than or equal to 1, and less than or equal to runtime.NumCPU * 100.
    # (0, 100]
    maxReadConcurrentRatio: 1
    cpuRatio: 10 # ratio used to estimate read task cpu usage.
    maxTimestampLag: 86400
    # read task schedule policy: fifo(by default), user-task-polling.
    scheduleReadPolicy:
      # fifo: A FIFO queue support the schedule.
      # user-task-polling:
      #     The user's tasks will be polled one by one and scheduled.
      #     Scheduling is fair on task granularity.
      #     The policy is based on the username for authentication.
      #     And an empty username is considered the same user.
      #     When there are no multi-users, the policy decay into FIFO
      name: fifo
      maxPendingTask: 10240
      # user-task-polling configure:
      taskQueueExpire: 60 # 1 min by default, expire time of inner user task queue since queue is empty.
      enableCrossUserGrouping: false # false by default Enable Cross user grouping when using user-task-polling policy. (close it if task of any user can not merge others).
      maxPendingTaskPerUser: 1024 # 50 by default, max pending task in scheduler per user.

[yhmo]

Read this discussion: #33621
The directories are empty but the data is still on disk. The file is invisible since unlink() is called after mmap.
Our engineer might make some changes to keep the files visible in future.

[Ald392]

Thanks @yhmo this clarified how QueryNode uses disk when using mmap. I had some additional doubts :

• From what I understand QueryNode uses significant local disk storage for these operations : mmap, chunk cache, Disk indexes and when a query involves output fields it will be loading the necessary output fields from bucket to query node disk. Like if the query needs entire collection data along with all the output fields then all data of the collection in the bucket will be downloaded to disk.
  And each of these operations would be downloading the data to local storage path (/var/lib/milvus/data/). And it will be invisible here except for some empty directories.

• So using the QueryNode configuration I had shared, I told that I was getting low ephemeral storage error. Since here mmap is not enabled and cache is also limited to 2Gb can it be due to my queries requiring significant data to be loaded into the disk from bucket.
  Or this loading of collection data to disk done in the same 2GB memory limit of cache. And my problem might be due to some different reason.

• Is there a way to track the disk usage by /var/lib/milvus/data/ ? ("du" and "ls -lart" will not work as unlink() is used)

• Is the local storage path (/var/lib/milvus/data/) used in any other nodes other than the QueryNode ?

[Ald392]

I was encountering this when using both IVF_SQ8 and HNSW indexing and the "warmup" parameter of cache was not defined, so I think it will be using the default value "disable".

For the query node pod the resources config used was the following

resources:
      limits:
        ephemeral-storage: 2Gi
        memory: 28Gi
      requests:
        cpu: "6"
        ephemeral-storage: 1.5Gi
        memory: 27Gi

I think this limiting of ephemeral-storage to 2GB and configuring QueryNode cache as 2GB in the milvus.yaml might have been the cause of the issue. Triggering low on ephemeral storage error in situations where the QueryNode used significant portions of its allocated disk cache space that then gets accounted in ephemeral-storage usage of the pod.
This is based on the assumption that the cache stores data other than the original vector data. Because in the case of HNSW, original vector data is there in the index itself so no downloading is required. Am I right ?

[yhmo]

In v2.4.0, the default value of warmup is "async"

milvus/pkg/util/paramtable/component_param.go

Line 2191 in ffb6edd

DefaultValue: "async",

From v2.4.3, the dafault value of warmup was changed to "disable"

milvus/pkg/util/paramtable/component_param.go

Line 2350 in 7384bfe

DefaultValue: "disable",

Your milvus version is v2.4.0. So, once you have called collection.load(), the original vectors data will be downloaded to local and mmap to memory.
You can manually set it to "disable" and restart the milvus server.

[Ald392]

Thanks @yhmo I will try that.

• Can you verify whether this disk space would be used for downloading columns other than the original vector data.
• I believe the documentation needs update as its still mentioning "disable" as default value for warmup: https://milvus.io/docs/chunk_cache.md

[yhmo]

"warmup" is only for vector original data.
The description in the milvus.yaml:
If set to "sync" or "async" the original vector data will be synchronously/asynchronously loaded into the chunk cache during the load process. This approach has the potential to substantially reduce query/search latency for a specific duration post-load, albeit accompanied by a concurrent increase in disk usage

The document says the default value is "disable" based on the latest version behavior. It is correct. Only v2.4.0 ~ v2.4.2 are "async".

[Ald392]

Thanks for the help.

[navneetrajput06]

@yhmo We have increased ephemeral storage to 30GB but still pods are getting crashed with limit exceeds. below is the error for your reference. because of this nodes are going in Evicted and we can not increase it further as total limit on node is 50GB only.

"Warning Evicted 10m (x7 over 14m) kubelet Pod ephemeral local storage usage exceeds the total limit of containers 30Gi."

Why query node writing so much local data and can we set some buffer setting to ignore local storage rather if this can be processed in memory?
Below are the query node resource settings:
queryNode:
paused: false
replicas: 10
resources:
limits:
cpu: "16"
ephemeral-storage: 30Gi
memory: 384Gi