[Bug]: L0 delete compaction is not triggered again after the number of Flushed-L0 segments no longer increases

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240204-18b979d9-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):  pulsar  
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. deploy milvus and enable L0 segment
2. create collection with 2 shards -> index -> load -> insert 5m-128d data -> flush -> index again -> load
3. concurrent: search + upsert + flush

'concurrent_params': {'concurrent_number': 50,
                                                       'during_time': '10h',
                                                       'interval': 120,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 4,
                                                       'params': {'nq': 100,
                                                                  'top_k': 100,
                                                                  'search_param': {'ef': 128},
                                                                  'timeout': 120}},
                                                      {'type': 'flush',
                                                       'weight': 2,
                                                       'params': {'timeout': 120}},
                                                      {'type': 'upsert',
                                                       'weight': 4,
                                                       'params': {'nb': 100,
                                                                  'start_id': 0,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'timeout': 120}}]}

4. Starting around 2024-02-05 7:00, the L0 delete compaction is no longer triggered
   metrics of level-zero-upsert-13
   

Expected Behavior

No response

Steps To Reproduce

argo: https://argo-workflows.zilliz.cc/archived-workflows/qa/49d1b28c-5a67-4c14-ba28-f58b7a6d673b?nodeId=level-zero-stable-13a-2745047073

Milvus Log

pods:

level-zero-upsert-13-etcd-0                                       1/1     Running            0                 26h
level-zero-upsert-13-etcd-1                                       1/1     Running            0                 26h
level-zero-upsert-13-etcd-2                                       1/1     Running            0                 26h
level-zero-upsert-13-milvus-datanode-76574659b8-5wgxv             1/1     Running            0                 26h
level-zero-upsert-13-milvus-datanode-76574659b8-zsp4p             1/1     Running            0                 26h
level-zero-upsert-13-milvus-indexnode-776c568f59-cxrpv            1/1     Running            0                 26h
level-zero-upsert-13-milvus-indexnode-776c568f59-sndzg            1/1     Running            0                 26h
level-zero-upsert-13-milvus-mixcoord-86946954b7-j8f42             1/1     Running            0                 26h
level-zero-upsert-13-milvus-proxy-7c65458c4c-x9ksw                1/1     Running            0                 26h
level-zero-upsert-13-milvus-querynode-0-7dd76577c5-bv8pm          1/1     Running            0                 26h
level-zero-upsert-13-milvus-querynode-0-7dd76577c5-kwzrv          1/1     Running            0                 26h
level-zero-upsert-13-milvus-querynode-0-7dd76577c5-wsl4t          1/1     Running            0                 26h
level-zero-upsert-13-minio-0                                      1/1     Running            0                 26h
level-zero-upsert-13-minio-1                                      1/1     Running            0                 26h
level-zero-upsert-13-minio-2                                      1/1     Running            0                 26h
level-zero-upsert-13-minio-3                                      1/1     Running            0                 26h
level-zero-upsert-13-pulsar-bookie-0                              1/1     Running            0                 26h
level-zero-upsert-13-pulsar-bookie-1                              1/1     Running            0                 26h
level-zero-upsert-13-pulsar-bookie-2                              1/1     Running            0                 26h
level-zero-upsert-13-pulsar-bookie-init-7ljdm                     0/1     Completed          0                 26h
level-zero-upsert-13-pulsar-broker-0                              1/1     Running            0                 26h
level-zero-upsert-13-pulsar-proxy-0                               1/1     Running            0                 26h
level-zero-upsert-13-pulsar-pulsar-init-ntxbg                     0/1     Completed          0                 26h
level-zero-upsert-13-pulsar-recovery-0                            1/1     Running            0                 26h
level-zero-upsert-13-pulsar-zookeeper-0                           1/1     Running            0                 26h
level-zero-upsert-13-pulsar-zookeeper-1                           1/1     Running            0                 26h
level-zero-upsert-13-pulsar-zookeeper-2                           1/1     Running            0                 26h

Anything else?

No response

[xiaofan-luan]

/assign @ThreadDao

[ThreadDao]

fixed master-20240311-a99143dd-amd64