Out-of-memory during 'compact' the results of bulk-load (~=10TB) #11809

ski422 · 2023-09-09T07:32:47Z

Hi to all Rocksdb users and dev group!

I experienced the out-of-memory while running 'compact' to compact the results of bulk-load (all SSTables are in level-0), and if anyone has experienced the same phenomenon, I'd like to ask for advice.

Goal

Loading 10TB of data on multiple SSDs in a single machine (all SSDs are controlled as raid-0). As a result of bulk-load, about 180~190 thousand of SSTables are written in level-0.

Behavior

After the bulk-load was finished, I tried to run 'compact' as described in ./tools/benchmark.sh script. However, the compaction tried to read the all SSTables into memory (I have 500GB), and it finally caused out-of-memory.

Question

Why does 'compact' try to read the entire level-0? Because of the 'compact' calls CompactRange() without specified range? or will the same phenomenon occur even if I designate the compaction range (since most of the key ranges overlap in level-0)?

It would be a great help if you could share your experience of trying to load more than a few TB of data.

anand1976 · 2023-09-11T22:22:12Z

@ski422 Can you specify the RocksDB release you saw this on, as well as the exact command, and the OPTIONS and LOG files?

ski422 · 2023-09-12T01:55:47Z

@ski422 Can you specify the RocksDB release you saw this on, as well as the exact command, and the OPTIONS and LOG files?

I used RocksDB version 7.10.0 and executed ./tools/benchmark.sh as follow.

$./tools/benchmark.sh bulkload

As you can find in benchmark.sh, bulk-load executes 'fillrandom' without auto-compaction and 'compact' to compact all files in level-0.

Summarized adjusted options are as follows.

key_size=48
value_size=43
num_keys=120825453602 (approximated size of the total data=10TB)
target_file_mb=64
compression_type=none
max_background_jobs=64
max_write_buffer_number=1024
cache_type=hyper_clock_cache
partition_index=true
partition_index_and_filters=true

After the fillrandom is finished RocksDB created about 190 thousand of SST files in level-0. However, when the 'compact' is executed, it tries to read all SST files and it arises out-of-memory error. The OPTIONS and LOG files of 'compact' is as follows.

.
.# This is a RocksDB option file.
.#
.# For detailed file format spec, please refer to the example file
.# in examples/rocksdb_option_file_example.ini
.#

[Version]
rocksdb_version=7.10.0
options_file_version=1.1

[DBOptions]
max_background_flushes=-1
compaction_readahead_size=2097152
wal_bytes_per_sync=0
bytes_per_sync=1048576
max_open_files=-1
stats_history_buffer_size=1048576
stats_dump_period_sec=600
stats_persist_period_sec=600
delete_obsolete_files_period_micros=21600000000
max_total_wal_size=0
strict_bytes_per_sync=false
delayed_write_rate=8388608
avoid_flush_during_shutdown=false
writable_file_max_buffer_size=1048576
max_subcompactions=1
max_background_compactions=-1
max_background_jobs=64
lowest_used_cache_tier=kNonVolatileBlockTier
bgerror_resume_retry_interval=1000000
max_bgerror_resume_count=2147483647
best_efforts_recovery=false
write_dbid_to_manifest=false
avoid_unnecessary_blocking_io=false
atomic_flush=false
log_readahead_size=0
dump_malloc_stats=true
info_log_level=DEBUG_LEVEL
write_thread_max_yield_usec=100
max_write_batch_group_size_bytes=1048576
wal_compression=kNoCompression
write_thread_slow_yield_usec=3
enable_pipelined_write=true
persist_stats_to_disk=false
max_manifest_file_size=1073741824
WAL_size_limit_MB=0
fail_if_options_file_error=false
max_log_file_size=0
manifest_preallocation_size=4194304
listeners={ErrorHandlerListener}
log_file_time_to_roll=0
allow_data_in_errors=false
WAL_ttl_seconds=0
recycle_log_file_num=0
file_checksum_gen_factory=nullptr
keep_log_file_num=1000
db_write_buffer_size=0
table_cache_numshardbits=4
use_adaptive_mutex=false
allow_ingest_behind=false
skip_checking_sst_file_sizes_on_db_open=false
skip_stats_update_on_db_open=false
random_access_max_buffer_size=1048576
access_hint_on_compaction_start=NORMAL
allow_concurrent_memtable_write=true
track_and_verify_wals_in_manifest=false
paranoid_checks=true
max_file_opening_threads=16
verify_sst_unique_id_in_manifest=true
avoid_flush_during_recovery=false
flush_verify_memtable_count=true
db_host_id=hostname
error_if_exists=false
wal_recovery_mode=kPointInTimeRecovery
enable_thread_tracking=false
is_fd_close_on_exec=true
enforce_single_del_contracts=true
create_missing_column_families=true
create_if_missing=true
use_fsync=false
wal_filter=nullptr
allow_2pc=false
use_direct_io_for_flush_and_compaction=true
manual_wal_flush=false
enable_write_thread_adaptive_yield=true
use_direct_reads=true
allow_mmap_writes=false
allow_fallocate=true
two_write_queues=false
allow_mmap_reads=false
unordered_write=false
advise_random_on_open=true

[CFOptions "default"]
memtable_protection_bytes_per_key=0
sample_for_compression=0
blob_file_starting_level=0
blob_compaction_readahead_size=0
blob_garbage_collection_force_threshold=1.000000
enable_blob_garbage_collection=false
min_blob_size=0
last_level_temperature=kUnknown
enable_blob_files=false
target_file_size_base=67108864
max_sequential_skip_in_iterations=8
prepopulate_blob_cache=kDisable
compaction_options_fifo={allow_compaction=true;age_for_warm=0;max_table_files_size=0;}
max_bytes_for_level_multiplier=8.000000
max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
max_bytes_for_level_base=1073741824
experimental_mempurge_threshold=0.000000
write_buffer_size=67108864
bottommost_compression=kDisableCompressionOption
prefix_extractor=nullptr
blob_file_size=268435456
memtable_huge_page_size=0
max_successive_merges=0
compression_opts={max_dict_buffer_bytes=0;use_zstd_dict_trainer=true;enabled=false;parallel_threads=1;zstd_max_train_bytes=0;strategy=0;max_dict_bytes=0;level=32767;window_bits=-14;}
arena_block_size=1048576
memtable_whole_key_filtering=false
target_file_size_multiplier=1
max_write_buffer_number=8
blob_compression_type=kNoCompression
compression=kNoCompression
level0_stop_writes_trigger=30
level0_slowdown_writes_trigger=20
level0_file_num_compaction_trigger=4
ignore_max_compaction_bytes_for_input=true
max_compaction_bytes=1677721600
compaction_options_universal={allow_trivial_move=false;incremental=false;stop_style=kCompactionStopStyleTotalSize;compression_size_percent=-1;max_size_amplification_percent=200;max_merge_width=4294967295;min_merge_width=2;size_ratio=1;}
memtable_prefix_bloom_size_ratio=0.000000
hard_pending_compaction_bytes_limit=137438953472
bottommost_compression_opts={max_dict_buffer_bytes=0;use_zstd_dict_trainer=true;enabled=false;parallel_threads=1;zstd_max_train_bytes=0;strategy=0;max_dict_bytes=0;level=32767;window_bits=-14;}
blob_garbage_collection_age_cutoff=0.250000
ttl=2592000
soft_pending_compaction_bytes_limit=68719476736
inplace_update_num_locks=10000
paranoid_file_checks=false
check_flush_compaction_key_order=true
periodic_compaction_seconds=0
disable_auto_compactions=false
report_bg_io_stats=false
compaction_pri=kMinOverlappingRatio
compaction_style=kCompactionStyleLevel
merge_operator=nullptr
table_factory=BlockBasedTable
memtable_factory=SkipListFactory
comparator=leveldb.BytewiseComparator
compaction_filter_factory=nullptr
num_levels=7
min_write_buffer_number_to_merge=1
bloom_locality=0
max_write_buffer_size_to_maintain=0
max_write_buffer_number_to_maintain=0
sst_partitioner_factory=nullptr
preserve_internal_time_seconds=0
preclude_last_level_data_seconds=0
optimize_filters_for_hits=false
level_compaction_dynamic_file_size=true
memtable_insert_with_hint_prefix_extractor=nullptr
level_compaction_dynamic_level_bytes=false
inplace_update_support=false
compaction_filter=nullptr
force_consistency_checks=true

[TableOptions/BlockBasedTable "default"]
num_file_reads_for_auto_readahead=2
initial_auto_readahead_size=8192
metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;}
enable_index_compression=true
pin_top_level_index_and_filter=false
read_amp_bytes_per_bit=0
verify_compression=false
prepopulate_block_cache=kDisable
format_version=5
partition_filters=true
metadata_block_size=4096
max_auto_readahead_size=262144
index_block_restart_interval=1
block_size_deviation=10
block_size=4096
detect_filter_construct_corruption=false
no_block_cache=false
checksum=kCRC32c
filter_policy=bloomfilter:10:false
data_block_hash_table_util_ratio=0.750000
block_restart_interval=16
index_type=kTwoLevelIndexSearch
pin_l0_filter_and_index_blocks_in_cache=false
data_block_index_type=kDataBlockBinarySearch
cache_index_and_filter_blocks_with_high_priority=true
whole_key_filtering=true
index_shortening=kShortenSeparatorsAndSuccessor
cache_index_and_filter_blocks=false
block_align=false
optimize_filters_for_memory=false
flush_block_policy_factory=FlushBlockBySizePolicyFactory

===============================LOG file=================================
2023/09/11-12:41:03.690589 140221488028480 RocksDB version: 7.10.0
2023/09/11-12:41:03.690638 140221488028480 Git sha b42cf1f0170618382111a709883e59f87743ff33
2023/09/11-12:41:03.690641 140221488028480 Compile date 2023-09-08 15:02:12
2023/09/11-12:41:03.690644 140221488028480 DB SUMMARY
2023/09/11-12:41:03.690646 140221488028480 DB Session ID: QPQJ75JJHM2I6JLN1E0H
2023/09/11-12:41:03.809511 140221488028480 CURRENT file: CURRENT
2023/09/11-12:41:03.809517 140221488028480 IDENTITY file: IDENTITY
2023/09/11-12:41:03.809525 140221488028480 MANIFEST file: MANIFEST-198126 size: 35338749 Bytes
2023/09/11-12:41:03.809527 140221488028480 SST files in /home/smrc/nvme_raid/10TB_load_vanilla dir, Total Num: 198107, files: 000008.sst 000009.sst 000010.sst 000011.sst 000012.sst 000013.sst 000014.sst 000015.sst 000016.sst
2023/09/11-12:41:03.809529 140221488028480 Write Ahead Log file in /home/smrc/nvme_raid/10TB_load_vanilla: 000004.log size: 0 ; 198117.log size: 0 ; 198121.log size: 0 ; 198125.log size: 0 ;
2023/09/11-12:41:03.810031 140221488028480 Options.error_if_exists: 0
2023/09/11-12:41:03.810034 140221488028480 Options.create_if_missing: 1
2023/09/11-12:41:03.810035 140221488028480 Options.paranoid_checks: 1
2023/09/11-12:41:03.810036 140221488028480 Options.flush_verify_memtable_count: 1
2023/09/11-12:41:03.810037 140221488028480 Options.track_and_verify_wals_in_manifest: 0
2023/09/11-12:41:03.810038 140221488028480 Options.verify_sst_unique_id_in_manifest: 1
2023/09/11-12:41:03.810039 140221488028480 Options.env: 0x55aec549e120
2023/09/11-12:41:03.810040 140221488028480 Options.fs: PosixFileSystem
2023/09/11-12:41:03.810041 140221488028480 Options.info_log: 0x55aec5b27cb0
2023/09/11-12:41:03.810042 140221488028480 Options.max_file_opening_threads: 16
2023/09/11-12:41:03.810043 140221488028480 Options.statistics: (nil)
2023/09/11-12:41:03.810044 140221488028480 Options.use_fsync: 0
2023/09/11-12:41:03.810045 140221488028480 Options.max_log_file_size: 0
2023/09/11-12:41:03.810046 140221488028480 Options.max_manifest_file_size: 1073741824
2023/09/11-12:41:03.810047 140221488028480 Options.log_file_time_to_roll: 0
2023/09/11-12:41:03.810048 140221488028480 Options.keep_log_file_num: 1000
2023/09/11-12:41:03.810049 140221488028480 Options.recycle_log_file_num: 0
2023/09/11-12:41:03.810050 140221488028480 Options.allow_fallocate: 1
2023/09/11-12:41:03.810051 140221488028480 Options.allow_mmap_reads: 0
2023/09/11-12:41:03.810052 140221488028480 Options.allow_mmap_writes: 0
2023/09/11-12:41:03.810053 140221488028480 Options.use_direct_reads: 1
2023/09/11-12:41:03.810054 140221488028480 Options.use_direct_io_for_flush_and_compaction: 1
2023/09/11-12:41:03.810055 140221488028480 Options.create_missing_column_families: 1
2023/09/11-12:41:03.810056 140221488028480 Options.db_log_dir:
2023/09/11-12:41:03.810057 140221488028480 Options.wal_dir:
2023/09/11-12:41:03.810058 140221488028480 Options.table_cache_numshardbits: 4
2023/09/11-12:41:03.810059 140221488028480 Options.WAL_ttl_seconds: 0
2023/09/11-12:41:03.810060 140221488028480 Options.WAL_size_limit_MB: 0
2023/09/11-12:41:03.810060 140221488028480 Options.max_write_batch_group_size_bytes: 1048576
2023/09/11-12:41:03.810061 140221488028480 Options.manifest_preallocation_size: 4194304
2023/09/11-12:41:03.810062 140221488028480 Options.is_fd_close_on_exec: 1
2023/09/11-12:41:03.810063 140221488028480 Options.advise_random_on_open: 1
2023/09/11-12:41:03.810064 140221488028480 Options.db_write_buffer_size: 0
2023/09/11-12:41:03.810065 140221488028480 Options.write_buffer_manager: 0x55aec5b27e80
2023/09/11-12:41:03.810066 140221488028480 Options.access_hint_on_compaction_start: 1
2023/09/11-12:41:03.810067 140221488028480 Options.random_access_max_buffer_size: 1048576
2023/09/11-12:41:03.810068 140221488028480 Options.use_adaptive_mutex: 0
2023/09/11-12:41:03.810069 140221488028480 Options.rate_limiter: (nil)
2023/09/11-12:41:03.810071 140221488028480 Options.sst_file_manager.rate_bytes_per_sec: 0
2023/09/11-12:41:03.810072 140221488028480 Options.wal_recovery_mode: 2
2023/09/11-12:41:03.810073 140221488028480 Options.enable_thread_tracking: 0
2023/09/11-12:41:03.810074 140221488028480 Options.enable_pipelined_write: 1
2023/09/11-12:41:03.810075 140221488028480 Options.unordered_write: 0
2023/09/11-12:41:03.810075 140221488028480 Options.allow_concurrent_memtable_write: 1
2023/09/11-12:41:03.810076 140221488028480 Options.enable_write_thread_adaptive_yield: 1
2023/09/11-12:41:03.810077 140221488028480 Options.write_thread_max_yield_usec: 100
2023/09/11-12:41:03.810078 140221488028480 Options.write_thread_slow_yield_usec: 3
2023/09/11-12:41:03.810079 140221488028480 Options.row_cache: None
2023/09/11-12:41:03.810080 140221488028480 Options.wal_filter: None
2023/09/11-12:41:03.810081 140221488028480 Options.avoid_flush_during_recovery: 0
2023/09/11-12:41:03.810082 140221488028480 Options.allow_ingest_behind: 0
2023/09/11-12:41:03.810083 140221488028480 Options.two_write_queues: 0
2023/09/11-12:41:03.810084 140221488028480 Options.manual_wal_flush: 0
2023/09/11-12:41:03.810085 140221488028480 Options.wal_compression: 0
2023/09/11-12:41:03.810086 140221488028480 Options.atomic_flush: 0
2023/09/11-12:41:03.810086 140221488028480 Options.avoid_unnecessary_blocking_io: 0
2023/09/11-12:41:03.810087 140221488028480 Options.persist_stats_to_disk: 0
2023/09/11-12:41:03.810088 140221488028480 Options.write_dbid_to_manifest: 0
2023/09/11-12:41:03.810089 140221488028480 Options.log_readahead_size: 0
2023/09/11-12:41:03.810090 140221488028480 Options.file_checksum_gen_factory: Unknown
2023/09/11-12:41:03.810091 140221488028480 Options.best_efforts_recovery: 0
2023/09/11-12:41:03.810092 140221488028480 Options.max_bgerror_resume_count: 2147483647
2023/09/11-12:41:03.810093 140221488028480 Options.bgerror_resume_retry_interval: 1000000
2023/09/11-12:41:03.810094 140221488028480 Options.allow_data_in_errors: 0
2023/09/11-12:41:03.810095 140221488028480 Options.db_host_id: hostname
2023/09/11-12:41:03.810096 140221488028480 Options.enforce_single_del_contracts: true
2023/09/11-12:41:03.810097 140221488028480 Options.max_background_jobs: 64
2023/09/11-12:41:03.810098 140221488028480 Options.max_background_compactions: -1
2023/09/11-12:41:03.810099 140221488028480 Options.max_subcompactions: 1
2023/09/11-12:41:03.810100 140221488028480 Options.avoid_flush_during_shutdown: 0
2023/09/11-12:41:03.810101 140221488028480 Options.writable_file_max_buffer_size: 1048576
2023/09/11-12:41:03.810101 140221488028480 Options.delayed_write_rate : 8388608
2023/09/11-12:41:03.810102 140221488028480 Options.max_total_wal_size: 0
2023/09/11-12:41:03.810103 140221488028480 Options.delete_obsolete_files_period_micros: 21600000000
2023/09/11-12:41:03.810104 140221488028480 Options.stats_dump_period_sec: 600
2023/09/11-12:41:03.810105 140221488028480 Options.stats_persist_period_sec: 600
2023/09/11-12:41:03.810106 140221488028480 Options.stats_history_buffer_size: 1048576
2023/09/11-12:41:03.810107 140221488028480 Options.max_open_files: -1
2023/09/11-12:41:03.810109 140221488028480 Options.bytes_per_sync: 1048576
2023/09/11-12:41:03.810110 140221488028480 Options.wal_bytes_per_sync: 0
2023/09/11-12:41:03.810111 140221488028480 Options.strict_bytes_per_sync: 0
2023/09/11-12:41:03.810112 140221488028480 Options.compaction_readahead_size: 2097152
2023/09/11-12:41:03.810113 140221488028480 Options.max_background_flushes: -1
2023/09/11-12:41:03.810113 140221488028480 Compression algorithms supported:
2023/09/11-12:41:03.810115 140221488028480 kZSTDNotFinalCompression supported: 0
2023/09/11-12:41:03.810116 140221488028480 kZSTD supported: 0
2023/09/11-12:41:03.810117 140221488028480 kXpressCompression supported: 0
2023/09/11-12:41:03.810119 140221488028480 kLZ4HCCompression supported: 0
2023/09/11-12:41:03.810120 140221488028480 kLZ4Compression supported: 0
2023/09/11-12:41:03.810121 140221488028480 kBZip2Compression supported: 0
2023/09/11-12:41:03.810122 140221488028480 kZlibCompression supported: 0
2023/09/11-12:41:03.810123 140221488028480 kSnappyCompression supported: 1
2023/09/11-12:41:03.810125 140221488028480 Fast CRC32 supported: Supported on x86
2023/09/11-12:41:03.810126 140221488028480 DMutex implementation: pthread_mutex_t
2023/09/11-12:41:03.810325 140221488028480 [/version_set.cc:5802] Recovering from manifest file: /home/smrc/nvme_raid/10TB_load_vanilla/MANIFEST-198126
2023/09/11-12:41:03.810586 140221488028480 [/column_family.cc:632] --------------- Options for column family [default]:
2023/09/11-12:41:03.810589 140221488028480 Options.comparator: leveldb.BytewiseComparator
2023/09/11-12:41:03.810590 140221488028480 Options.merge_operator: None
2023/09/11-12:41:03.810591 140221488028480 Options.compaction_filter: None
2023/09/11-12:41:03.810592 140221488028480 Options.compaction_filter_factory: None
2023/09/11-12:41:03.810593 140221488028480 Options.sst_partitioner_factory: None
2023/09/11-12:41:03.810594 140221488028480 Options.memtable_factory: SkipListFactory
2023/09/11-12:41:03.810595 140221488028480 Options.table_factory: BlockBasedTable
2023/09/11-12:41:03.810618 140221488028480 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aec5b28e70)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 0
index_type: 2
data_block_index_type: 0
index_shortening: 2
data_block_hash_table_util_ratio: 0.750000
checksum: 1
no_block_cache: 0
block_cache: 0x55aec5adb440
block_cache_name: HyperClockCache
block_cache_options:
capacity : 274877906944
num_shard_bits : 6
strict_capacity_limit : 0
memory_allocator : None
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 1
use_delta_encoding: 1
filter_policy: bloomfilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 5
enable_index_compression: 1
block_align: 0
max_auto_readahead_size: 262144
prepopulate_block_cache: 0
initial_auto_readahead_size: 8192
num_file_reads_for_auto_readahead: 2
2023/09/11-12:41:03.810620 140221488028480 Options.write_buffer_size: 67108864
2023/09/11-12:41:03.810621 140221488028480 Options.max_write_buffer_number: 8
2023/09/11-12:41:03.810622 140221488028480 Options.compression: NoCompression
2023/09/11-12:41:03.810624 140221488028480 Options.bottommost_compression: Disabled
2023/09/11-12:41:03.810625 140221488028480 Options.prefix_extractor: nullptr
2023/09/11-12:41:03.810626 140221488028480 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2023/09/11-12:41:03.810627 140221488028480 Options.num_levels: 7
2023/09/11-12:41:03.810628 140221488028480 Options.min_write_buffer_number_to_merge: 1
2023/09/11-12:41:03.810629 140221488028480 Options.max_write_buffer_number_to_maintain: 0
2023/09/11-12:41:03.810631 140221488028480 Options.max_write_buffer_size_to_maintain: 0
2023/09/11-12:41:03.810632 140221488028480 Options.bottommost_compression_opts.window_bits: -14
2023/09/11-12:41:03.810633 140221488028480 Options.bottommost_compression_opts.level: 32767
2023/09/11-12:41:03.810634 140221488028480 Options.bottommost_compression_opts.strategy: 0
2023/09/11-12:41:03.810635 140221488028480 Options.bottommost_compression_opts.max_dict_bytes: 0
2023/09/11-12:41:03.810637 140221488028480 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2023/09/11-12:41:03.810638 140221488028480 Options.bottommost_compression_opts.parallel_threads: 1
2023/09/11-12:41:03.810639 140221488028480 Options.bottommost_compression_opts.enabled: false
2023/09/11-12:41:03.810640 140221488028480 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
2023/09/11-12:41:03.810641 140221488028480 Options.bottommost_compression_opts.use_zstd_dict_trainer: true
2023/09/11-12:41:03.810642 140221488028480 Options.compression_opts.window_bits: -14
2023/09/11-12:41:03.810643 140221488028480 Options.compression_opts.level: 32767
2023/09/11-12:41:03.810644 140221488028480 Options.compression_opts.strategy: 0
2023/09/11-12:41:03.810646 140221488028480 Options.compression_opts.max_dict_bytes: 0
2023/09/11-12:41:03.810647 140221488028480 Options.compression_opts.zstd_max_train_bytes: 0
2023/09/11-12:41:03.810648 140221488028480 Options.compression_opts.use_zstd_dict_trainer: true
2023/09/11-12:41:03.810649 140221488028480 Options.compression_opts.parallel_threads: 1
2023/09/11-12:41:03.810650 140221488028480 Options.compression_opts.enabled: false
2023/09/11-12:41:03.810651 140221488028480 Options.compression_opts.max_dict_buffer_bytes: 0
2023/09/11-12:41:03.810652 140221488028480 Options.level0_file_num_compaction_trigger: 4
2023/09/11-12:41:03.810653 140221488028480 Options.level0_slowdown_writes_trigger: 20
2023/09/11-12:41:03.810654 140221488028480 Options.level0_stop_writes_trigger: 30
2023/09/11-12:41:03.810655 140221488028480 Options.target_file_size_base: 67108864
2023/09/11-12:41:03.810657 140221488028480 Options.target_file_size_multiplier: 1
2023/09/11-12:41:03.810658 140221488028480 Options.max_bytes_for_level_base: 1073741824
2023/09/11-12:41:03.810659 140221488028480 Options.level_compaction_dynamic_level_bytes: 0
2023/09/11-12:41:03.810660 140221488028480 Options.max_bytes_for_level_multiplier: 8.000000
2023/09/11-12:41:03.810662 140221488028480 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2023/09/11-12:41:03.810663 140221488028480 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2023/09/11-12:41:03.810664 140221488028480 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2023/09/11-12:41:03.810665 140221488028480 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2023/09/11-12:41:03.810666 140221488028480 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2023/09/11-12:41:03.810667 140221488028480 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2023/09/11-12:41:03.810668 140221488028480 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2023/09/11-12:41:03.810669 140221488028480 Options.max_sequential_skip_in_iterations: 8
2023/09/11-12:41:03.810670 140221488028480 Options.max_compaction_bytes: 1677721600
2023/09/11-12:41:03.810672 140221488028480 Options.ignore_max_compaction_bytes_for_input: true
2023/09/11-12:41:03.810673 140221488028480 Options.arena_block_size: 1048576
2023/09/11-12:41:03.810674 140221488028480 Options.soft_pending_compaction_bytes_limit: 68719476736
2023/09/11-12:41:03.810675 140221488028480 Options.hard_pending_compaction_bytes_limit: 137438953472
2023/09/11-12:41:03.810676 140221488028480 Options.disable_auto_compactions: 0
2023/09/11-12:41:03.810678 140221488028480 Options.compaction_style: kCompactionStyleLevel
2023/09/11-12:41:03.810679 140221488028480 Options.compaction_pri: kMinOverlappingRatio
2023/09/11-12:41:03.810681 140221488028480 Options.compaction_options_universal.size_ratio: 1
2023/09/11-12:41:03.810682 140221488028480 Options.compaction_options_universal.min_merge_width: 2
2023/09/11-12:41:03.810683 140221488028480 Options.compaction_options_universal.max_merge_width: 4294967295
2023/09/11-12:41:03.810684 140221488028480 Options.compaction_options_universal.max_size_amplification_percent: 200
2023/09/11-12:41:03.810685 140221488028480 Options.compaction_options_universal.compression_size_percent: -1
2023/09/11-12:41:03.810686 140221488028480 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2023/09/11-12:41:03.810687 140221488028480 Options.compaction_options_fifo.max_table_files_size: 0
2023/09/11-12:41:03.810689 140221488028480 Options.compaction_options_fifo.allow_compaction: 1
2023/09/11-12:41:03.810693 140221488028480 Options.table_properties_collectors:
2023/09/11-12:41:03.810694 140221488028480 Options.inplace_update_support: 0
2023/09/11-12:41:03.810695 140221488028480 Options.inplace_update_num_locks: 10000
2023/09/11-12:41:03.810696 140221488028480 Options.memtable_prefix_bloom_size_ratio: 0.000000
2023/09/11-12:41:03.810698 140221488028480 Options.memtable_whole_key_filtering: 0
2023/09/11-12:41:03.810699 140221488028480 Options.memtable_huge_page_size: 0
2023/09/11-12:41:03.810700 140221488028480 Options.bloom_locality: 0
2023/09/11-12:41:03.810701 140221488028480 Options.max_successive_merges: 0
2023/09/11-12:41:03.810702 140221488028480 Options.optimize_filters_for_hits: 0
2023/09/11-12:41:03.810703 140221488028480 Options.paranoid_file_checks: 0
2023/09/11-12:41:03.810704 140221488028480 Options.force_consistency_checks: 1
2023/09/11-12:41:03.810705 140221488028480 Options.report_bg_io_stats: 0
2023/09/11-12:41:03.810706 140221488028480 Options.ttl: 2592000
2023/09/11-12:41:03.810707 140221488028480 Options.periodic_compaction_seconds: 0
2023/09/11-12:41:03.810709 140221488028480 Options.preclude_last_level_data_seconds: 0
2023/09/11-12:41:03.810710 140221488028480 Options.preserve_internal_time_seconds: 0
2023/09/11-12:41:03.810711 140221488028480 Options.enable_blob_files: false
2023/09/11-12:41:03.810712 140221488028480 Options.min_blob_size: 0
2023/09/11-12:41:03.810713 140221488028480 Options.blob_file_size: 268435456
2023/09/11-12:41:03.810714 140221488028480 Options.blob_compression_type: NoCompression
2023/09/11-12:41:03.810715 140221488028480 Options.enable_blob_garbage_collection: false
2023/09/11-12:41:03.810716 140221488028480 Options.blob_garbage_collection_age_cutoff: 0.250000
2023/09/11-12:41:03.810718 140221488028480 Options.blob_garbage_collection_force_threshold: 1.000000
2023/09/11-12:41:03.810719 140221488028480 Options.blob_compaction_readahead_size: 0
2023/09/11-12:41:03.810720 140221488028480 Options.blob_file_starting_level: 0
2023/09/11-12:41:03.810721 140221488028480 Options.experimental_mempurge_threshold: 0.000000
2023/09/11-12:41:37.283858 140221488028480 [/version_set.cc:5849] Recovered from manifest file:/home/smrc/nvme_raid/10TB_load_vanilla/MANIFEST-198126 succeeded,manifest_file_number is 198126, next_file_number is 198128, last_sequence is 120825453602, log_number is 198122,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 4
2023/09/11-12:41:37.283943 140221488028480 [/version_set.cc:5858] Column family [default] (ID 0), log number is 198122
2023/09/11-12:41:37.286053 140221488028480 [/db_impl/db_impl_open.cc:539] DB ID: 8df89b0a-ce0f-446d-ad1b-4147700d9b41
2023/09/11-12:41:37.902490 140221488028480 EVENT_LOG_v1 {"time_micros": 1694436097902477, "job": 1, "event": "recovery_started", "wal_files": [4, 198117, 198121, 198125]}
2023/09/11-12:41:37.902504 140221488028480 [/db_impl/db_impl_open.cc:1032] Skipping log #4 since it is older than min log to keep #198122
2023/09/11-12:41:37.902506 140221488028480 [/db_impl/db_impl_open.cc:1032] Skipping log #198117 since it is older than min log to keep #198122
2023/09/11-12:41:37.902507 140221488028480 [/db_impl/db_impl_open.cc:1032] Skipping log #198121 since it is older than min log to keep #198122
2023/09/11-12:41:37.902510 140221488028480 [/db_impl/db_impl_open.cc:1045] Recovering log #198125 mode 2
2023/09/11-12:41:37.902601 140221488028480 EVENT_LOG_v1 {"time_micros": 1694436097902599, "job": 1, "event": "recovery_finished"}
2023/09/11-12:41:37.945462 140221488028480 [/version_set.cc:5322] Creating manifest 198130
2023/09/11-12:41:38.359291 140221488028480 [WARN] [/column_family.cc:960] [default] Stopping writes because we have 198107 level-0 files
2023/09/11-12:41:38.617951 140221488028480 [DEBUG] [/db_impl/db_impl_files.cc:367] [JOB 2] Delete /home/smrc/nvme_raid/10TB_load_vanilla/MANIFEST-198126 type=3 #198126 -- OK
2023/09/11-12:41:38.645866 140221488028480 [/db_impl/db_impl_open.cc:1988] SstFileManager instance 0x55aec5b302d0
2023/09/11-12:41:39.352493 140221488028480 DB pointer 0x55aec5b2a140
2023/09/11-12:41:39.391343 140207177324288 [/db_impl/db_impl_compaction_flush.cc:2311] [default] WaitUntilFlushWouldNotStallWrites waiting on stall conditions to clear
2023/09/11-12:41:39.541723 140212870964992 [/compaction/compaction_job.cc:1998] [default] [JOB 3] Compacting 198107@0 files to L1, score 49526.75
2023/09/11-12:41:39.541783 140212870964992 [/compaction/compaction_job.cc:2002] [default]: Compaction start summary: Base version 2 Base level 0, inputs: [198114(19MB) 198113(57MB) 198112(57MB) 198111(57MB) 198110(57MB) 198109(57MB) 198108(57MB) 198107(57MB) 198106(57MB) 198105(57MB) 198104(57MB) 198103(57MB) 198102(57MB) 198101(57MB) 198100(57MB) 198099(57MB) 198098(57MB) 198097(57MB) 198096(57MB) 198095(57MB) 198094(57MB) 198093(57MB) 198092(57MB) 198091(57MB) 198090(57MB) 198089(57MB) 198088(57MB) 198087(57MB) 198086(57MB) 198085(57MB) 198084(57MB) 198083(57MB) 198082(57MB) 198081(57MB) 198080(57MB) 198079(57MB) 198078(57MB) 198077(57MB) 198076(57MB) 198075(57MB) 198074(57MB) 198073(57MB) 198072(57MB) 198071(57MB) 198070(57MB) 198069(57MB) 198068(57MB) 198067(57MB) 198066(57MB) 198065(57MB) 198064(57MB) 198063(57MB) 198062(57MB) 198061(57MB) 198060(57MB) 198059(57MB) 198058(57MB) 198057(57MB) 198056(57MB) 198055(57MB) 198054(57MB) 198053(57MB) 198052(57MB) 198051(57MB) 198050(57MB) 198049(57MB) 198048(57MB) 198047(57MB) 198046(57MB) 198045(57MB) 198044(57MB) 198043(57MB) 198042(57MB) 198041(57MB) 198040(57MB) 198039(57MB) 198038(57MB) 198037(57MB) 198036(57MB) 198035(57MB) 198034(57MB) 198033(57MB) 198032(57MB) 198031(57MB) 198030(57MB) 198029(57MB) 198028(57MB) 198027(57MB) 198026(57MB) 198025(57MB) 198024(57MB) 198023(57MB) 198022(57MB) 198021(57MB) 198020(57MB) 198019(57MB) 198018(57MB) 198017(57MB) 198016(57MB) 198015(57MB) 198014(57MB) 198013(57MB) 198012(57MB) 198011(57MB) 198010(57MB) 198009(57MB) 198008(57MB) 198007(57MB) 198006(57MB) 198005(57MB) 198004(57MB) 198003(57MB) 198002(57MB) 198001(57MB) 198000(57MB) 197999(57MB) 197998(57MB) 197997(57MB) 197996(57MB) 197995(57MB) 197994(57MB) 197993(57MB) 197992(57MB) 197991(57MB) 197990(57MB) 197989(57MB) 197988(57MB) 197987(57MB) 197986(57MB) 197985(57MB) 197984(57MB) 197983(57MB) 197982(57MB) 197981(57MB) 197980(57MB) 197979(57MB) 197978(57MB) 197977(57MB) 197976(57MB) 197975(57MB) 197974(57MB) 197973(57MB) 197972(57MB) 197971(57MB) 197970(57MB) 197969(57MB) 197968(57MB) 197967(57MB) 197966(57MB) 197965(57MB) 197964(57MB) 197963(57MB) 197962(57MB) 197961(57MB) 197960(57MB) 197959(57MB) 197958(57MB) 197957(57MB) 197956(57MB) 197955(57MB) 197954(57MB) 197953(57MB) 197952(57MB) 197951(57MB) 197950(57MB) 197949(57MB) 197948(57MB) 197947(57MB) 197946(57MB) 197945(57MB) 197944(57MB) 197943(57MB) 197942(57MB) 197941(57MB) 197940(57MB) 197939(57MB) 197938(57MB)]
2023/09/11-12:41:39.577540 140212870964992 EVENT_LOG_v1 {"time_micros": 1694436099541798, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [198114, 198113, 198112, 198111, 198110, 198109, 198108, 198107, 198106, 198105, 198104, 198103, 198102, 198101, 198100, 198099, 198098, 198097, 198096, 198095, 198094, 198093, 198092, 198091, 198090, 198089, 198088, 198087, 198086, 198085, 198084, 198083, 198082, 198081, 198080, 198079, 198078, 198077, 198076, 198075, 198074, 198073, 198072, 198071, 198070, 198069, 198068, 198067, 198066, 198065, 198064, 198063, 198062, 198061, 198060, 198059, 198058, 198057, 198056, 198055, 198054, 198053, 198052, 198051, 198050, 198049, 198048, 198047, 198046, 198045, 198044, 198043, 198042, 198041, 198040, 198039, 198038, 198037, 198036, 198035, 198034, 198033, 198032, 198031, 198030, 198029, 198028, 198027, 198026, 198025, 198024, 198023, 198022, 198021, 198020, 198019, 198018, 198017, 198016, 198015, 198014, 198013, 198012, 198011, 198010, 198009, 198008, 198007, 198006, 198005, 198004, 198003, 198002, 198001, 198000, 197999, 197998, 197997, 197996, 197995, 197994, 197993, 197992, 197991, 197990, 197989, 197988, 197987, 197986, 197985, 197984, 197983, 197982, 197981, 197980, 197979, 197978, 197977, 197976, 197975, 197974, 197973, 197972, 197971, 197970, 197969, 197968, 197967, 197966, 197965, 197964, 197963, 197962, 197961, 197960, 197959, 197958, 197957, 197956, 197955, 197954, 197953, 197952, 197951, 197950, 197949, 197948, 197947, 197946, 197945, 197944, 197943, 197942, 197941, 197940, 197939, 197938, 197937, 197936, 197935, 197934, 197933, 197932, 197931, 197930, 197929, 197928, 197927, 197926, 197925, 197924, 197923, 197922, 197921, 197920, 197919, 197918, 197917, 197916, 197915, 197914, 197913, 197912, 197911, 197910, 197909, 197908, 197907, 197906, 197905, 197904, 197903, 197902, 197901, 197900, 197899, 197898, 197897, 197896, 197895, 197894, 197893, 197892, 197891, 197890, 197889, 197888, 197887, 197886, 197885, 197884, 197883, 197882, 197881, 197880, 197879, 197878, 197877, 197876, 197875, 197874, 197873, 197872, 197871, 197870, 197869, 197868, 197867, 197866, 197865, 197864, 197863, 197862, 197861, 197860, 197859, 197858, 197857, 197856, 197855, 197854, 197853, 197852, 197851, 197850, 197849, 197848, 197847, 197846, 197845, 197844, 197843, 197842, 197841, 197840, 197839, 197838, 197837, 197836, 197835, 197834, 197833, 197832, 197831, 197830, 197829, 197828, 197827, 197826, 197825, 197824, 197823, 197822, 197821, 197820, 197819, 197818, 197817, 197816, 197815, 197814, 197813, 197812, 197811, 197810, 197809, 197808, 197807, 197806, 197805, 197804, 197803, 197802, 197801, 197800, 197799, 197798, 197797, 197796, 197795, 197794, 197793, 197792, 197791, 197790, 197789, 197788, 197787, 197786, 197785, 197784, 197783, 197782, 197781, 197780, 197779, 197778, 197777, 197776, 197775, 197774, 197773, 197772, 197771, 197770, 197769, 197768, 197767, 197766, 197765, 197764, 197763, 197762, 197761, 197760, 197759, 197758, 197757, 197756, 197755, 197754, 197753, 197752, 197751, 197750, 197749, 197748, 197747, 197746, 197745, 197744, 197743, 197742, 197741, 197740, 197739, 197738, 197737, 197736, 197735, 197734, 197733, 197732, 197731, 197730, 197729, 197728, 197727, 197726, 197725, 197724, 197723, 197722, 197721, 197720, 197719, 197718, 197717, 197716, 197715, 197714, 197713, 197712, 197711, 197710, 197709, 197708, 197707, 197706, 197705, 197704, 197703, 197702, 197701, 197700, 197699, 197698, 197697, 197696, 197695, 197694, 197693, 197692, 197691, 197690, 197689, 197688, 197687, 197686, 197685, 197684, 197683, 197682, 197681, 197680, 197679, 197678, 197677, 197676, 197675, 197674, 197673, 197672, 197671, 197670, 197669, 197668, 197667, 197666, 197665, 197664, 197663, 197662, 197661, 197660, 197659, 197658, 197657, 197656, 197655, 197654, 197653, 197652, 197651, 197650, 197649, 197648, 197647, 197646, 197645, 197644, 197643, 197642, 197641, 197640, 197639, 197638, 197637, 197636, 197635, 197634, 197633, 197632, 197631, 197630, 197629, 197628, 197627, 197626, 197625, 197624, 197623, 197622, 197621, 197620, 197619, 197618, 197617, 197616, 197615, 197614, 197613, 197612, 197611, 197610, 197609, 197608, 197607, 197606, 197605, 197604, 197603, 197602, 197601, 197600, 197599, 197598, 197597, 197596, 197595, 197594, 197593, 197592, 197591, 197590, 197589, 197588, 197587, 197586, 197585, 197584, 197583, 197582, 197581, 197580, 197579, 197578, 197577, 197576, 197575, 197574, 197573, 197572, 197571, 197570, 197569, 197568, 197567, 197566, 197565, 197564, 197563, 197562, 197561, 197560, 197559, 197558, 197557, 197556, 197555, 197554, 197553, 197552, 197551, 197550, 197549, 197548, 197547, 197546, 197545, 197544, 197543, 197542, 197541, 197540, 197539, 197538, 197537, 197536, 197535, 197534, 197533, 197532, 197531, 197530, 197529, 197528, 197527, 197526, 197525, 197524, 197523, 197522, 197521, 197520, 197519, 197518, 197517, 197516, 197515, 197514, 197513, 197512, 197511, 197510, 197509, 197508, 197507, 197506, 197505, 197504, 197503, 197502, 197501, 197500, 197499, 197498, 197497, 197496, 197495, 197494, 197493, 197492, 197491, 197490, 197489, 197488, 197487, 197486, 197485, 197484, 197483, 197482, 197481, 197480, 197479, 197478, 197477, 197476, 197475, 197474, 197473, 197472, 197471, 197470, 197469, 197468, 197467, 197466, 197465, 197464, 197463, 197462, 197461, 197460, 197459, 197458, 197457, 197456, 197455, 197454, 197453, 197452, 197451, 197450, 197449, 197448, 197447, 197446, 197445, 197444, 197443, 197442, 197441, 197440, 197439, 197438, 197437, 197436, 197435, 197434, 197433, 197432, 197431, 197430, 197429, 197428, 197427, 197426, 197425, 197424, 197423, 197422, 197421, 197420, 197419, 197418, 197417, 197416, 197415, 197414, 197413, 197412, 197411, 197410, 197409, 197408, 197407, 197406, 197405, 197404, 197403, 197402, 197401, 197400, 197399, 197398, 197397, 197396, 197395, 197394, 197393, 197392, 197391, 197390, 197389, 197388, 197387, 197386, 197385, 197384, 197383, 197382, 197381, 197380, 197379, 197378, 197377, 197376, 197375, 197374, 197373, 197372, 197371, 197370, 197369, 197368, 197367, 197366, 197365, 197364, 197363, 197362, 197361, 197360, 197359, 197358, 197357, 197356, 197355, 197354, 197353, 197352, 197351, 197350, 197349, 197348, 197347, 197346, 197345, 197344, 197343, 197342, 197341, 197340, 197339, 197338, 197337, 197336, 197335, 197334, 197333, 197332, 197331, 197330, 197329, 197328, 197327, 197326, 197325, 197324, 197323, 197322, 197321, 197320, 197319, 197318, 197317, 197316, 197315, 197314, 197313, 197312, 197311, 197310, 197309, 197308, 197307, 197306, 197305, 197304, 197303, 197302, 197301, 197300, 197299, 197298, 197297, 197296, 197295, 197294, 197293, 197292, 197291, 197290, 197289, 197288, 197287, 197286, 197285, 197284, 197283, 197282, 197281, 197280, 197279, 197278, 197277, 197276, 197275, 197274, 197273, 197272, 197271, 197270, 197269, 197268, 197267, 197266, 197265, 197264, 197263, 197262, 197261, 197260, 197259, 197258, 197257, 197256, 197255, 197254, 197253, 197252, 197251, 197250, 197249, 197248, 197247, 197246, 197245, 197244, 197243, 197242, 197241, 197240, 197239, 197238, 197237, 197236, 197235, 197234, 197233, 197232, 197231, 197230, 197229, 197228, 197227, 197226, 197225, 197224, 197223, 197222, 197221, 197220, 197219, 197218, 197217, 197216, 197215, 197214, 197213, 197212, 197211, 197210, 197209, 197208, 197207, 197206, 197205, 197204, 197203, 197202, 197201, 197200, 197199, 197198, 197197, 197196, 197195, 197194, 197193, 197192, 197191, 197190, 197189, 197188, 197187, 197186, 197185, 197184, 197183, 197182, 197181, 197180, 197179, 197178, 197177, 197176, 197175, 197174, 197173, 197172, 197171, 197170, 197169, 197168, 197167, 197166, 197165, 197164, 197163, 197162, 197161, 197160, 197159, 197158, 197157, 197156, 197155, 197154, 197153, 197152, 197151, 197150, 197149, 197148, 197147, 197146, 197145, 197144, 197143, 197142, 197141, 197140, 197139, 197138, 197137, 197136, 197135, 197134, 197133, 197132, 197131, 197130, 197129, 197128, 197127, 197126, 197125, 197124, 197123, 197122, 197121, 197120, 197119, 197118, 197117, 197116, 197115, 197114, 197113, 197112, 197111, 197110, 197109, 197108, 197107, 197106, 197105, 197104, 197103, 197102, 197101, 197100, 197099, 197098, 197097, 197096, 197095, 197094, 197093, 197092, 197091, 197090, 197089, 197088, 197087, 197086, 197085, 197084, 197083, 197082, 197081, 197080, 197079, 197078, 197077, 197076, 197075, 197074, 197073, 197072, 197071, 197070, 197069, 197068, 197067, 197066, 197065, 197064, 197063, 197062, 197061, 197060, 197059, 197058, 197057, 197056, 197055, 197054, 197053, 197052, 197051, 197050, 197049, 197048, 197047, 197046, 197045, 197044, 197043, 197042, 197041, 197040, 197039, 197038, 197037, 197036, 197035, 197034, 197033, 197032, 197031, 197030, 197029, 197028, 197027, 197026, 197025, 197024, 197023, 197022, 197021, 197020, 197019, 197018, 197017, 197016, 197015, 197014, 197013, 197012, 197011, 197010, 197009, 197008, 197007, 197006, 197005, 197004, 197003, 197002, 197001, 197000, 196999, 196998, 196997, 196996, 196995, 196994, 196993, 196992, 196991, 196990, 196989, 196988, 196987, 196986, 196985, 196984, 196983, 196982, 196981, 196980, 196979, 196978, 196977, 196976, 196975, 196974, 196973, 196972, 196971, 196970, 196969, 196968, 196967, 196966, 196965, 196964, 196963, 196962, 196961, 196960, 196959, 196958, 196957, 196956, 196955, 196954, 196953, 196952, 196951, 196950, 196949, 196948, 196947, 196946, 196945, 196944, 196943, 196942, 196941, 196940, 196939, 196938, 196937, 196936, 196935, 196934, 196933, 196932, 196931, 196930, 196929, 196928, 196927, 196926, 196925, 196924, 196923, 196922, 196921, 196920, 196919, 196918, 196917, 196916, 196915, 196914, 196913, 196912, 196911, 196910, 196909, 196908, 196907, 196906, 196905, 196904, 196903, 196902, 196901, 196900, 196899, 196898, 196897, 196896, 196895, 196894, 196893, 196892, 196891, 196890, 196889, 196888, 196887, 196886, 196885, 196884, 196883, 196882, 196881, 196880, 196879, 196878, 196877, 196876, 196875, 196874, 196873, 196872, 196871, 196870, 196869, 196868, 196867, 196866, 196865, 196864, 196863, 196862, 196861, 196860, 196859, 196858, 196857, 196856, 196855, 196854, 196853, 196852, 196851, 196850, 196849, 196848, 196847, 196846, 196845, 196844, 196843, 196842, 196841, 196840, 196839, 196838, 196837, 196836, 196835, 196834, 196833, 196832, 196831, 196830, 196829, 196828, 196827, 196826, 196825, 196824, 196823, 196822, 196821, 196820, 196819, 196818, 196817, 196816, 196815, 196814, 196813, 196812, 196811, 196810, 196809, 196808, 196807, 196806, 196805, 196804, 196803, 196802, 196801, 196800, 196799, 196798, 196797, 196796, 196795, 196794, 196793, 196792, 196791, 196790, 196789, 196788, 196787, 196786, 196785, 196784, 196783, 196782, 196781, 196780, 196779, 196778, 196777, 196776, 196775, 196774, 196773, 196772, 196771, 196770, 196769, 196768, 196767, 196766, 196765, 196764, 196763, 196762, 196761, 196760, 196759, 196758, 196757, 196756, 196755, 196754, 196753, 196752, 196751, 196750, 196749, 196748, 196747, 196746, 196745, 196744, 196743, 196742, 196741, 196740, 196739, 196738, 196737, 196736, 196735, 196734, 196733, 196732, 196731, 196730, 196729, 196728, 196727, 196726, 196725, 196724, 196723, 196722, 196721, 196720, 196719, 196718, 196717, 196716, 196715, 196714, 196713, 196712, 196711, 196710, 196709, 196708, 196707, 196706, 196705, 196704, 196703, 196702, 196701, 196700, 196699, 196698, 196697, 196696, 196695, 196694, 196693, 196692, 196691, 196690, 196689, 196688, 196687, 196686, 196685, 196684, 196683, 196682, 196681, 196680, 196679, 196678, 196677, 196676, 196675, 196674, 196673, 196672, 196671, 196670, 196669, 196668, 196667, 196666, 196665, 196664, 196663, 196662, 196661, 196660, 196659, 196658, 196657, 196656, 196655, 196654, 196653, 196652, 196651, 196650, 196649, 196648, 196647, 196646, 196645, 196644, 196643, 196642, 196641, 196640, 196639, 196638, 196637, 196636, 196635, 196634, 196633, 196632, 196631, 196630, 196629, 196628, 196627, 196626, 196625, 196624, 196623, 196622, 196621, 196620, 196619, 196618, 196617, 196616, 196615, 196614, 196613, 196612, 196611, 196610, 196609, 196608, 196607, 196606, 196605, 196604, 196603, 196602, 196601, 196600, 196599, 196598, 196597, 196596, 196595, 196594, 196593, 196592, 196591, 196590, 196589, 196588, 196587, 196586, 196585, 196584, 196583, 196582, 196581, 196580, 196579, 196578, 196577, 196576, 196575, 196574, 196573, 196572, 196571, 196570, 196569, 196568, 196567, 196566, 196565, 196564, 196563, 196562, 196561, 196560, 196559, 196558, 196557, 196556, 196555, 196554, 196553, 196552, 196551, 196550, 196549, 196548, 196547, 196546, 196545, 196544, 196543, 196542, 196541, 196540, 196539, 196538, 196537, 196536, 196535, 196534, 196533, 196532, 196531, 196530, 196529, 196528, 196527, 196526, 196525, 196524, 196523, 196522, 196521, 196520, 196519, 196518, 196517, 196516, 196515, 196514, 196513, 196512, 196511, 196510, 196509, 196508, 196507, 196506, 196505, 196504, 196503, 196502, 196501, 196500, 196499, 196498, 196497, 196496, 196495, 196494, 196493, 196492, 196491, 196490, 196489, 196488, 196487, 196486, 196485, 196484, 196483, 196482, 196481, 196480, 196479, 196478, 196477, 196476, 196475, 196474, 196473, 196472, 196471, 196470, 196469, 196468, 196467, 196466, 196465, 196464, 196463, 196462, 196461, 196460, 196459, 196458, 196457, 196456, 196455, 196454, 196453, 196452, 196451, 196450, 196449, 196448, 196447, 196446, 196445, 196444, 196443, 196442, 196441, 196440, 196439, 196438, 196437, 196436, 196435, 196434, 196433, 196432, 196431, 196430, 196429, 196428, 196427, 196426, 196425, 196424, 196423, 196422, 196421, 196420, 196419, 196418, 196417, 196416, 196415, 196414, 196413, 196412, 196411, 196410, 196409, 196408, 196407, 196406, 196405, 196404, 196403, 196402, 196401, 196400, 196399, 196398, 196397, 196396, 196395, 196394, 196393, 196392, 196391, 196390, 196389, 196388, 196387, 196386, 196385, 196384, 196383, 196382, 196381, 196380, 196379, 196378, 196377, 196376, 196375, 196374, 196373, 196372, 196371, 196370, 196369, 196368, 196367, 196366, 196365, 196364, 196363, 196362, 196361, 196360, 196359, 196358, 196357, 196356, 196355, 196354, 196353, 196352, 196351, 196350, 196349, 196348, 196347, 196346, 196345, 196344, 196343, 196342, 196341, 196340, 196339, 196338, 196337, 196336, 196335, 196334, 196333, 196332, 196331, 196330, 196329, 196328, 196327, 196326, 196325, 196324, 196323, 196322, 196321, 196320, 196319, 196318, 196317, 196316, 196315, 196314, 196313, 196312, 196311, 196310, 196309, 196308, 196307, 196306, 196305, 196304, 196303, 196302, 196301, 196300, 196299, 196298, 196297, 196296, 196295, 196294, 196293, 196292, 196291, 196290, 196289, 196288, 196287, 196286, 196285, 196284, 196283, 196282, 196281, 196280, 196279, 196278, 196277, 196276, 196275, 196274, 196273, 196272, 196271, 196270, 196269, 196268, 196267, 196266, 196265, 196264, 196263, 196262, 196261, 196260, 196259, 196258, 196257, 196256, 196255, 196254, 196253, 196252, 196251, 196250, 196249, 196248, 196247, 196246, 196245, 196244, 196243, 196242, 196241, 196240, 196239, 196238, 196237, 196236, 196235, 196234, 196233, 196232, 196231, 196230, 196229, 196228, 196227, 196226, 196225, 196224, 196223, 196222, 196221, 196220, 196219, 196218, 196217, 196216, 196215, 196214, 196213, 196212, 196211, 196210, 196209, 196208, 196207, 196206, 196205, 196204, 196203, 196202, 196201, 196200, 196199, 196198, 196197, 196196, 196195, 196194, 196193, 196192, 196191, 196190, 196189, 196188, 196187, 196186, 196185, 196184, 196183, 196182, 196181, 196180, 196179, 196178, 196177, 196176, 196175, 196174, 196173, 196172, 196171, 196170, 196169, 196168, 196167, 196166, 196165, 196164, 196163, 196162, 196161, 196160, 196159, 196158, 196157, 196156, 196155, 196154, 196153, 196152, 196151, 196150, 196149, 196148, 196147, 196146, 196145, 196144, 196143, 196142, 196141, 196140, 196139, 196138, 196137, 196136, 196135, 196134, 196133, 196132, 196131, 196130, 196129, 196128, 196127, 196126, 196125, 196124, 196123, 196122, 196121, 196120, 196119, 196118, 196117, 196116, 196115, 196114, 196113, 196112, 196111, 196110, 196109, 196108, 196107, 196106, 196105, 196104, 196103, 196102, 196101, 196100, 196099, 196098, 196097, 196096, 196095, 196094, 196093, 196092, 196091, 196090, 196089, 196088, 196087, 196086, 196085, 196084, 196083, 196082, 196081, 196080, 196079, 196078, 196077, 196076, 196075, 196074, 196073, 196072, 196071, 196070, 196069, 196068, 196067, 196066, 196065, 196064, 196063, 196062, 196061, 196060, 196059, 196058, 196057, 196056, 196055, 196054, 196053, 196052, 196051, 196050, 196049, 196048, 196047, 196046, 196045, 196044, 196043, 196042, 196041, 196040, 196039, 196038, 196037, 196036, 196035, 196034, 196033, 196032, 196031, 196030, 196029, 196028, 196027, 196026, 196025, 196024, 196023, 196022, 196021, 196020, 196019, 196018, 196017, 196016, 196015, 196014, 196013, 196012, 196011, 196010, 196009, 196008, 196007, 196006, 196005, 196004, 196003, 196002, 196001, 196000, 195999, 195998, 195997, 195996, 195995, 195994, 195993, 195992, 195991, 195990, 195989, 195988, 195987, 195986, 195985, 195984, 195983, 195982, 195981, 195980, 195979, 195978, 195977, 195976, 195975, 195974, 195973, 195972, 195971, 195970, 195969, 195968, 195967, 195966, 195965, 195964, 195963, 195962, 195961, 195960, 195959, 195958, 195957, 195956, 195955, 195954, 195953, 195952, 195951, 195950, 195949, 195948, 195947, 195946, 195945, 195944, 195943, 195942, 195941, 195940, 195939, 195938, 195937, 195936, 195935, 195934, 195933, 195932, 195931, 195930, 195929, 195928, 195927, 195926, 195925, 195924, 195923, 195922, 195921, 195920, 195919, 195918, 195917, 195916, 195915, 195914, 195913, 195912, 195911, 195910, 195909, 195908, 195907, 195906, 195905, 195904, 195903, 195902, 195901, 195900, 195899, 195898, 195897, 195896, 195895, 195894, 195893, 195892, 195891, 195890, 195889, 195888, 195887, 195886, 195885, 195884, 195883, 195882, 195881, 195880, 195879, 195878, 195877, 195876, 195875, 195874, 195873, 195872, 195871, 195870, 195869, 195868, 195867, 195866, 195865, 195864, 195863, 195862, 195861, 195860, 195859, 195858, 195857, 195856, 195855, 195854, 195853, 195852, 195851, 195850, 195849, 195848, 195847, 195846, 195845, 195844, 195843, 195842, 195841, 195840, 195839, 195838, 195837, 195836, 195835, 195834, 195833, 195832, 195831, 195830, 195829, 195828, 195827, 195826, 195825, 195824, 195823, 195822, 195821, 195820, 195819, 195818, 195817, 195816, 195815, 195814, 195813, 195812, 195811, 195810, 195809, 195808, 195807, 195806, 195805, 195804, 195803, 195802, 195801, 195800, 195799, 195798, 195797, 195796, 195795, 195794, 195793, 195792, 195791, 195790, 195789, 195788, 195787, 195786, 195785, 195784, 195783, 195782, 195781, 195780, 195779, 195778, 195777, 195776, 195775, 195774, 195773, 195772, 195771, 195770, 195769, 195768, 195767, 195766, 195765, 195764, 195763, 195762, 195761, 195760, 195759, 195758, 195757, 195756, 195755, 195754, 195753, 195752, 195751, 195750, 195749, 195748, 195747, 195746, 195745, 195744, 195743, 195742, 195741, 195740, 195739, 195738, 195737, 195736, 195735, 195734, 195733, 195732, 195731, 195730, 195729, 195728, 195727, 195726, 195725, 195724, 195723, 195722, 195721, 195720, 195719, 195718, 195717, 195716, 195715, 195714, 195713, 195712, 195711, 195710, 195709, 195708, 195707, 195706, 195705, 195704, 195703, 195702, 195701, 195700, 195699, 195698, 195697, 195696, 195695, 195694, 195693, 195692, 195691, 195690, 195689, 195688, 195687, 195686, 195685, 195684, 195683, 195682, 195681, 195680, 195679, 195678, 195677, 195676, 195675, 195674, 195673, 195672, 195671, 195670, 195669, 195668, 195667, 195666, 195665, 195664, 195663, 195662, 195661, 195660, 195659, 195658, 195657, 195656, 195655, 195654, 195653, 195652, 195651, 195650, 195649, 195648, 195647, 195646, 195645, 195644, 195643, 195642, 195641, 195640, 195639, 195638, 195637, 195636, 195635, 195634, 195633, 195632, 195631, 195630, 195629, 195628, 195627, 195626, 195625, 195624, 195623, 195622, 195621, 195620, 195619, 195618, 195617, 195616, 195615, 195614, 195613, 195612, 195611, 195610, 195609, 195608, 195607, 195606, 195605, 195604, 195603, 195602, 195601, 195600, 195599, 195598, 195597, 195596, 195595, 195594, 195593, 195592, 195591, 195590, 195589, 195588, 195587, 195586, 195585, 195584, 195583, 195582, 195581, 195580, 195579, 195578, 195577, 195576, 195575, 195574, 195573, 195572, 195571, 195570, 195569, 195568, 195567, 195566, 195565, 195564, 195563, 195562, 195561, 195560, 195559, 195558, 195557, 195556, 195555, 195554, 195553, 195552, 195551, 195550, 195549, 195548, 195547, 195546, 195545, 195544, 195543, 195542, 195541, 195540, 195539, 195538, 195537, 195536, 195535, 195534, 195533, 195532, 195531, 195530, 195529, 195528, 195527, 195526, 195525, 195524, 195523, 195522, 195521, 195520, 195519, 195518, 195517, 195516, 195515, 195514, 195513, 195512, 195511, 195510, 195509, 195508, 195507, 195506, 195505, 195504, 195503, 195502, 195501, 195500, 195499, 195498, 195497, 195496, 195495, 195494, 195493, 195492, 195491, 195490, 195489, 195488, 195487, 195486, 195485, 195484, 195483, 195482, 195481, 195480, 195479, 195478, 195477, 195476, 195475, 195474, 195473, 195472, 195471, 195470, 195469, 195468, 195467, 195466, 195465, 195464, 195463, 195462, 195461, 195460, 195459, 195458, 195457, 195456, 195455, 195454, 195453, 195452, 195451, 195450, 195449, 195448, 195447, 195446, 195445, 195444, 195443, 195442, 195441, 195440, 195439, 195438, 195437, 195436, 195435, 195434, 195433, 195432, 195431, 195430, 195429, 195428, 195427, 195426, 195425, 195424, 195423, 195422, 195421, 195420, 195419, 195418, 195417, 195416, 195415, 195414, 195413, 195412, 195411, 195410, 195409, 195408, 195407, 195406, 195405, 195404, 195403, 195402, 195401, 195400, 195399, 195398, 195397, 195396, 195395, 195394, 195393, 195392, 195391, 195390, 195389, 195388, 195387, 195386, 195385, 195384, 195383, 195382, 195381, 195380, 195379, 195378, 195377, 195376, 195375, 195374, 195373, 195372, 195371, 195370, 195369, 195368, 195367, 195366, 195365, 195364, 195363, 195362, 195361, 195360, 195359, 195358, 195357, 195356, 195355, 195354, 195353, 195352, 195351, 195350, 195349, 195348, 195347, 195346, 195345, 195344, 195343, 195342, 195341, 195340, 195339, 195338, 195337, 195336, 195335, 195334, 195333, 195332, 195331, 195330, 195329, 195328, 195327, 195326, 195325, 195324, 195323, 195322, 195321, 195320, 195319, 195318, 195317, 195316, 195315, 195314, 195313, 195312, 195311, 195310, 195309, 195308, 195307, 195306, 195305, 195304, 195303, 195302, 195301, 195300, 195299, 195298, 195297, 195296, 195295, 195294, 195293, 195292, 195291, 195290, 195289, 195288, 195287, 195286, 195285, 195284, 195283, 195282, 195281, 195280, 195279, 195278, 195277, 195276, 195275, 195274, 195273, 195272, 195271, 195270, 195269, 195268, 195267, 195266, 195265, 195264, 195263, 195262, 195261, 195260, 195259, 195258, 195257, 195256, 195255, 195254, 195253, 195252, 195251, 195250, 195249, 195248, 195247, 195246, 195245, 195244, 195243, 195242, 195241, 195240, 195239, 195238, 195237, 195236, 195235, 195234, 195233, 195232, 195231, 195230, 195229, 195228, 195227, 195226, 195225, 195224, 195223, 195222, 195221, 195220, 195219, 195218, 195217, 195216, 195215, 195214, 195213, 195212, 195211, 195210, 195209, 195208, 195207, 195206, 195205, 195204, 195203, 195202, 195201, 195200, 195199, 195198, 195197, 195196, 195195, 195194, 195193, 195192, 195191, 195190, 195189, 195188, 195187, 195186, 195185, 195184, 195183, 195182, 195181, 195180, 195179, 195178, 195177, 195176, 195175, 195174, 195173, 195172, 195171, 195170, 195169, 195168, 195167, 195166, 195165, 195164, 195163, 195162, 195161, 195160, 195159, 195158, 195157, 195156, 195155, 195154, 195153, 195152, 195151, 195150, 195149, 195148, 195147, 195146, 195145, 195144, 195143, 195142, 195141, 195140, 195139, 195138, 195137, 195136, 195135, 195134, 195133, 195132, 195131, 195130, 195129, 195128, 195127, 195126, 195125, 195124, 195123, 195122, 195121, 195120, 195119, 195118, 195117, 195116, 195115, 195114, 195113, 195112, 195111, 195110, 195109, 195108, 195107, 195106, 195105, 195104, 195103, 195102, 195101, 195100, 195099, 195098, 195097, 195096, 195095, 195094, 195093, 195092, 195091, 195090, 195089, 195088, 195087, 195086, 195085, 195084, 195083, 195082, 195081, 195080, 195079, 195078, 195077, 195076, 195075, 195074, 195073, 195072, 195071, 195070, 195069, 195068, 195067, 195066, 195065, 195064, 195063, 195062, 195061, 195060, 195059, 195058, 195057, 195056, 195055, 195054, 195053, 195052, 195051, 195050, 195049, 195048, 195047, 195046, 195045, 195044, 195043, 195042, 195041, 195040, 195039, 195038, 195037, 195036, 195035, 195034, 195033, 195032, 195031, 195030, 195029, 195028, 195027, 195026, 195025, 195024, 195023, 195022, 195021, 195020, 195019, 195018, 195017, 195016, 195015, 195014, 195013, 195012, 195011, 195010, 195009, 195008, 195007, 195006, 195005, 195004, 195003, 195002, 195001, 195000, 194999, 194998, 194997, 194996, 194995, 194994, 194993, 194992, 194991, 194990, 194989, 194988, 194987, 194986, 194985, 194984, 194983, 194982, 194981, 194980, 194979, 194978, 194977, 194976, 194975, 194974, 194973, 194972, 194971, 194970, 194969, 194968, 194967, 194966, 194965, 194964, 194963, 194962, 194961, 194960, 194959, 194958, 194957, 194956, 194955, 194954, 194953, 194952, 194951, 194950, 194949, 194948, 194947, 194946, 194945, 194944, 194943, 194942, 194941, 194940, 194939, 194938, 194937, 194936, 194935, 194934, 194933, 194932, 194931, 194930, 194929, 194928, 194927, 194926, 194925, 194924, 194923, 194922, 194921, 194920, 194919, 194918, 194917, 194916, 194915, 194914, 194913, 194912, 194911, 194910, 194909, 194908, 194907, 194906, 194905, 194904, 194903, 194902, 194901, 194900, 194899, 194898, 194897, 194896, 194895, 194894, 194893, 194892, 194891, 194890, 194889, 194888, 194887, 194886, 194885, 194884, 194883, 194882, 194881, 194880, 194879, 194878, 194877, 194876, 194875, 194874, 194873, 194872, 194871, 194870, 194869, 194868, 194867, 194866, 194865, 194864, 194863, 194862, 194861, 194860, 194859, 194858, 194857, 194856, 194855, 194854, 194853, 194852, 194851, 194850, 194849, 194848, 194847, 194846, 194845, 194844, 194843, 194842, 194841, 194840, 194839, 194838, 194837, 194836, 194835, 194834, 194833, 194832, 194831, 194830, 194829, 194828, 194827, 194826, 194825, 194824, 194823, 194822, 194821, 194820, 194819, 194818, 194817, 194816, 194815, 194814, 194813, 194812, 194811, 194810, 194809, 194808, 194807, 194806, 194805, 194804, 194803, 194802, 194801, 194800, 194799, 194798, 194797, 194796, 194795, 194794, 194793, 194792, 194791, 194790, 194789, 194788, 194787, 194786, 194785, 194784, 194783, 194782, 194781, 194780, 194779, 194778, 194777, 194776, 194775, 194774, 194773, 194772, 194771, 194770, 194769, 194768, 194767, 194766, 194765, 194764, 194763, 194762, 194761, 194760, 194759, 194758, 194757, 194756, 194755, 194754, 194753, 194752, 194751, 194750, 194749, 194748, 194747, 194746, 194745, 194744, 194743, 194742, 194741, 194740, 194739, 194738, 194737, 194736, 194735, 194734, 194733, 194732, 194731, 194730, 194729, 194728, 194727, 194726, 194725, 194724, 194723, 194722, 194721, 194720, 194719, 194718, 194717, 194716, 194715, 194714, 194713, 194712, 194711, 194710, 194709, 194708, 194707, 194706, 194705, 194704, 194703, 194702, 194701, 194700, 194699, 194698, 194697, 194696, 194695, 194694, 194693, 194692, 194691, 194690, 194689, 194688, 194687, 194686, 194685, 194684, 194683, 194682, 194681, 194680, 194679, 194678, 194677, 194676, 194675, 194674, 194673, 194672, 194671, 194670, 194669, 194668, 194667, 194666, 194665, 194664, 194663, 194662, 194661, 194660, 194659, 194658, 194657, 194656, 194655, 194654, 194653, 194652, 194651, 194650, 194649, 194648, 194647, 194646, 194645, 194644, 194643, 194642, 194641, 194640, 194639, 194638, 194637, 194636, 194635, 194634, 194633, 194632, 194631, 194630, 194629, 194628, 194627, 194626, 194625, 194624, 194623, 194622, 194621, 194620, 194619, 194618, 194617, 194616, 194615, 194614, 194613, 194612, 194611, 194610, 194609, 194608, 194607, 194606, 194605, 194604, 194603, 194602, 194601, 194600, 194599, 194598, 194597, 194596, 194595, 194594, 194593, 194592, 194591, 194590, 194589, 194588, 194587, 194586, 194585, 194584, 194583, 194582, 194581, 194580, 194579, 194578, 194577, 194576, 194575, 194574, 194573, 194572, 194571, 194570, 194569, 194568, 194567, 194566, 194565, 194564, 194563, 194562, 194561, 194560, 194559, 194558, 194557, 194556, 194555, 194554, 194553, 194552, 194551, 194550, 194549, 194548, 194547, 194546, 194545, 194544, 194543, 194542, 194541, 194540, 194539, 194538, 194537, 194536, 194535, 194534, 194533, 194532, 194531, 194530, 194529, 194528, 194527, 194526, 194525, 194524, 194523, 194522, 194521, 194520, 194519, 194518, 194517, 194516, 194515, 194514, 194513, 194512, 194511, 194510, 194509, 194508, 194507, 194506, 194505, 194504, 194503, 194502, 194501, 194500, 194499, 194498, 194497, 194496, 194495, 194494, 194493, 194492, 194491, 194490, 194489, 194488, 194487, 194486, 194485, 194484, 194483, 194482, 194481, 194480, 194479, 194478, 194477, 194476, 194475, 194474, 194473, 194472, 194471, 194470, 194469, 194468, 194467, 194466, 194465, 194464, 194463, 194462, 194461, 194460, 194459, 194458, 194457, 194456, 194455, 194454, 194453, 194452, 194451, 194450, 194449, 194448, 194447, 194446, 194445, 194444, 194443, 194442, 194441, 194440, 194439, 194438, 194437, 194436, 194435, 194434, 194433, 194432, 194431, 194430, 194429, 194428, 194427, 194426, 194425, 194424, 194423, 194422, 194421, 194420, 194419, 194418, 194417, 194416, 194415, 194414, 194413, 194412, 194411, 194410, 194409, 194408, 194407, 194406, 194405, 194404, 194403, 194402, 194401, 194400, 194399, 194398, 194397, 194396, 194395, 194394, 194393, 194392, 194391, 194390, 194389, 194388, 194387, 194386, 194385, 194384, 194383, 194382, 194381, 194380, 194379, 194378, 194377, 194376, 194375, 194374, 194373, 194372, 194371, 194370, 194369, 194368, 194367, 194366, 194365, 194364, 194363, 194362, 194361, 194360, 194359, 194358, 194357, 194356, 194355, 194354, 194353, 194352, 194351, 194350, 194349, 194348, 194347, 194346, 194345, 194344, 194343, 194342, 194341, 194340, 194339, 194338, 194337, 194336, 194335, 194334, 194333, 194332, 194331, 194330, 194329, 194328, 194327, 194326, 194325, 194324, 194323, 194322, 194321, 194320, 194319, 194318, 194317, 194316, 194315, 194314, 194313, 194312, 194311, 194310, 194309, 194308, 194307, 194306, 194305, 194304, 194303, 194302, 194301, 194300, 194299, 194298, 194297, 194296, 194295, 194294, 194293, 194292, 194291, 194290, 194289, 194288, 194287, 194286, 194285, 194284, 194283, 194282, 194281, 194280, 194279, 194278, 194277, 194276, 194275, 194274, 194273, 194272, 194271, 194270, 194269, 194268, 194267, 194266, 194265, 194264, 194263, 194262, 194261, 194260, 194259, 194258, 194257, 194256, 194255, 194254, 194253, 194252, 194251, 194250, 194249, 194248, 194247, 194246, 194245, 194244, 194243, 194242, 194241, 194240, 194239, 194238, 194237, 194236, 194235, 194234, 194233, 194232, 194231, 194230, 194229, 194228, 194227, 194226, 194225, 194224, 194223, 194222, 194221, 194220, 194219, 194218, 194217, 194216, 194215, 194214, 194213, 194212, 194211, 194210, 194209, 194208, 194207, 194206, 194205, 194204, 194203, 194202, 194201, 194200, 194199, 194198, 194197, 194196, 194195, 194194, 194193, 194192, 194191, 194190, 194189, 194188, 194187, 194186, 194185, 194184, 194183, 194182, 194181, 194180, 194179, 194178, 194177, 194176, 194175, 194174, 194173, 194172, 194171, 194170, 194169, 194168, 194167, 194166, 194165, 194164, 194163, 194162, 194161, 194160, 194159, 194158, 194157, 194156, 194155, 194154, 194153, 194152, 194151, 194150, 194149, 194148, 194147, 194146, 194145, 194144, 194143, 194142, 194141, 194140, 194139, 194138, 194137, 194136, 194135, 194134, 194133, 194132, 194131, 194130, 194129, 194128, 194127, 194126, 194125, 194124, 194123, 194122, 194121, 194120, 194119, 194118, 194117, 194116, 194115, 194114, 194113, 194112, 194111, 194110, 194109, 194108, 194107, 194106, 194105, 194104, 194103, 194102, 194101, 194100, 194099, 194098, 194097, 194096, 194095, 194094, 194093, 194092, 194091, 194090, 194089, 194088, 194087, 194086, 194085, 194084, 194083, 194082, 194081, 194080, 194079, 194078, 194077, 194076, 194075, 194074, 194073, 194072, 194071, 194070, 194069, 194068, 194067, 194066, 194065, 194064, 194063, 194062, 194061, 194060, 194059, 194058, 194057, 194056, 194055, 194054, 194053, 194052, 194051, 194050, 194049, 194048, 194047, 194046, 194045, 194044, 194043, 194042, 194041, 194040, 194039, 194038, 194037, 194036, 194035, 194034, 194033, 194032, 194031, 194030, 194029, 194028, 194027, 194026, 194025, 194024, 194023, 194022, 194021, 194020, 194019, 194018, 194017, 194016, 194015, 194014, 194013, 194012, 194011, 194010, 194009, 194008, 194007, 194006, 194005, 194004, 194003, 194002, 194001, 194000, 193999, 193998, 193997, 193996, 193995, 193994, 193993, 193992, 193991, 193990, 193989, 193988, 193987, 193986, 193985, 193984, 193983, 193982, 193981, 193980, 193979, 193978, 193977, 193976, 193975, 193974, 193973, 193972, 193971, 193970, 193969, 193968, 193967, 193966, 193965, 193964, 193963, 193962, 193961, 193960, 193959, 193958, 193957, 193956, 193955, 193954, 193953, 193952, 193951, 193950, 193949, 193948, 193947, 193946, 193945, 193944, 193943, 193942, 193941, 193940, 193939, 193938, 193937, 193936, 193935, 193934, 193933, 193932, 193931, 193930, 193929, 193928, 193927, 193926, 193925, 193924, 193923, 193922, 193921, 193920, 193919, 193918, 193917, 193916, 193915, 193914, 193913, 193912, 193911, 193910, 193909, 193908, 193907, 193906, 193905, 193904, 193903, 193902, 193901, 193900, 193899, 193898, 193897, 193896, 193895, 193894, 193893, 193892, 193891, 193890, 193889, 193888, 193887, 193886, 193885, 193884, 193883, 193882, 193881, 193880, 193879, 193878, 193877, 193876, 193875, 193874, 193873, 193872, 193871, 193870, 193869, 193868, 193867, 193866, 193865, 193864, 193863, 193862, 193861, 193860, 193859, 193858, 193857, 193856, 193855, 193854, 193853, 193852, 193851, 193850, 193849, 193848, 193847, 193846, 193845, 193844, 193843, 193842, 193841, 193840, 193839, 193838, 193837, 193836, 193835, 193834, 193833, 193832, 193831, 193830, 193829, 193828, 193827, 193826, 193825, 193824, 193823, 193822, 193821, 193820, 193819, 193818, 193817, 193816, 193815, 193814, 193813, 193812, 193811, 193810, 193809, 193808, 193807, 193806, 193805, 193804, 193803, 193802, 193801, 193800, 193799, 193798, 193797, 193796, 193795, 193794, 193793, 193792, 193791, 193790, 193789, 193788, 193787, 193786, 193785, 193784, 193783, 193782, 193781, 193780, 193779, 193778, 193777, 193776, 193775, 193774, 193773, 193772, 193771, 193770, 193769, 193768, 193767, 193766, 193765, 193764, 193763, 193762, 193761, 193760, 193759, 193758, 193757, 193756, 193755, 193754, 193753, 193752, 193751, 193750, 193749, 193748, 193747, 193746, 193745, 193744, 193743, 193742, 193741, 193740, 193739, 193738, 193737, 193736, 193735, 193734, 193733, 193732, 193731, 193730, 193729, 193728, 193727, 193726, 193725, 193724, 193723, 193722, 193721, 193720, 193719, 193718, 193717, 193716, 193715, 193714, 193713, 193712, 193711, 193710, 193709, 193708, 193707, 193706, 193705, 193704, 193703, 193702, 193701, 193700, 193699, 193698, 193697, 193696, 193695, 193694, 193693, 193692, 193691, 193690, 193689, 193688, 193687, 193686, 193685, 193684, 193683, 193682, 193681, 193680, 193679, 193678, 193677, 193676, 193675, 193674, 193673, 193672, 193671, 193670, 193669, 193668, 193667, 193666, 193665, 193664, 193663, 193662, 193661, 193660, 193659, 193658, 193657, 193656, 193655, 193654, 193653, 193652, 193651, 193650, 193649, 193648, 193647, 193646, 193645, 193644, 193643, 193642, 193641, 193640, 193639, 193638, 193637, 193636, 193635, 193634, 193633, 193632, 193631, 193630, 193629, 193628, 193627, 193626, 193625, 193624, 193623, 193622, 193621, 193620, 193619, 193618, 193617, 193616, 193615, 193614, 193613, 193612, 193611, 193610, 193609, 193608, 193607, 193606, 193605, 193604, 193603, 193602, 193601, 193600, 193599, 193598, 193597, 193596, 193595, 193594, 193593, 193592, 193591, 193590, 193589, 193588, 193587, 193586, 193585, 193584, 193583, 193582, 193581, 193580, 193579, 193578, 193577, 193576, 193575, 193574, 193573, 193572, 193571, 193570, 193569, 193568, 193567, 193566, 193565, 193564, 193563, 193562, 193561, 193560, 193559, 193558, 193557, 193556, 193555, 193554, 193553, 193552, 193551, 193550, 193549, 193548, 193547, 193546, 193545, 193544, 193543, 193542, 193541, 193540, 193539, 193538, 193537, 193536, 193535, 193534, 193533, 193532, 193531, 193530, 193529, 193528, 193527, 193526, 193525, 193524, 193523, 193522, 193521, 193520, 193519, 193518, 193517, 193516, 193515, 193514, 193513, 193512, 193511, 193510, 193509, 193508, 193507, 193506, 193505, 193504, 193503, 193502, 193501, 193500, 193499, 193498, 193497, 193496, 193495, 193494, 193493, 193492, 193491, 193490, 193489, 193488, 193487, 193486, 193485, 193484, 193483, 193482, 193481, 193480, 193479, 193478, 193477, 193476, 193475, 193474, 193473, 193472, 193471, 193470, 193469, 193468, 193467, 193466, 193465, 193464, 193463, 193462, 193461, 193460, 193459, 193458, 193457, 193456, 193455, 193454, 193453, 193452, 193451, 193450, 193449, 193448, 193447, 193446, 193445, 193444, 193443, 193442, 193441, 193440, 193439, 193438, 193437, 193436, 193435, 193434, 193433, 193432, 193431, 193430, 193429, 193428, 193427, 193426, 193425, 193424, 193423, 193422, 193421, 193420, 193419, 193418, 193417, 193416, 193415, 193414, 193413, 193412, 193411, 193410, 193409, 193408, 193407, 193406, 193405, 193404, 193403, 193402, 193401, 193400, 193399, 193398, 193397, 193396, 193395, 193394, 193393, 193392, 193391, 193390, 193389, 193388, 193387, 193386, 193385, 193384, 193383, 193382, 193381, 193380, 193379, 193378, 193377, 193376, 193375, 193374, 193373, 193372, 193371, 193370, 193369, 193368, 193367, 193366, 193365, 193364, 193363, 193362, 193361, 193360, 193359, 193358, 193357, 193356, 193355, 193354, 193353, 193352, 193351, 193350, 193349, 193348, 193347, 193346, 193345, 193344, 193343, 193342, 193341, 193340, 193339, 193338, 193337, 193336, 193335, 193334, 193333, 193332, 193331, 193330, 193329, 193328, 193327, 193326, 193325, 193324, 193323, 193322, 193321, 193320, 193319, 193318, 193317, 193316, 193315, 193314, 193313, 193312, 193311, 193310, 193309, 193308, 193307, 193306, 193305, 193304, 193303, 193302, 193301, 193300, 193299, 193298, 193297, 193296, 193295, 193294, 193293, 193292, 193291, 193290, 193289, 193288, 193287, 193286, 193285, 193284, 193283, 193282, 193281, 193280, 193279, 193278, 193277, 193276, 193275, 193274, 193273, 193272, 193271, 193270, 193269, 193268, 193267, 193266, 193265, 193264, 193263, 193262, 193261, 193260, 193259, 193258, 193257, 193256, 193255, 193254, 193253, 193252, 193251, 193250, 193249, 193248, 193247, 193246, 193245, 193244, 193243, 193242, 193241, 193240, 193239, 193238, 193237, 193236, 193235, 193234, 193233, 193232, 193231, 193230, 193229, 193228, 193227, 193226, 193225, 193224, 193223, 193222, 193221, 193220, 193219, 193218, 193217, 193216, 193215, 193214, 193213, 193212, 193211, 193210, 193209, 193208, 193207, 193206, 193205, 193204, 193203, 193202, 193201, 193200, 193199, 193198, 193197, 193196, 193195, 193194, 193193, 193192, 193191, 193190, 193189, 193188, 193187, 193186, 193185, 193184, 193183, 193182, 193181, 193180, 193179, 193178, 193177, 193176, 193175, 193174, 193173, 193172, 193171, 193170, 193169, 193168, 193167, 193166, 193165, 193164, 193163, 193162, 193161, 193160, 193159, 193158, 193157, 193156, 193155, 193154, 193153, 193152, 193151, 193150, 193149, 193148, 193147, 193146, 193145, 193144, 193143, 193142, 193141, 193140, 193139, 193138, 193137, 193136, 193135, 193134, 193133, 193132, 193131, 193130, 193129, 193128, 193127, 193126, 193125, 193124, 193123, 193122, 193121, 193120, 193119, 193118, 193117, 193116, 193115, 193114, 193113, 193112, 193111, 193110, 193109, 193108, 193107, 193106, 193105, 193104, 193103, 193102, 193101, 193100, 193099, 193098, 193097, 193096, 193095, 193094, 193093, 193092, 193091, 193090, 193089, 193088, 193087, 193086, 193085, 193084, 193083, 193082, 193081, 193080, 193079, 193078, 193077, 193076, 193075, 193074, 193073, 193072, 193071, 193070, 193069, 193068, 193067, 193066, 193065, 193064, 193063, 193062, 193061, 193060, 193059, 193058, 193057, 193056, 193055, 193054, 193053, 193052, 193051, 193050, 193049, 193048, 193047, 193046, 193045, 193044, 193043, 193042, 193041, 193040, 193039, 193038, 193037, 193036, 193035, 193034, 193033, 193032, 193031, 193030, 193029, 193028, 193027, 193026, 193025, 193024, 193023, 193022, 193021, 193020, 193019, 193018, 193017, 193016, 193015, 193014, 193013, 193012, 193011, 193010, 193009, 193008, 193007, 193006, 193005, 193004, 193003, 193002, 193001, 193000, 192999, 192998, 192997, 192996, 192995, 192994, 192993, 192992, 192991, 192990, 192989, 192988, 192987, 192986, 192985, 192984, 192983, 192982, 192981, 192980, 192979, 192978, 192977, 192976, 192975, 192974, 192973, 192972, 192971, 192970, 192969, 192968, 192967, 192966, 192965, 192964, 192963, 192962, 192961, 192960, 192959, 192958, 192957, 192956, 192955, 192954, 192953, 192952, 192951, 192950, 192949, 192948, 192947, 192946, 192945, 192944, 192943, 192942, 192941, 192940, 192939, 192938, 192937, 192936, 192935, 192934, 192933, 192932, 192931, 192930, 192929, 192928, 192927, 192926, 192925, 192924, 192923, 192922, 192921, 192920, 192919, 192918, 192917, 192916, 192915, 192914, 192913, 192912, 192911, 192910, 192909, 192908, 192907, 192906, 192905, 192904, 192903, 192902, 192901, 192900, 192899, 192898, 192897, 192896, 192895, 192894, 192893, 192892, 192891, 192890, 192889, 192888, 192887, 192886, 192885, 192884, 192883, 192882, 192881, 192880, 192879, 192878, 192877, 192876, 192875, 192874, 192873, 192872, 192871, 192870, 192869, 192868, 192867, 192866, 192865, 192864, 192863, 192862, 192861, 192860, 192859, 192858, 192857, 192856, 192855, 192854, 192853, 192852, 192851, 192850, 192849, 192848, 192847, 192846, 192845, 192844, 192843, 192842, 192841, 192840, 192839, 192838, 192837, 192836, 192835, 192834, 192833, 192832, 192831, 192830, 192829, 192828, 192827, 192826, 192825, 192824, 192823, 192822, 192821, 192820, 192819, 192818, 192817, 192816, 192815, 192814, 192813, 192812, 192811, 192810, 192809, 192808, 192807, 192806, 192805, 192804, 192803, 192802, 192801, 192800, 192799, 192798, 192797, 192796, 192795, 192794, 192793, 192792, 192791, 192790, 192789, 192788, 192787, 192786, 192785, 192784, 192783, 192782, 192781, 192780, 192779, 192778, 192777, 192776, 192775, 192774, 192773, 192772, 192771, 192770, 192769, 192768, 192767, 192766, 192765, 192764, 192763, 192762, 192761, 192760, 192759, 192758, 192757, 192756, 192755, 192754, 192753, 192752, 192751, 192750, 192749, 192748, 192747, 192746, 192745, 192744, 192743, 192742, 192741, 192740, 192739, 192738, 192737, 192736, 192735, 192734, 192733, 192732, 192731, 192730, 192729, 192728, 192727, 192726, 192725, 192724, 192723, 192722, 192721, 192720, 192719, 192718, 192717, 192716, 192715, 192714, 192713, 192712, 192711, 192710, 192709, 192708, 192707, 192706, 192705, 192704, 192703, 192702, 192701, 192700, 192699, 192698, 192697, 192696, 192695, 192694, 192693, 192692, 192691, 192690, 192689, 192688, 192687, 192686, 192685, 192684, 192683, 192682, 192681, 192680, 192679, 192678, 192677, 192676, 192675, 192674, 192673, 192672, 192671, 192670, 192669, 192668, 192667, 192666, 192665, 192664, 192663, 192662, 192661, 192660, 192659, 192658, 192657, 192656, 192655, 192654, 192653, 192652, 192651, 192650, 192649, 192648, 192647, 192646, 192645, 192644, 192643, 192642, 192641, 192640, 192639, 192638, 192637, 192636, 192635, 192634, 192633, 192632, 192631, 192630, 192629, 192628, 192627, 192626, 192625, 192624, 192623, 192622, 192621, 192620, 192619, 192618, 192617, 192616, 192615, 192614, 192613, 192612, 192611, 192610, 192609, 192608, 192607, 192606, 192605, 192604, 192603, 192602, 192601, 192600, 192599, 192598, 192597, 192596, 192595, 192594, 192593, 192592, 192591, 192590, 192589, 192588, 192587, 192586, 192585, 192584, 192583, 192582, 192581, 192580, 192579, 192578, 192577, 192576, 192575, 192574, 192573, 192572, 192571, 192570, 192569, 192568, 192567, 192566, 192565, 192564, 192563, 192562, 192561, 192560, 192559, 192558, 192557, 192556, 192555, 192554, 192553, 192552, 192551, 192550, 192549, 192548, 192547, 192546, 192545, 192544, 192543, 192542, 192541, 192540, 192539, 192538, 192537, 192536, 192535, 192534, 192533, 192532, 192531, 192530, 192529, 192528, 192527, 192526, 192525, 192524, 192523, 192522, 192521, 192520, 192519, 192518, 192517, 192516, 192515, 192514, 192513, 192512, 192511, 192510, 192509, 192508, 192507, 192506, 192505, 192504, 192503, 192502, 192501, 192500, 192499, 192498, 192497, 192496, 192495, 192494, 192493, 192492, 192491, 192490, 192489, 192488, 192487, 192486, 192485, 192484, 192483, 192482, 192481, 192480, 192479, 192478, 192477, 192476, 192475, 192474, 192473, 192472, 192471, 192470, 192469, 192468, 192467, 192466, 192465, 192464, 192463, 192462, 192461, 192460, 192459, 192458, 192457, 192456, 192455, 192454, 192453, 192452, 192451, 192450, 192449, 192448, 192447, 192446, 192445, 192444, 192443, 192442, 192441, 192440, 192439, 192438, 192437, 192436, 192435, 192434, 192433, 192432, 192431, 192430, 192429, 192428, 192427, 192426, 192425, 192424, 192423, 192422, 192421, 192420, 192419, 192418, 192417, 192416, 192415, 192414, 192413, 192412, 192411, 192410, 192409, 192408, 192407, 192406, 192405, 192404, 192403, 192402, 192401, 192400, 192399, 192398, 192397, 192396, 192395, 192394, 192393, 192392, 192391, 192390, 192389, 192388, 192387, 192386, 192385, 192384, 192383, 192382, 192381, 192380, 192379, 192378, 192377, 192376, 192375, 192374, 192373, 192372, 192371, 192370, 192369, 192368, 192367, 192366, 192365, 192364, 192363, 192362, 192361, 192360, 192359, 192358, 192357, 192356, 192355, 192354, 192353, 192352, 192351, 192350, 192349, 192348, 192347, 192346, 192345, 192344, 192343, 192342, 192341, 192340, 192339, 192338, 192337, 192336, 192335, 192334, 192333, 192332, 192331, 192330, 192329, 192328, 192327, 192326, 192325, 192324, 192323, 192322, 192321, 192320, 192319, 192318, 192317, 192316, 192315, 192314, 192313, 192312, 192311, 192310, 192309, 192308, 192307, 192306, 192305, 192304, 192303, 192302, 192301, 192300, 192299, 192298, 192297, 192296, 192295, 192294, 192293, 192292, 192291, 192290, 192289, 192288, 192287, 192286, 192285, 192284, 192283, 192282, 192281, 192280, 192279, 192278, 192277, 192276, 192275, 192274, 192273, 192272, 192271, 192270, 192269, 192268, 192267, 192266, 192265, 192264, 192263, 192262, 192261, 192260, 192259, 192258, 192257, 192256, 192255, 192254, 192253, 192252, 192251, 192250, 192249, 192248, 192247, 192246, 192245, 192244, 192243, 192242, 192241, 192240, 192239, 192238, 192237, 192236, 192235, 192234, 192233, 192232, 192231, 192230, 192229, 192228, 192227, 192226, 192225, 192224, 192223, 192222, 192221, 192220, 192219, 192218, 192217, 192216, 192215, 192214, 192213, 192212, 192211, 192210, 192209, 192208, 192207, 192206, 192205, 192204, 192203, 192202, 192201, 192200, 192199, 192198, 192197, 192196, 192195, 192194, 192193, 192192, 192191, 192190, 192189, 192188, 192187, 192186, 192185, 192184, 192183, 192182, 192181, 192180, 192179, 192178, 192177, 192176, 192175, 192174, 192173, 192172, 192171, 192170, 192169, 192168, 192167, 192166, 192165, 192164, 192163, 192162, 192161, 192160, 192159, 192158, 192157, 192156, 192155, 192154, 192153, 192152, 192151, 192150, 192149, 192148, 192147, 192146, 192145, 192144, 192143, 192142, 192141, 192140, 192139, 192138, 192137, 192136, 192135, 192134, 192133, 192132, 192131, 192130, 192129, 192128, 192127, 192126, 192125, 192124, 192123, 192122, 192121, 192120, 192119, 192118, 192117, 192116, 192115, 192114, 192113, 192112, 192111, 192110, 192109, 192108, 192107, 192106, 192105, 192104, 192103, 192102, 192101, 192100, 192099, 192098, 192097, 192096, 192095, 192094, 192093, 192092, 192091, 192090, 192089, 192088, 192087, 192086, 192085, 192084, 192083, 192082, 192081, 192080, 192079, 192078, 192077, 192076, 192075, 192074, 192073, 192072, 192071, 192070, 192069, 192068, 192067, 192066, 192065, 192064, 192063, 192062, 192061, 192060, 192059, 192058, 192057, 192056, 192055, 192054, 192053, 192052, 192051, 192050, 192049, 192048, 192047, 192046, 192045, 192044, 192043, 192042, 192041, 192040, 192039, 192038, 192037, 192036, 192035, 192034, 192033, 192032, 192031, 192030, 192029, 192028, 192027, 192026, 192025, 192024, 192023, 192022, 192021, 192020, 192019, 192018, 192017, 192016, 192015, 192014, 192013, 192012, 192011, 192010, 192009, 192008, 192007, 192006, 192005, 192004, 192003, 192002, 192001, 192000, 191999, 191998, 191997, 191996, 191995, 191994, 191993, 191992, 191991, 191990, 191989, 191988, 191987, 191986, 191985, 191984, 191983, 191982, 191981, 191980, 191979, 191978, 191977, 191976, 191975, 191974, 191973, 191972, 191971, 191970, 191969, 191968, 191967, 191966, 191965, 191964, 191963, 191962, 191961, 191960, 191959, 191958, 191957, 191956, 191955, 191954, 191953, 191952, 191951, 191950, 191949, 191948, 191947, 191946, 191945, 191944, 191943, 191942, 191941, 191940, 191939, 191938, 191937, 191936, 191935, 191934, 191933, 191932, 191931, 191930, 191929, 191928, 191927, 191926, 191925, 191924, 191923, 191922, 191921, 191920, 191919, 191918, 191917, 191916, 191915, 191914, 191913, 191912, 191911, 191910, 191909, 191908, 191907, 191906, 191905, 191904, 191903, 191902, 191901, 191900, 191899, 191898, 191897, 191896, 191895, 191894, 191893, 191892, 191891, 191890, 191889, 191888, 191887, 191886, 191885, 191884, 191883, 191882, 191881, 191880, 191879, 191878, 191877, 191876, 191875, 191874, 191873, 191872, 191871, 191870, 191869, 191868, 191867, 191866, 191865, 191864, 191863, 191862, 191861, 191860, 191859, 191858, 191857, 191856, 191855, 191854, 191853, 191852, 191851, 191850, 191849, 191848, 191847, 191846, 191845, 191844, 191843, 191842, 191841, 191840, 191839, 191838, 191837, 191836, 191835, 191834, 191833, 191832, 191831, 191830, 191829, 191828, 191827, 191826, 191825, 191824, 191823, 191822, 191821, 191820, 191819, 191818, 191817, 191816, 191815, 191814, 191813, 191812, 191811, 191810, 191809, 191808, 191807, 191806, 191805, 191804, 191803, 191802, 191801, 191800, 191799, 191798, 191797, 191796, 191795, 191794, 191793, 191792, 191791, 191790, 191789, 191788, 191787, 191786, 191785, 191784, 191783, 191782, 191781, 191780, 191779, 191778, 191777, 191776, 191775, 191774, 191773, 191772, 191771, 191770, 191769, 191768, 191767, 191766, 191765, 191764, 191763, 191762, 191761, 191760, 191759, 191758, 191757, 191756, 191755, 191754, 191753, 191752, 191751, 191750, 191749, 191748, 191747, 191746, 191745, 191744, 191743, 191742, 191741, 191740, 191739, 191738, 191737, 191736, 191735, 191734, 191733, 191732, 191731, 191730, 191729, 191728, 191727, 191726, 191725, 191724, 191723, 191722, 191721, 191720, 191719, 191718, 191717, 191716, 191715, 191714, 191713, 191712, 191711, 191710, 191709, 191708, 191707, 191706, 191705, 191704, 191703, 191702, 191701, 191700, 191699, 191698, 191697, 191696, 191695, 191694, 191693, 191692, 191691, 191690, 191689, 191688, 191687, 191686, 191685, 191684, 191683, 191682, 191681, 191680, 191679, 191678, 191677, 191676, 191675, 191674, 191673, 191672, 191671, 191670, 191669, 191668, 191667, 191666, 191665, 191664, 191663, 191662, 191661, 191660, 191659, 191658, 191657, 191656, 191655, 191654, 191653, 191652, 191651, 191650, 191649, 191648, 191647, 191646, 191645, 191644, 191643, 191642, 191641, 191640, 191639, 191638, 191637, 191636, 191635, 191634, 191633, 191632, 191631, 191630, 191629, 191628, 191627, 191626, 191625, 191624, 191623, 191622, 191621, 191620, 191619, 191618, 191617, 191616, 191615, 191614, 191613, 191612, 191611, 191610, 191609, 191608, 191607, 191606, 191605, 191604, 191603, 191602, 191601, 191600, 191599, 191598, 191597, 191596, 191595, 191594, 191593, 191592, 191591, 191590, 191589, 191588, 191587, 191586, 191585, 191584, 191583, 191582, 191581, 191580, 191579, 191578, 191577, 191576, 191575, 191574, 191573, 191572, 191571, 191570, 191569, 191568, 191567, 191566, 191565, 191564, 191563, 191562, 191561, 191560, 191559, 191558, 191557, 191556, 191555, 191554, 191553, 191552, 191551, 191550, 191549, 191548, 191547, 191546, 191545, 191544, 191543, 191542, 191541, 191540, 191539, 191538, 191537, 191536, 191535, 191534, 191533, 191532, 191531, 191530, 191529, 191528, 191527, 191526, 191525, 191524, 191523, 191522, 191521, 191520, 191519, 191518, 191517, 191516, 191515, 191514, 191513, 191512, 191511, 191510, 191509, 191508, 191507, 191506, 191505, 191504, 191503, 191502, 191501, 191500, 191499, 191498, 191497, 191496, 191495, 191494, 191493, 191492, 191491, 191490, 191489, 191488, 191487, 191486, 191485, 191484, 191483, 191482, 191481, 191480, 191479, 191478, 191477, 191476, 191475, 191474, 191473, 191472, 191471, 191470, 191469, 191468, 191467, 191466, 191465, 191464, 191463, 191462, 191461, 191460, 191459, 191458, 191457, 191456, 191455, 191454, 191453, 191452, 191451, 191450, 191449, 191448, 191447, 191446, 191445, 191444, 191443, 191442, 191441, 191440, 191439, 191438, 191437, 191436, 191435, 191434, 191433, 191432, 191431, 191430, 191429, 191428, 191427, 191426, 191425, 191424, 191423, 191422, 191421, 191420, 191419, 191418, 191417, 191416, 191415, 191414, 191413, 191412, 191411, 191410, 191409, 191408, 191407, 191406, 191405, 191404, 191403, 191402, 191401, 191400, 191399, 191398, 191397, 191396, 191395, 191394, 191393, 191392, 191391, 191390, 191389, 191388, 191387, 191386, 191385, 191384, 191383, 191382, 191381, 191380, 191379, 191378, 191377, 191376, 191375, 191374, 191373, 191372, 191371, 191370, 191369, 191368, 191367, 191366, 191365, 191364, 191363, 191362, 191361, 191360, 191359, 191358, 191357, 191356, 191355, 191354, 191353, 191352, 191351, 191350, 191349, 191348, 191347, 191346, 191345, 191344, 191343, 191342, 191341, 191340, 191339, 191338, 191337, 191336, 191335, 191334, 191333, 191332, 191331, 191330, 191329, 191328, 191327, 191326, 191325, 191324, 191323, 191322, 191321, 191320, 191319, 191318, 191317, 191316, 191315, 191314, 191313, 191312, 191311, 191310, 191309, 191308, 191307, 191306, 191305, 191304, 191303, 191302, 191301, 191300, 191299, 191298, 191297, 191296, 191295, 191294, 191293, 191292, 191291, 191290, 191289, 191288, 191287, 191286, 191285, 191284, 191283, 191282, 191281, 191280, 191279, 191278, 191277, 191276, 191275, 191274, 191273, 191272, 191271, 191270, 191269, 191268, 191267, 191266, 191265, 191264, 191263, 191262, 191261, 191260, 191259, 191258, 191257, 191256, 191255, 191254, 191253, 191252, 191251, 191250, 191249, 191248, 191247, 191246, 191245, 191244, 191243, 191242, 191241, 191240, 191239, 191238, 191237, 191236, 191235, 191234, 191233, 191232, 191231, 191230, 191229, 191228, 191227, 191226, 191225, 191224, 191223, 191222, 191221, 191220, 191219, 191218, 191217, 191216, 191215, 191214, 191213, 191212, 191211, 191210, 191209, 191208, 191207, 191206, 191205, 191204, 191203, 191202, 191201, 191200, 191199, 191198, 191197, 191196, 191195, 191194, 191193, 191192, 191191, 191190, 191189, 191188, 191187, 191186, 191185, 191184, 191183, 191182, 191181, 191180, 191179, 191178, 191177, 191176, 191175, 191174, 191173, 191172, 191171, 191170, 191169, 191168, 191167, 191166, 191165, 191164, 191163, 191162, 191161, 191160, 191159, 191158, 191157, 191156, 191155, 191154, 191153, 191152, 191151, 191150, 191149, 191148, 191147, 191146, 191145, 191144, 191143, 191142, 191141, 191140, 191139, 191138, 191137, 191136, 191135, 191134, 191133, 191132, 191131, 191130, 191129, 191128, 191127, 191126, 191125, 191124, 191123, 191122, 191121, 191120, 191119, 191118, 191117, 191116, 191115, 191114, 191113, 191112, 191111, 191110, 191109, 191108, 191107, 191106, 191105, 191104, 191103, 191102, 191101, 191100, 191099, 191098, 191097, 191096, 191095, 191094, 191093, 191092, 191091, 191090, 191089, 191088, 191087, 191086, 191085, 191084, 191083, 191082, 191081, 191080, 191079, 191078, 191077, 191076, 191075, 191074, 191073, 191072, 191071, 191070, 191069, 191068, 191067, 191066, 191065, 191064, 191063, 191062, 191061, 191060, 191059, 191058, 191057, 191056, 191055, 191054, 191053, 191052, 191051, 191050, 191049, 191048, 191047, 191046, 191045, 191044, 191043, 191042, 191041, 191040, 191039, 191038, 191037, 191036, 191035, 191034, 191033, 191032, 191031, 191030, 191029, 191028, 191027, 191026, 191025, 191024, 191023, 191022, 191021, 191020, 191019, 191018, 191017, 191016, 191015, 191014, 191013, 191012, 191011, 191010, 191009, 191008, 191007, 191006, 191005, 191004, 191003, 191002, 191001, 191000, 190999, 190998, 190997, 190996, 190995, 190994, 190993, 190992, 190991, 190990, 190989, 190988, 190987, 190986, 190985, 190984, 190983, 190982, 190981, 190980, 190979, 190978, 190977, 190976, 190975, 190974, 190973, 190972, 190971, 190970, 190969, 190968, 190967, 190966, 190965, 190964, 190963, 190962, 190961, 190960, 190959, 190958, 190957, 190956, 190955, 190954, 190953, 190952, 190951, 190950, 190949, 190948, 190947, 190946, 190945, 190944, 190943, 190942, 190941, 190940, 190939, 190938, 190937, 190936, 190935, 190934, 190933, 190932, 190931, 190930, 190929, 190928, 190927, 190926, 190925, 190924, 190923, 190922, 190921, 190920, 190919, 190918, 190917, 190916, 190915, 190914, 190913, 190912, 190911, 190910, 190909, 190908, 190907, 190906, 190905, 190904, 190903, 190902, 190901, 190900, 190899, 190898, 190897, 190896, 190895, 190894, 190893, 190892, 190891, 190890, 190889, 190888, 190887, 190886, 190885, 190884, 190883, 190882, 190881, 190880, 190879, 190878, 190877, 190876, 190875, 190874, 190873, 190872, 190871, 190870, 190869, 190868, 190867, 190866, 190865, 190864, 190863, 190862, 190861, 190860, 190859, 190858, 190857, 190856, 190855, 190854, 190853, 190852, 190851, 190850, 190849, 190848, 190847, 190846, 190845, 190844, 190843, 190842, 190841, 190840, 190839, 190838, 190837, 190836, 190835, 190834, 190833, 190832, 190831, 190830, 190829, 190828, 190827, 190826, 190825, 190824, 190823, 190822, 190821, 190820, 190819, 190818, 190817, 190816, 190815, 190814, 190813, 190812, 190811, 190810, 190809, 190808, 190807, 190806, 190805, 190804, 190803, 190802, 190801, 190800, 190799, 190798, 190797, 190796, 190795, 190794, 190793, 190792, 190791, 190790, 190789, 190788, 190787, 190786, 190785, 190784, 190783, 190782, 190781, 190780, 190779, 190778, 190777, 190776, 190775, 190774, 190773, 190772, 190771, 190770, 190769, 190768, 190767, 190766, 190765, 190764, 190763, 190762, 190761, 190760, 190759, 190758, 190757, 190756, 190755, 190754, 190753, 190752, 190751, 190750, 190749, 190748, 190747, 190746, 190745, 190744, 190743, 190742, 190741, 190740, 190739, 190738, 190737, 190736, 190735, 190734, 190733, 190732, 190731, 190730, 190729, 190728, 190727, 190726, 190725, 190724, 190723, 190722, 190721, 190720, 190719, 190718, 190717, 190716, 190715, 190714, 190713, 190712, 190711, 190710, 190709, 190708, 190707, 190706, 190705, 190704, 190703, 190702, 190701, 190700, 190699, 190698, 190697, 190696, 190695, 190694, 190693, 190692, 190691, 190690, 190689, 190688, 190687, 190686, 190685, 190684, 190683, 190682, 190681, 190680, 190679, 190678, 190677, 190676, 190675, 190674, 190673, 190672, 190671, 190670, 190669, 190668, 190667, 190666, 190665, 190664, 190663, 190662, 190661, 190660, 190659, 190658, 190657, 190656, 190655, 190654, 190653, 190652, 190651, 190650, 190649, 190648, 190647, 190646, 190645, 190644, 190643, 190642, 190641, 190640, 190639, 190638, 190637, 190636, 190635, 190634, 190633, 190632, 190631, 190630, 190629, 190628, 190627, 190626, 190625, 190624, 190623, 190622, 190621, 190620, 190619, 190618, 190617, 190616, 190615, 190614, 190613, 190612, 190611, 190610, 190609, 190608, 190607, 190606, 190605, 190604, 190603, 190602, 190601, 190600, 190599, 190598, 190597, 190596, 190595, 190594, 190593, 190592, 190591, 190590, 190589, 190588, 190587, 190586, 190585, 190584, 190583, 190582, 190581, 190580, 190579, 190578, 190577, 190576, 190575, 190574, 190573, 190572, 190571, 190570, 190569, 190568, 190567, 190566, 190565, 190564, 190563, 190562, 190561, 190560, 190559, 190558, 190557, 190556, 190555, 190554, 190553, 190552, 190551, 190550, 190549, 190548, 190547, 190546, 190545, 190544, 190543, 190542, 190541, 190540, 190539, 190538, 190537, 190536, 190535, 190534, 190533, 190532, 190531, 190530, 190529, 190528, 190527, 190526, 190525, 190524, 190523, 190522, 190521, 190520, 190519, 190518, 190517, 190516, 190515, 190514, 190513, 190512, 190511, 190510, 190509, 190508, 190507, 190506, 190505, 190504, 190503, 190502, 190501, 190500, 190499, 190498, 190497, 190496, 190495, 190494, 190493, 190492, 190491, 190490, 190489, 190488, 190487, 190486, 190485, 190484, 190483, 190482, 190481, 190480, 190479, 190478, 190477, 190476, 190475, 190474, 190473, 190472, 190471, 190470, 190469, 190468, 190467, 190466, 190465, 190464, 190463, 190462, 190461, 190460, 190459, 190458, 190457, 190456, 190455, 190454, 190453, 190452, 190451, 190450, 190449, 190448, 190447, 190446, 190445, 190444, 190443, 190442, 190441, 190440, 190439, 190438, 190437, 190436, 190435, 190434, 190433, 190432, 190431, 190430, 190429, 190428, 190427, 190426, 190425, 190424, 190423, 190422, 190421, 190420, 190419, 190418, 190417, 190416, 190415, 190414, 190413, 190412, 190411, 190410, 190409, 190408, 190407, 190406, 190405, 190404, 190403, 190402, 190401, 190400, 190399, 190398, 190397, 190396, 190395, 190394, 190393, 190392, 190391, 190390, 190389, 190388, 190387, 190386, 190385, 190384, 190383, 190382, 190381, 190380, 190379, 190378, 190377, 190376, 190375, 190374, 190373, 190372, 190371, 190370, 190369, 190368, 190367, 190366, 190365, 190364, 190363, 190362, 190361, 190360, 190359, 190358, 190357, 190356, 190355, 190354, 190353, 190352, 190351, 190350, 190349, 190348, 190347, 190346, 190345, 190344, 190343, 190342, 190341, 190340, 190339, 190338, 190337, 190336, 190335, 190334, 190333, 190332, 190331, 190330, 190329, 190328, 190327, 190326, 190325, 190324, 190323, 190322, 190321, 190320, 190319, 190318, 190317, 190316, 190315, 190314, 190313, 190312, 190311, 190310, 190309, 190308, 190307, 190306, 190305, 190304, 190303, 190302, 190301, 190300, 190299, 190298, 190297, 190296, 190295, 190294, 190293, 190292, 190291, 190290, 190289, 190288, 190287, 190286, 190285, 190284, 190283, 190282, 190281, 190280, 190279, 190278, 190277, 190276, 190275, 190274, 190273, 190272, 190271, 190270, 190269, 190268, 190267, 190266, 190265, 190264, 190263, 190262, 190261, 190260, 190259, 190258, 190257, 190256, 190255, 190254, 190253, 190252, 190251, 190250, 190249, 190248, 190247, 190246, 190245, 190244, 190243, 190242, 190241, 190240, 190239, 190238, 190237, 190236, 190235, 190234, 190233, 190232, 190231, 190230, 190229, 190228, 190227, 190226, 190225, 190224, 190223, 190222, 190221, 190220, 190219, 190218, 190217, 190216, 190215, 190214, 190213, 190212, 190211, 190210, 190209, 190208, 190207, 190206, 190205, 190204, 190203, 190202, 190201, 190200, 190199, 190198, 190197, 190196, 190195, 190194, 190193, 190192, 190191, 190190, 190189, 190188, 190187, 190186, 190185, 190184, 190183, 190182, 190181, 190180, 190179, 190178, 190177, 190176, 190175, 190174, 190173, 190172, 190171, 190170, 190169, 190168, 190167, 190166, 190165, 190164, 190163, 190162, 190161, 190160, 190159, 190158, 190157, 190156, 190155, 190154, 190153, 190152, 190151, 190150, 190149, 190148, 190147, 190146, 190145, 190144, 190143, 190142, 190141, 190140, 190139, 190138, 190137, 190136, 190135, 190134, 190133, 190132, 190131, 190130, 190129, 190128, 190127, 190126, 190125, 190124, 190123, 190122, 190121, 190120, 190119, 190118, 190117, 190116, 190115, 190114, 190113, 190112, 190111, 190110, 190109, 190108, 190107, 190106, 190105, 190104, 190103, 190102, 190101, 190100, 190099, 190098, 190097, 190096, 190095, 190094, 190093, 190092, 190091, 190090, 190089, 190088, 190087, 190086, 190085, 190084, 190083, 190082, 190081, 190080, 190079, 190078, 190077, 190076, 190075, 190074, 190073, 190072, 190071, 190070, 190069, 190068, 190067, 190066, 190065, 190064, 190063, 190062, 190061, 190060, 190059, 190058, 190057, 190056, 190055, 190054, 190053, 190052, 190051, 190050, 190049, 190048, 190047, 190046, 190045, 190044, 190043, 190042, 190041, 190040, 190039, 190038, 190037, 190036, 190035, 190034, 190033, 190032, 190031, 190030, 190029, 190028, 190027, 190026, 190025, 190024, 190023, 190022, 190021, 190020, 190019, 190018, 190017, 190016, 190015, 190014, 190013, 190012, 190011, 190010, 190009, 190008, 190007, 190006, 190005, 190004, 190003, 190002, 190001, 190000, 189999, 189998, 189997, 189996, 189995, 189994, 189993, 189992, 189991, 189990, 189989, 189988, 189987, 189986, 189985, 189984, 189983, 189982, 189981, 189980, 189979, 189978, 189977, 189976, 189975, 189974, 189973, 189972, 189971, 189970, 189969, 189968, 189967, 189966, 189965, 189964, 189963, 189962, 189961, 189960, 189959, 189958, 189957, 189956, 189955, 189954, 189953, 189952, 189951, 189950, 189949, 189948, 189947, 189946,
2023/09/11-12:41:48.767879 140207160538880 [/db_impl/db_impl.cc:1115] ------- DUMPING STATS -------
2023/09/11-12:41:48.767973 140207160538880 [/db_impl/db_impl.cc:1116]
** DB Stats **
Uptime(secs): 45.0 total, 45.0 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

.. Compaction Stats [default]
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)

L0 198107/198107 10.93 TB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 198107/198107 10.93 TB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0

. Compaction Stats [default]
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)

Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0

Uptime(secs): 45.0 total, 45.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 1 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 1 total count
Block cache HyperClockCache@0x55aec5adb440#81287 capacity: 256.00 GB usage: 205.56 GB table_size: 134217728 occupancy: 2061072 collections: 1 last_copies: 0 last_secs: 9.40834 secs_since: 0
Block cache entry stats(count,size,portion): FilterBlock(41008013,143.94 GB,56.2267%) IndexBlock(41008013,53.62 GB,20.9439%) Misc(5761,22.53 MB,0.00859318%)

..** File Read Latency Histogram By Level [default] **

anand1976 · 2023-09-12T19:01:43Z

My guess is the sheer number of files in L0 being compacted (~190K), and the compaction_readahead_size of 2MB, causes a lot of memory to be consumed by the compaction iterator. Can you run a heap profile to confirm?
CC @cbi42

ski422 · 2023-09-13T01:42:17Z

My guess is the sheer number of files in L0 being compacted (~190K), and the compaction_readahead_size of 2MB, causes a lot of memory to be consumed by the compaction iterator. Can you run a heap profile to confirm? CC @cbi42

However, it looks like the default value of the compaction_readahead_size is 2MB and we cannot make it smaller.

From what I've debugged, calling SeekToFirst() method of MergingIterator (inside of ProcessKeyValueCompaction) consumes all the memory. To be specific, the first for-loop (as below) in SeekToFirst() method (in table/merging_iterator.cc) was executing before out-of-memory.

for (auto& child : children_) {
child.iter.SeekToFirst();
AddToMinHeapOrCheckStatus(&child);
}

cbi42 · 2023-09-14T00:32:44Z

From what I've debugged, calling SeekToFirst() method of MergingIterator (inside of ProcessKeyValueCompaction) consumes all the memory. To be specific, the first for-loop (as below) in SeekToFirst() method (in table/merging_iterator.cc) was executing before out-of-memory.

for (auto& child : children_) {
child.iter.SeekToFirst();
AddToMinHeapOrCheckStatus(&child);
}

Read ahead happens during the SeekToFirst() phase which is likely the reason for OOM. You can try reduce compaction_readahead_size or split the bulk load into smaller batches.

ski422 · 2023-09-15T05:35:51Z

Read ahead happens during the SeekToFirst() phase which is likely the reason for OOM. You can try reduce compaction_readahead_size or split the bulk load into smaller batches.

@cbi42 Thanks for your advise.

However, can you please explain more details about 'split the bulk load into smaller batches'?

I missed the term 'batch'. Is it about the compaction key range?

Moreover, I'd appreciate it if you could tell me how to recude the compaction_readahead_size in the db_bench run.

cbi42 · 2023-09-16T20:15:16Z

'split the bulk load into smaller batches'?

I meant load some L0 files and do compaction on them at a time, instead of loading all the L0 files and do a giant compaction.

how to recude the compaction_readahead_size in the db_bench run.

You should be able to configure it with ./db_bench --compaction_readahead_size=..

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Out-of-memory during 'compact' the results of bulk-load (~=10TB) #11809

Out-of-memory during 'compact' the results of bulk-load (~=10TB) #11809

ski422 commented Sep 9, 2023

anand1976 commented Sep 11, 2023

ski422 commented Sep 12, 2023

anand1976 commented Sep 12, 2023

ski422 commented Sep 13, 2023

cbi42 commented Sep 14, 2023

ski422 commented Sep 15, 2023

cbi42 commented Sep 16, 2023

Out-of-memory during 'compact' the results of bulk-load (~=10TB) #11809

Out-of-memory during 'compact' the results of bulk-load (~=10TB) #11809

Comments

ski422 commented Sep 9, 2023

Goal

Behavior

Question

anand1976 commented Sep 11, 2023

ski422 commented Sep 12, 2023

..** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)

.** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)

anand1976 commented Sep 12, 2023

ski422 commented Sep 13, 2023

cbi42 commented Sep 14, 2023

ski422 commented Sep 15, 2023

cbi42 commented Sep 16, 2023

.. Compaction Stats [default]
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)

. Compaction Stats [default]
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)