ledger-tool: Make joining AccountsBackgroundService optional

AccountsBackgroundService performs several operations that can take a
long time to complete and do not check the exit flag mid-operation.
Thus, ledger-tool can get hung up for a while waiting for ABS to finish.

The majority of the ledger-tool commands that process a ledger do not
need to wait for ABS to finish in order to perform the desired work. So,
return a handle to the ABS thread and allow the caller to decide whether
to join ABS or not.

As of right now, create-snapshot is the only command that requires ABS
to have finished

ledger-tool/src/ledger_utils.rs

@@ -57,6 +57,14 @@ use {
 pub(crate) struct LoadAndProcessLedgerOutput {
     pub bank_forks: Arc<RwLock<BankForks>>,
     pub starting_snapshot_hashes: Option<StartingSnapshotHashes>,
+    // Typically, we would want to join all threads before returning. However,
+    // AccountsBackgroundService (ABS) performs several long running operations
+    // that don't respond to the exit flag. Blocking on these operations could
+    // significantly delay getting results that do not need ABS to finish. So,
+    // skip joining ABS and instead let the caller decide whether to block or
+    // not. It is safe to let ABS continue in the background, and ABS will stop
+    // if/when it finally checks the exit flag
+    pub accounts_background_service: AccountsBackgroundService,
 }
 
 const PROCESS_SLOTS_HELP_STRING: &str =
@@ -411,11 +419,11 @@ pub fn load_and_process_ledger(
     .map(|_| LoadAndProcessLedgerOutput {
         bank_forks,
         starting_snapshot_hashes,
+        accounts_background_service,
     })
     .map_err(LoadAndProcessLedgerError::ProcessBlockstoreFromRoot);
 
     exit.store(true, Ordering::Relaxed);
-    accounts_background_service.join().unwrap();
     accounts_hash_verifier.join().unwrap();
     if let Some(service) = transaction_status_service {
         service.join().unwrap();

ledger-tool/src/main.rs

@@ -2040,6 +2040,7 @@ fn main() {
                     let LoadAndProcessLedgerOutput {
                         bank_forks,
                         starting_snapshot_hashes,
+                        accounts_background_service,
                     } = load_and_process_ledger_or_exit(
                         arg_matches,
                         &genesis_config,
@@ -2049,6 +2050,11 @@ fn main() {
                         incremental_snapshot_archive_path,
                         None,
                     );
+                    // Snapshot creation will implicitly perform AccountsDb
+                    // flush and clean operations. These operations cannot be
+                    // run concurrently, so ensure ABS is stopped to avoid that
+                    // possibility.
+                    accounts_background_service.join().unwrap();
                     let mut bank = bank_forks
                         .read()
                         .unwrap()