[sled-agent] Re-install Omicron Zones, rather than assuming they work (…

…#2340)

This forces deletion and re-installation of "installed-but-not-running"
zones which are discovered by the Sled Agent.

This is admittedly a bit of a heavy hammer - we could validate the
installation looks like what we'd hope - but it quickly mitigates some
issues seen, where a zone failing to install (and retrying later) does
not succeed, because the configuration gets out-of-date.

sled-agent/src/illumos/zone.rs

@@ -204,6 +204,11 @@ impl Zones {
         Ok(())
     }
 
+    /// Installs a zone with the provided arguments.
+    ///
+    /// - If a zone with the name `zone_name` exists and is currently running,
+    /// we return immediately.
+    /// - Otherwise, the zone is deleted.
     #[allow(clippy::too_many_arguments)]
     pub async fn install_omicron_zone(
         log: &Logger,
@@ -221,11 +226,9 @@ impl Zones {
                 zone.name(),
                 zone.state()
             );
-            if zone.state() == zone::State::Installed
-                || zone.state() == zone::State::Running
-            {
+            if zone.state() == zone::State::Running {
                 // TODO: Admittedly, the zone still might be messed up. However,
-                // for now, we assume that "installed" means "good to go".
+                // for now, we assume that "running" means "good to go".
                 return Ok(());
             } else {
                 info!(