remote backend fails in un-initted working directory with prefixed workspaces

[nfagerlund]

Terraform Version

• 0.11.13
• 0.12.0 beta 2

Terraform Configuration Files

terraform {
  backend "remote" {
    organization = "examplecorp-free"

    workspaces {
      prefix = "apiserver-"
    }
  }
}

Steps to Reproduce

First, act as the "primary user."

• Set up a TFE free organization, get API keys going, set up your CLI config file, etc.
• Make a simple placeholder Terraform configuration with one null resource.
• Configure the remote backend to use multiple workspaces, using the prefix argument, as show above.
• Do whatever you need to do to get several workspaces (with state) into TFE. (There's multiple steps here, but they're not what we're interested in. Just make sure you have some workspaces like appserver-prod, appserver-dev, etc.)

Next, act as the "new collaborator/teammate." You have not used this Terraform configuration before, and are trying to get up and running for the first time. Ideally, you'll be able to just jump in and start using your team's shared state.

• Make sure you have the right credentials etc. set up for TFE. (It's fine to just keep using your primary user's credentials; we're focused on terraform init's behavior, not on permissions or anything.)
• Check out a fresh copy of the placeholder Terraform configuration. It should have the backend configuration in it, but should not have a .terraform directory or any state or anything. Pristine.
• Run terraform init.

Expected Behavior

Terraform sees that there's no prior state to reconcile, and sets up workspaces according to the configured prefix. You end up with several workspaces with short local names like prod, dev, etc., and can switch to one of your choosing and start editing the config and running plans. These workspaces use the remote workspaces' state according to the prefix mapping.

Since you didn't have prior state and hadn't previously selected a workspace, you don't have an expected initial workspace. Maybe it's the first alphabetical one, maybe it's random, who knows. (I'd expect init to tell me which workspace it picked, and I'd also expect that there's a low chance it's the one I wanted and I'll need to switch anyway.)

Actual Behavior

Terraform 0.11.13

╚ᐅ terraform init

Initializing the backend...

Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.
Error loading state: default state not supported
You can create a new workspace with the "workspace new" command.

I guess I need to select a workspace?

╚ᐅ terraform workspace list
  prod

It looks like init got far enough to give me a workspace list, but not far enough to download plugins or anything. OK!

╚ᐅ terraform workspace select prod
Switched to workspace "prod".

╚ᐅ terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "null" (2.1.2)...
- Downloading plugin for provider "random" (2.1.2)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.null: version = "~> 2.1"
* provider.random: version = "~> 2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

At this point things work, but that was kind of unpleasant.

Terraform 0.12 beta 2

╚ᐅ terraform012 init

Initializing the backend...

Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.

Error: No workspaces are migrated.

Use the "terraform workspace" command to create and select a new workspace.
If the backend already contains existing workspaces, you may need to update
the backend configuration.

Guess I need to list workspaces and choose one.

╚ᐅ terraform workspace list
Backend reinitialization required. Please run "terraform init".
Reason: Backend configuration changed for "remote"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

Failed to load backend: Initialization required. Please see the error message above.

Oh..... that won't even work. So, now what? I guess I comment out the backend, run terraform init, uncomment the backend, init again, choose a bogus name for the default workspace, and then delete the unwanted default workspace after everything's up and running?

Additional Context

• The onboarding process for new team members is fine if you're using a single workspace with the name argument. It's just the multi-workspace setup that's hosed.
• This stems from the special default unnamed workspace being forbidden with prefixed workspaces. (This isn't a workspace named default, I think, it's the way the workspace code chooses to display the unnamed state file that isn't in the state.d directory.) Except the new teammate doesn't actually have that workspace; they don't have anything.
• If there IS state in the default workspace, init asks you to give it a new name while it's setting up the remote backend.

[nfagerlund]

@mildwonkey, I think you're taking over the maintenance of the backend, right?

@tr0njavolta, just pinging to make sure you get future notifications on this issue while I'm out.

[apparentlymart]

Given how this was resolved in the 0.11 case (by selecting a workspace and then re-running terraform init), it may be possible to work around this by putting a valid workspace name in the file .terraform/environment (this file's name is a holdover from the original naming of the workspaces feature) before running terraform init. Not that this is a solution by any means, but hopefully allows progress in the short term until we figure out how best to address this.

The solution to this may also interact with #21133, which would include fetching all of the workspace names from the backend as part of terraform init, which could give us enough information for a better UX here.

[nfagerlund]

I made a typo that blew everything up, but it's better now

HEY, that totally works, on both 0.11.13 and 0.12 beta! You just have to make sure the workspace name you choose actually exists (and that you don't have a dumb typo in your backend config after testing twelve different things).

mkdir .terraform
echo "prod" > .terraform/environment 
terraform init

@apparentlymart Good guess!

[svanharmelen]

There is actually already logic for this use case, but that logic was only called during a specific migration scenario. I reshuffled the logic a bit so its now always called after creating a backend from config to make sure it will always perform this check.

This results in the following UX, which is exactly what we want (AFAIK):

I'll make a PR for the changes in a bit and will link them to this issue.

[svanharmelen]

Those ^^ two PR fix this issue for both v0.11.x and v0.12.x.

[nfagerlund]

sweet, thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.