PSS: Implement initialisation of new working directory (or use of `-reconfigure` flag) while using `state_store` #37732

SarahFrench · Oct 7, 2025

Description

This PR allows an init command to successfully initialise a project that uses the new state_store block:

when the working directory doesn't have a backend state file yet/is uninitialised.
when the working directory does have a backend state file but it's ignored due to using the -reconfigure flag.

Behaviour changes versus the equivalent logic for backends

1) Problem: Creation of workspaces is not explicit and happens as a side effect of other code

Expand to read details about current behaviour with `backend`

Currently during an init command when using a backend block the state file for a new workspace is first created as a side effect of obtaining a state manager for that workspace's state. What this looks like depends on whether it's a custom workspace or the "default" workspace.

During an init the default default workspace's state file may be created, depending on the backend in use. If it's not made during init it'll be created later when the first apply persists state. When using the local backend the default workspace file is only made at apply time. When using the gcs backend this code responds to there being no state present for a given workspace by creating that state file, including for the default workspace. Other backends like s3 do this but only for custom workspaces, and the default workspace's state file is only made at apply time.

The code linked above is also used when new custom workspaces are created when a user runs a workspace new commands. This code creates the new workspace as a side effect of obtaining a state manager.

Something worth calling out: remote-state backends create state files for non-default workspaces when obtaining a state manager via the StateMgr method to ensure that the new workspace is listed by the Workspaces method later in the same operation. Also, all remote-state backends' Workspaces methods are hardcoded to report that the "default" workspace exits, regardless of whether there is a default workspace state file.

Overall this shows that Terraform core relies on remote-state backends being implemented in a consistent way for Terraform operations to work as expected. This is a risk, as we cannot rely on all pluggable state storage implementations being implemented in a consistent way to match how remote-state backends behave today.

Solution: Make creation of workspaces (i.e. persisting empty state in a way that signifies a new workspace) an explicit responsibility of Terraform core code

Terraform will now explicitly create custom workspaces by writing an empty state to a new file via the state store.

The default workspace will need special handling, as it's not consciously created by users like custom workspaces are. Instead the init command will need to detect if the default workspace is selected but doesn't currently exist, and then persist an empty state file for the default workspace. This is achieved here:

terraform/internal/command/meta_backend.go

Lines 1784 to 1794 in 07e4d23

    
           if ws == backend.DefaultStateName { 
        
           	// Users control if the default workspace is created through the -create-default-workspace flag (defaults to true) 
        
           	if opts.CreateDefaultWorkspace { 
        
           		diags = diags.Append(m.createDefaultWorkspace(c, b)) 
        
           	} else { 
        
           		diags = diags.Append(&hcl.Diagnostic{ 
        
           			Severity: hcl.DiagWarning, 
        
           			Summary:  "The default workspace does not exist", 
        
           			Detail:   "Terraform has been configured to skip creation of the default workspace in the state store. To create it, either remove the `-create-default-workspace=false` flag and re-run the 'init' command, or create it using a 'workspace new' command", 
        
           		}) 
        
           	}

Creating the default workspace will happen by default and will not ask the user for confirmation, which matches the UX with backends. If a user wants to skip this behaviour in CI they can set -create-default-workspace=false and -input=false. The -create-default-workspace flag default to true, and it cannot be set to false if Terraform is not in CI (i.e. -input=true`).

2) Before: Users prompted for input if required fields aren't in the configuration (or present in override data)

If a user doesn't supply a value for a required field in a backend they're prompted for a value through the terminal:

terraform/internal/command/meta_backend.go

Lines 1812 to 1818 in d573104

    
           // TODO: test 
        
           if m.Input() { 
        
           	var err error 
        
           	configVal, err = m.inputForSchema(configVal, schema) 
        
           	if err != nil { 
        
           		diags = diags.Append(fmt.Errorf("Error asking for input to configure backend %q: %s", c.Type, err)) 
        
           	}

After: Terraform will return errors to users and prompt them to supply a value for the required field via other methods.

We've purposefully avoided prompting users for missing values. Instead, users should supply values through configuration, overrides, or ENVs. If users require the ability to supply values through input during another operation we should wait for feature requests to learn more about the reasons/use case.

Target Release

N/A

Rollback Plan

If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

This change is user-facing and I added a changelog entry.
This change is not user-facing.

github-actions · Oct 7, 2025

The equivalence tests failed. Please investigate here.

SarahFrench

Leaving some thoughts of my own & highlighting some things, as I appreciate this is a large PR to parse. There are a bunch of changes but hopefully it's manageable once the test fixture changes are hidden and once it's clear that all the new code is feeding into or is called by the new stateStore_C_s method.

internal/command/meta_providers.go

internal/command/meta_backend.go

SarahFrench · Oct 8, 2025

I just force pushed to remove some commits that included changes to try debug the test failure on the PR, no code changes were added other than the new commit 4c2528d

Ideally I'd like to get that removed test code working, but I'm unsure why the testStdinPipe helper fails in this test but other tests pass ok 🤔

internal/command/command_test.go

internal/command/meta_backend.go

internal/command/meta_backend_test.go

internal/command/init_test.go

internal/command/arguments/init.go

SarahFrench · Oct 13, 2025

Force push was to rebase onto #37762 and change any new errors in this PR into functions in meta_backend_errors.go

This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.

…hat data isn't empty

…kspaces

…ng code to access the provider schema when using provider configuration data.

…version Version. This is needed for using locks when creating the backend state file.

…eStore_C_s`. Default to creating the default workspace if no workspaces exist.

…n PSS tests

The `testingOverrides` field will only be set in tests, so this should not impact end users.

… time (and do the same when forced by -reconfigure flag). Remove replaced tests.

…e default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests.

…n't exist, but other workspaces do exist. The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`.

This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input.

…t is enabled or disabled

…ils of errors copying data to stdin. Note: We cannot call t.Fatal from a non-test goroutine.

…or reattached provider is in use.

…atching entry in required_providers

internal/command/meta_backend.go

…rkspaces exist.

radeksimko

I have yet to give this a full (manual) test run but the code generally LGTM aside from that one concern regarding the new flag.

internal/command/arguments/init.go

…nit func.

… enabled, enforce coupling of PSS-related flags when in use.

SarahFrench added the no-changelog-needed Add this to your PR if the change does not require a changelog entry label Oct 7, 2025

SarahFrench commented Oct 7, 2025

View reviewed changes

internal/command/meta_providers.go Show resolved Hide resolved

internal/command/meta_backend.go Show resolved Hide resolved

internal/command/meta_backend.go Outdated Show resolved Hide resolved

SarahFrench force-pushed the pss/init-empty-workdir branch from a41b9da to 4c2528d Compare October 8, 2025 11:03

SarahFrench marked this pull request as ready for review October 8, 2025 11:07

SarahFrench requested a review from a team as a code owner October 8, 2025 11:07

radeksimko reviewed Oct 8, 2025

View reviewed changes

SarahFrench commented Oct 9, 2025

View reviewed changes

internal/command/arguments/init.go Show resolved Hide resolved

SarahFrench commented Oct 9, 2025

View reviewed changes

internal/command/arguments/init.go Outdated Show resolved Hide resolved

SarahFrench mentioned this pull request Oct 10, 2025

Refactor backend error strings as custom errors or pre-made diagnostics #37762

Merged

3 tasks

SarahFrench force-pushed the pss/init-empty-workdir branch from 6db7a4b to 5bf78bf Compare October 13, 2025 11:57

SarahFrench added 17 commits October 13, 2025 16:07

Minor fixes in diagnostics

a828718

This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.

Rename test to make it specific to use of backend block in config

7cc8316

Update initBackend to accept whole initArgs collection

2e7b18d

Only process --backend-config data, when setting up a backend, if t…

9adf497

…hat data isn't empty

Simplify how mock provider factories are made in tests

c13e2f1

Update mock provider's default logic to track and manage existing wor…

4f59cb1

…kspaces

Add ProviderSchema method to Pluggable structs. This allows calli…

c287a97

…ng code to access the provider schema when using provider configuration data.

Add function for converting a providerreqs.Version to a hashicorp/go-…

99d6229

…version Version. This is needed for using locks when creating the backend state file.

Implement initial version of init new working directories using `stat…

5d42a91

…eStore_C_s`. Default to creating the default workspace if no workspaces exist.

Update test fixtures to match the hashicorp/test mock provider used i…

27a73a5

…n PSS tests

Allow tests to obtain locks that include testingOverrides providers.

58cae04

The `testingOverrides` field will only be set in tests, so this should not impact end users.

Add tests showing TF can initialize a working directory for the first…

e27abd4

… time (and do the same when forced by -reconfigure flag). Remove replaced tests.

Add -create-default-workspace flag, to be used to disable creating th…

43ac7c4

…e default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests.

Allow reattached providers to be used during init for PSS

d6c3b4e

Rename variable to backendHash so relation to backend is clearer

9e649b9

Allow (m *Meta) Backend to return warning diagnostics

3e5fb58

Protect against nil testingOverrides in providerFactoriesFromLocks

d3d6ff8

SarahFrench added 10 commits October 13, 2025 16:07

Add test case seeing what happens if default workspace selected, does…

7f07422

…n't exist, but other workspaces do exist. The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`.

Address code consistency check failure on PR

2205034

Refactor use of mock in test that's experiencing EOF error...

14f7468

Allow -create-default-workspace to be used regardless of whether inpu…

94b3586

…t is enabled or disabled

Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable

6d235aa

Responses to feedback, including making testStdinPipe helper log deta…

93bfc82

…ils of errors copying data to stdin. Note: We cannot call t.Fatal from a non-test goroutine.

Use Errorf instead

ce8c27c

Allow backend state files to not include version data when a builtin …

4a28cf9

…or reattached provider is in use.

Add clarifying comment about re-attached providers when finding the m…

e79ea11

…atching entry in required_providers

SarahFrench force-pushed the pss/init-empty-workdir branch from 82372c1 to e79ea11 Compare October 13, 2025 15:07

Report that the default workspace was created to the view

8d4279e

SarahFrench commented Oct 13, 2025

View reviewed changes

internal/command/meta_backend.go Show resolved Hide resolved

SarahFrench requested a review from radeksimko October 13, 2025 16:49

Refactor: use error comparison via errors.Is to identify when no wo…

c32a903

…rkspaces exist.

radeksimko reviewed Oct 14, 2025

View reviewed changes

internal/command/arguments/init.go Show resolved Hide resolved

SarahFrench mentioned this pull request Oct 14, 2025

[WIP] PSS: Add logic for initialising a new working directory with PSS. #37646

Closed

SarahFrench added 4 commits October 14, 2025 17:07

Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseI…

d89b964

…nit func.

Validate that PSS-related flags can only be used when experiments are…

3d38b07

… enabled, enforce coupling of PSS-related flags when in use.

Slight rewording of output message about default workspace

8ee5c73

Update test to assert new output about default workspace

9c7012b

SarahFrench requested a review from radeksimko October 14, 2025 16:12

radeksimko approved these changes Oct 15, 2025

View reviewed changes

SarahFrench merged commit 6b73f71 into main Oct 15, 2025
7 checks passed

SarahFrench deleted the pss/init-empty-workdir branch October 15, 2025 09:44

SarahFrench mentioned this pull request Oct 15, 2025

Update the init command's help text to describe PSS-related flags #37773

Merged

3 tasks

	if ws == backend.DefaultStateName {
	// Users control if the default workspace is created through the -create-default-workspace flag (defaults to true)
	if opts.CreateDefaultWorkspace {
	diags = diags.Append(m.createDefaultWorkspace(c, b))
	} else {
	diags = diags.Append(&hcl.Diagnostic{
	Severity: hcl.DiagWarning,
	Summary: "The default workspace does not exist",
	Detail: "Terraform has been configured to skip creation of the default workspace in the state store. To create it, either remove the `-create-default-workspace=false` flag and re-run the 'init' command, or create it using a 'workspace new' command",
	})
	}

	// TODO: test
	if m.Input() {
	var err error
	configVal, err = m.inputForSchema(configVal, schema)
	if err != nil {
	diags = diags.Append(fmt.Errorf("Error asking for input to configure backend %q: %s", c.Type, err))
	}

Search code, repositories, users, issues, pull requests...

PSS: Implement initialisation of new working directory (or use of -reconfigure flag) while using state_store #37732

PSS: Implement initialisation of new working directory (or use of -reconfigure flag) while using state_store #37732

Conversation

SarahFrench commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Behaviour changes versus the equivalent logic for backends

1) Problem: Creation of workspaces is not explicit and happens as a side effect of other code

Solution: Make creation of workspaces (i.e. persisting empty state in a way that signifies a new workspace) an explicit responsibility of Terraform core code

2) Before: Users prompted for input if required fields aren't in the configuration (or present in override data)

After: Terraform will return errors to users and prompt them to supply a value for the required field via other methods.

Target Release

Rollback Plan

Changes to Security Controls

CHANGELOG entry

Uh oh!

github-actions bot commented Oct 7, 2025

Uh oh!

SarahFrench left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarahFrench commented Oct 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarahFrench commented Oct 13, 2025

Uh oh!

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PSS: Implement initialisation of new working directory (or use of `-reconfigure` flag) while using `state_store` #37732

PSS: Implement initialisation of new working directory (or use of `-reconfigure` flag) while using `state_store` #37732

SarahFrench commented Oct 7, 2025 •

edited

Loading