BUG: Deployment of switch color cron terraform failed

[mmarcotte]

Describe the Bug

This past weekend, we attempted to deploy to production, and it failed on the deploy-switch-colors-cron job.

We observed a few permissions errors at the top of the output of the Deploy Switch Colors Cron step:

Initiating Terraform state bucket creation for [arn:aws:iam:::user/CircleCI], bucket [.terraform.deploys], key [switch-colors-cron-prod.tfstate] in region [us-east-1]
make_bucket failed: s3://************.terraform.deploys An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Access Denied

An error occurred (AccessDenied) when calling the PutBucketVersioning operation: Access Denied
checking for the dynamodb lock table.

And then critically when it wen to refresh the state on the file, it also got an Access Denied 403 error:

Error refreshing state: AccessDenied: Access Denied
status code: 403, request id: SZB7QHWQWXTXYS9T, host id: UMF4okdn9JQ6EVmVW12J6Q0YejBYUBsYpBzhCyzFr2aBrWochWrRs15u/bQZm6Sj+9D51D/RY2U=

Business Impact/Reason for Severity

Unable to perform automated deployment

In which environment did you see this bug?

Prod

Who were you logged in as?

N/A

What were you doing when you discovered this bug? (Using the application, demoing, smoke tests, testing other functionality, etc.)

Deploying

To Reproduce

1. Deploy
2. See it fail on the switch color cron terraform job

Expected Behavior

Resources should be deployed per the terraform config

Actual Behavior

CircleCI job failed, state file not created, deployment is cancelled.

Cause of Bug, If Known

These steps:

      - deploy-switch-colors-cron:
          requires:
            - smoketests-readonly
      - wait-for-switch:
          type: approval
          requires:
            - deploy-switch-colors-cron
      - disable-switch-colors-cron:
          requires:
            - wait-for-switch

require this line added:

          <<: *build-and-deploy-with-context-defaults

Process for Logging a Bug:

• Complete the above information
• Add a severity tag (Critical, High Severity, Medium Severity or Low Severity). See below for priority definition.

Severity Definition:

• Critical Defect
  Blocks entire system's or module’s functionality
  No workarounds available
  Testing cannot proceed further without bug being fixed.

• High-severity Defect
  Affects key functionality of an application
  There's a workaround, but not obvious or easy
  App behaves in a way that is strongly different from the one stated in the requirements

• Medium-severity Defect
  A minor function does not behave in a way stated in the requirements.
  Workaround is available and easy

• Low-severity Defect
  Mostly related to an application’s UI
  Doesn't need a workaround, because it doesn't impact functionality

Definition of Ready for Bugs(Created 10-4-21)

Definition used: A failure or flaw in the system which produces an incorrect or undesired result that deviates from the expected result or behavior. (Note: Expected results are use cases that have been documented in past user stories as acceptance criteria and test cases, and do not include strange behavior unrelated to use cases.)

The following criteria must be met in order for the development team to begin work on the bug.

The bug must:

• Be focused on solving a user problem
• Contain data for all fields in the bug template, so the team can pick it up and begin working immediately

Process: If the unexpected results are new use cases that have been identified, but not yet built, new acceptance criteria and test cases should be captured in a new user story and prioritized by the product owner.

If the Court is not able to reproduce the bug, add the “Unable to reproduce” tag. This will provide visibility into the type of support that may be needed by the Court. In the event that the Court cannot reproduce the bug, the Court will work with Flexion to communicate what type of troubleshooting help may be needed.

Definition of Done (Updated 4-14-21)

Product Owner

• Bug fix has been validated in the Court's test environment

Engineering

• Automated test scripts have been written
• Field level and page level validation errors (front-end and server-side) integrated and functioning
• Verify that language for docket record for internal users and external users is identical
• New screens have been added to pa11y scripts
• All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html
• READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
• UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
• Interactors should validate entities before calling persistence methods
• Code refactored for clarity and to remove any known technical debt
• Deployed to the Court's test environment if prod-like data is required. Otherwise, deployed to any experimental environment for review.