[Bug]: InjectedIdentity uses wrong projectID when using cross-project setup

[Stazis555]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

gcp.upbound.io/v1beta1 - ProviderConfig
sql.gcp.upbound.io/v1beta2 - DatabaseInstance

Resource MRs required to reproduce the bug

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-sql
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-sql:v1.11.3
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1
    kind: DeploymentRuntimeConfig
    name: wi-drc

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: wi-drc
spec:
  serviceAccountTemplate:
    metadata:
      name:  gcp-provider-sa # This used to create iam-policy-binding on org level
        

apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: wi-pc
spec:
  credentials:
    source: InjectedIdentity # Same behaviour when setting None
  projectID: development

apiVersion: sql.gcp.upbound.io/v1beta1
kind: DatabaseInstance
metadata:
  name: test-db
spec:
  forProvider:
    databaseVersion: MYSQL_8_0
    deletionProtection: false
    region: europe-west1
    project: development # Obfuscating this and using only in ProviderConfig leads to the same result
    settings:
      - diskSize: 10
        tier: db-f1-micro
  providerConfigRef:
    name: wi-pc

Steps to Reproduce

Have 2 projects in the same organization (or have explicit SA binding from management project):

1. Management
2. Development

• Deploy Crossplane in Management
• Create the provider as mentioned above.
• Enable WorkloadIdentity on Management Cluster.
• Create IAM Policy binding for k8s SA mentioned in the DeploymentRuntimeConfig (following this example in Authenticate to Google Cloud APIs from GKE workloads). P.S. In our example we use not project level assignment, but organizarion level assignment, and Development project inherits k8s SA binding with needed role.

What happened?

Desired:

Provider respects the project mentioned in the ProviderConfig (or in DatabaseInstance) and creates resource in Development project in GCP.

Actual:

Provider ignores projectId field in ProviderConfig (and in DatabaseInstance as well) and tries to create it in Management Project (SQL API is not enabled there)

Relevant Error Output Snippet

observe failed: failed to observe the resource: [{0 Error when reading or editing SQL Database Instance "test-db": googleapi: Error 403: Cloud SQL Admin API has not been used in project <Management> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<Management>then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.ErrorInfo",
          "domain": "googleapis.com",
          "metadata": {
            "activationUrl": "https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<>",
            "consumer": "projects/<Management>",
            "containerInfo": "<Management>",
            "service": "sqladmin.googleapis.com",
            "serviceTitle": "Cloud SQL Admin API"
          },
          "reason": "SERVICE_DISABLED"
        },
        {
          "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
          "locale": "en-US",
          "message": "Cloud SQL Admin API has not been used in project <Management> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<Management> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."
        },
        {
          "@type": "type.googleapis.com/google.rpc.Help",
          "links": [
            {
              "description": "Google developers console API activation",
              "url": "https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<Management>"
            }
          ]
        }
      ]
      , accessNotConfigured  []}]

Crossplane Version

v1.17.0

Provider Version

v1.11.3

Kubernetes Version

Server Version: v1.30.8-gke.1261000

Kubernetes Distribution

GKE

Additional Info

Hello, We are experiencing the issue when trying to allow k8s Service Account to create resources in another project in the same organization when using InjectedIdentity source for ProviderConfig.

Our cloud topology:

• Management project where Crossplane is runnning
• Development - managed by Crossplane

We are trying to migrate to the WorkloadIdentity authentication, to get rid of Json Key for GCP SA. Allowing k8s SA to create resources in GCP via InjectedIdentity (or None) in ProviderConfig source is not working correctly when you specify the project, it ignores it completely.

Note:

Provider is working correctly when ProviderConfig source is set to ImpersonateServiceAccount and GCP SA is located inside Development Project with binding gcp-provider-sa k8s SA to Development Project with ServiceAccountTokenCreator role (allowing it to impersonate that GCP SA).

spec:
  credentials:
    impersonateServiceAccount:
      name: crossplane-provider@development.iam.gserviceaccount.com
    source: ImpersonateServiceAccount
  projectID: development

Since here ProviderConfig projectID is used, I assume that something weird is happening on upjet side when trying to authenticate automatically.

[alanrichman]

@Stazis555 A Workload Identity SA binding calls GCP APIs in the account the Identity Pool is provisioned, which means that the SA in the Management cluster in the Management project will call APIs in the Management project. The spec.projectID field maps to the project field of the GCP Terraform provider. This field does not control which project API calls are made in, it is a default location for resources to be provisioned in.

If your use case's requirements mean that you need to call APIs in the Development project rather than the Management project, the SA impersonation you posted about is the correct pathway. The GCP IAM API calls to perform that impersonation will take place in the Management project, then then the provisioning API calls will take place in the Development project.

If you do not care which project API calls are made in and only care where the resources end up, enable the sqladmin.googleapis.com API in the Management project. Since you have the project ID for provisioning set on the ProviderConfig and the DatabaseInstance to the Development project the resource will be provisioned in the Development project even though the API call will be made in the Management project.

[Stazis555]

@alanrichman Thank you, I will double check our implementation, because management API use hinted me that in will create SQL instance in that project. But apparently I was wrong.