RFE-2192: GCP Service Account Secrets in the Vault (PASM)

This RFE investigates adding GCP Service Account credentials as a managed secret type in PASM.

The goal is for the Vault to securely store a GCP service account key and return a short-lived Google access token during checkout. The private key is never returned to the user.

Two approaches were implemented and tested:

  1. Approach 1: Token-only
  2. Approach 2: AWS-style key rotation and short-lived tokens

1. Overview

Item

Details

RFE

RFE-2192

Secret type

GCP Service Account credentials

Secret type category

Managed secret

Checkout result

Short-lived GCP access token

Token lifetime

Approximately 1 hour

Approach 1

Vault stores the key and only generates tokens

Approach 2

Vault manages the full GCP key lifecycle and generates tokens

New dependency

None. google-auth is already available on the appliance


2. Approach Comparison

Topic

Approach 1: Token-only

Approach 2: AWS-style

Key rotation

Not supported

Supported

Vault creates GCP keys

No

Yes

Vault deletes GCP keys

No

Yes

Additional GCP role

None

Service Account Key Admin

Supplied key

Retained

Deleted after Vault creates its own key

Create time

Usually seconds

Can take up to ~90 seconds

Rotation

Not supported

Manual, duration, or box-level

Lease

Not supported

Not supported

Org policy blocking key creation

No impact

Blocks this approach

Leaked key

Valid until manually deleted

Invalid after successful rotation

Unit tests

24 passing

59 passing

End-to-end

Passed

17/17 passed

Recommendation

Use Approach 1 when:

  • Vault should not manage GCP keys.
  • Additional IAM permissions cannot be granted.
  • Manual key replacement is acceptable.
  • A simpler implementation is preferred.

Use Approach 2 when:

  • Automatic key rotation is required.
  • Vault should manage the complete key lifecycle.
  • Long-lived unchanged service account keys should be avoided.

3. Common Behaviour

Both approaches have the following behaviour:

  • The Vault stores the GCP service account private key.
  • Checkout generates a new access token.
  • Checkout returns:
    • token
    • expires_on
    • scope
  • The private key is never returned.
  • Google controls the access token lifetime, which is approximately 1 hour.
  • lease is not supported.
  • Credentials are validated against Google during create and value update.
  • The appliance requires outbound HTTPS access to oauth2.googleapis.com.
  • Approach 2 additionally requires access to iam.googleapis.com.
  • Checkout requires the Vault User role policy on the box.
  • Already-issued tokens remain valid even if the underlying service account key is subsequently deleted.

Secret configuration

 
{
  "type": "GCP Service Account credentials"
}


Secret data

Required:

 
gcp_service_account_key
 


Optional:

 
gcp_checkout_scope


Default scope:

 
https://www.googleapis.com/auth/cloud-platform



4. Approach 1: Token-only

4.1 How it works

  1. Admin creates a managed secret and provides the GCP service account key.
  2. Vault validates the key by requesting a token from Google.
  3. Vault stores the key.
  4. User calls CheckoutSecret.
  5. Vault uses the stored key to generate a new access token.
  6. Vault returns the token, expiry, and scope.
  7. The application uses the token until it expires and checks out another token when needed.

The Vault does not create, rotate, replace, or delete GCP service account keys.

4.2 Rotation and Lease

Rotation is not supported.

The following operations return HTTP 400:

  • Lease on create/update
  • Rotation configuration
  • Manual RotateSecret

The Vault daemon also skips this secret type during automatic rotation.

4.3 Replacing the Key

Key replacement is manual:

  1. Create a new key in Google Cloud.
  2. Update gcp_service_account_key in the Vault.
  3. Vault validates the new key.
  4. Delete or disable the old key in Google Cloud.

4.4 Testing

Unit Testing

24 unit tests passed.

Tests cover:

  • Raw JSON key
  • Base64-encoded key
  • Invalid JSON
  • Incorrect key type
  • Missing fields
  • Token generation
  • Scope handling
  • UTC expiry
  • Invalid private key
  • Google authentication errors
  • Network failures
  • Create
  • Validate
  • Checkout
  • Ensuring the private key is never returned

Appliance Testing

Test

Result

Create secret

HTTP 200

Checkout

HTTP 200

Token accepted by Google

Passed

Lease

HTTP 400

Rotation configuration

HTTP 400

Invalid key

HTTP 400

RotateSecret

HTTP 400

Delete

HTTP 200

Not tested

  • Multi-node cluster
  • HSM-enabled box
  • UI behaviour

4.5 Issues Found

Issue

Cause

Fix

RotateSecret could return HTTP 500

Rotation refusal was raised too deep in the flow

Added an up-front check returning HTTP 400

Daemon repeatedly attempted rotation

Managed secrets with box-level rotation were automatically considered

Skip GCP token-only secrets

File-based GCP key failed parsing

PASM File secret content is base64 encoded

Support raw and base64 JSON

Admin could not checkout

Vault User role is required

Add policy and log in again


5. Approach 2: AWS-style Key Rotation

Approach 2 extends the token functionality by allowing the Vault to manage the complete GCP service account key lifecycle.

5.1 How it works

Create

  1. Validate the supplied service account key.
  2. Create a new GCP service account key.
  3. Wait until Google fully accepts the new key.
  4. Store the new key in the Vault.
  5. Delete the original supplied key.

Checkout

  1. Read the Vault-owned key.
  2. Generate a short-lived access token.
  3. Return the token, expiry, and scope.

Rotation

  1. Read the existing key.
  2. Create a new key in Google.
  3. Verify that the new key works.
  4. Save the new key in the Vault.
  5. Delete the previous key.

This ordering ensures that the old working key is not deleted before the new key is safely stored.


5.2 Google Key Propagation Issue

One important issue was found during implementation.

A newly created GCP service account key is not necessarily accepted by all Google services at the same time.

For example:

  1. New key is created.
  2. Token generation succeeds.
  3. IAM request using the same key still fails.
  4. Google returns:

 
invalid_grant: Invalid JWT Signature


Because of this, checking only token generation was not sufficient.

Fix

A new key is now considered ready only when:

  • token generation succeeds, and
  • IAM key read succeeds.

The retry delays are:

 
2s
4s
8s
15s
15s
15s
15s
15s


Maximum wait is approximately 89 seconds.


5.3 Required GCP Permission

Approach 2 requires:

 
roles/iam.serviceAccountKeyAdmin


The role must be granted on the service account itself.

Example:

 
gcloud iam service-accounts add-iam-policy-binding <SERVICE_ACCOUNT_EMAIL> </span>
  --project=<PROJECT_ID> </span>
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" </span>
  --role="roles/iam.serviceAccountKeyAdmin"


Without this permission, Vault cannot create or delete service account keys.


5.4 Rotation

Approach 2 supports:

  • Manual rotation
  • Secret-level duration rotation
  • Box-level rotation

Minimum rotation duration:

 
30 minutes


Example:

 
"rotation": {
  "duration": "PT30M"
}


Lease is still not supported.


5.5 Scheduled Rotation Test

Automatic rotation was tested using:

 
PT30M


Result:

  • Daemon automatically rotated the secret at approximately 31.3 minutes
  • Previous key was removed
  • New key became active
  • Checkout worked after rotation
  • New access token was accepted by Google
  • Cleanup completed successfully

Result: PASSED


5.6 End-to-End Testing

17 of 17 checks passed.

The flow validated:

  • Login
  • Initial key exists in Google
  • Secret creation
  • Creation of Vault-managed key
  • Removal of supplied key
  • Checkout
  • Token validity
  • Manual rotation
  • New key activation
  • Removal of previous Vault key
  • Checkout after rotation
  • Lease rejection
  • Invalid-key rejection
  • Secret deletion
  • GCP key cleanup

5.7 Issues Found and Fixes

#

Issue

Cause

Fix

1

Key creation denied

Service account had no permission on itself

Granted Service Account Key Admin on itself

2

Invalid JWT Signature

New key was not propagated to all Google services

Added retry logic

3

New key remained after failure

Cleanup used the new key before it was fully working

Cleanup now uses old key first

4

Token check passed but IAM request failed

Token and IAM services accept new keys at different times

New key must pass both token and IAM checks


6. Key Findings

Google Cloud

  • Service accounts do not automatically have Key Admin permission on themselves.
  • A newly created key may not become usable across all Google services at the same time.
  • Token service may accept a key before IAM accepts it.
  • Deleting a key does not revoke previously generated access tokens.
  • A service account can have a maximum of 10 keys.

Vault

  • Cleanup order is important.
  • The existing working key should be preserved until the new key is verified and stored.
  • A failure during key creation should leave the original key usable.
  • A failure while saving a rotated key removes the newly created Google key.
  • Failure to delete the previous key after successful rotation is logged but does not fail the rotation.
  • Create and rotation can take roughly 90 seconds in the worst case.

7. Security Considerations

Both Approaches

  • Access tokens inherit the service account permissions allowed through their OAuth scope.
  • Apply least-privilege permissions to the service account.
  • Use a narrower gcp_checkout_scope where possible.
  • Service account key files should not remain on shared machines.
  • Remove local copies after they are added to PASM.

Approach 1

A leaked private key remains usable until somebody manually removes it from Google Cloud.

Approach 2

  • Always provide a dedicated key during create because Vault deletes the supplied key.
  • Grant Key Admin only on the required service account.
  • Keep capacity below Google's 10-key limit so rotation has room to create the replacement key.
  • iam.disableServiceAccountKeyCreation prevents this approach from working.
  • If deletion of an old key fails, manual cleanup may be required.

8. Final Recommendation

For the final PASM implementation, Approach 2 is the stronger option when automated credential lifecycle management is required.

It provides:

  • Short-lived checkout tokens
  • Automatic GCP key rotation
  • Manual and scheduled rotation
  • Key cleanup
  • Recovery/rollback handling
  • No unchanged long-lived key sitting indefinitely