RFE-2192: GCP Service Account Secrets in the Vault (PASM)
This RFE investigates adding GCP Service Account credentials as a managed secret type in PASM.
The goal is for the Vault to securely store a GCP service account key and return a short-lived Google access token during checkout. The private key is never returned to the user.
Two approaches were implemented and tested:
- Approach 1: Token-only
- Approach 2: AWS-style key rotation and short-lived tokens
1. Overview
Item | Details |
|---|---|
RFE | RFE-2192 |
Secret type |
|
Secret type category | Managed secret |
Checkout result | Short-lived GCP access token |
Token lifetime | Approximately 1 hour |
Approach 1 | Vault stores the key and only generates tokens |
Approach 2 | Vault manages the full GCP key lifecycle and generates tokens |
New dependency | None. |
2. Approach Comparison
Topic | Approach 1: Token-only | Approach 2: AWS-style |
|---|---|---|
Key rotation | Not supported | Supported |
Vault creates GCP keys | No | Yes |
Vault deletes GCP keys | No | Yes |
Additional GCP role | None | Service Account Key Admin |
Supplied key | Retained | Deleted after Vault creates its own key |
Create time | Usually seconds | Can take up to ~90 seconds |
Rotation | Not supported | Manual, duration, or box-level |
Lease | Not supported | Not supported |
Org policy blocking key creation | No impact | Blocks this approach |
Leaked key | Valid until manually deleted | Invalid after successful rotation |
Unit tests | 24 passing | 59 passing |
End-to-end | Passed | 17/17 passed |
Recommendation
Use Approach 1 when:
- Vault should not manage GCP keys.
- Additional IAM permissions cannot be granted.
- Manual key replacement is acceptable.
- A simpler implementation is preferred.
Use Approach 2 when:
- Automatic key rotation is required.
- Vault should manage the complete key lifecycle.
- Long-lived unchanged service account keys should be avoided.
3. Common Behaviour
Both approaches have the following behaviour:
- The Vault stores the GCP service account private key.
- Checkout generates a new access token.
- Checkout returns:
tokenexpires_onscope
- The private key is never returned.
- Google controls the access token lifetime, which is approximately 1 hour.
leaseis not supported.- Credentials are validated against Google during create and value update.
- The appliance requires outbound HTTPS access to
oauth2.googleapis.com. - Approach 2 additionally requires access to
iam.googleapis.com. - Checkout requires the Vault User role policy on the box.
- Already-issued tokens remain valid even if the underlying service account key is subsequently deleted.
Secret configuration
{
"type": "GCP Service Account credentials"
}
Secret data
Required:
gcp_service_account_key
Optional:
gcp_checkout_scope
Default scope:
https://www.googleapis.com/auth/cloud-platform
4. Approach 1: Token-only
4.1 How it works
- Admin creates a managed secret and provides the GCP service account key.
- Vault validates the key by requesting a token from Google.
- Vault stores the key.
- User calls
CheckoutSecret. - Vault uses the stored key to generate a new access token.
- Vault returns the token, expiry, and scope.
- The application uses the token until it expires and checks out another token when needed.
The Vault does not create, rotate, replace, or delete GCP service account keys.
4.2 Rotation and Lease
Rotation is not supported.
The following operations return HTTP 400:
- Lease on create/update
- Rotation configuration
- Manual
RotateSecret
The Vault daemon also skips this secret type during automatic rotation.
4.3 Replacing the Key
Key replacement is manual:
- Create a new key in Google Cloud.
- Update
gcp_service_account_keyin the Vault. - Vault validates the new key.
- Delete or disable the old key in Google Cloud.
4.4 Testing
Unit Testing
24 unit tests passed.
Tests cover:
- Raw JSON key
- Base64-encoded key
- Invalid JSON
- Incorrect key type
- Missing fields
- Token generation
- Scope handling
- UTC expiry
- Invalid private key
- Google authentication errors
- Network failures
- Create
- Validate
- Checkout
- Ensuring the private key is never returned
Appliance Testing
Test | Result |
|---|---|
Create secret | HTTP 200 |
Checkout | HTTP 200 |
Token accepted by Google | Passed |
Lease | HTTP 400 |
Rotation configuration | HTTP 400 |
Invalid key | HTTP 400 |
RotateSecret | HTTP 400 |
Delete | HTTP 200 |
Not tested
- Multi-node cluster
- HSM-enabled box
- UI behaviour
4.5 Issues Found
Issue | Cause | Fix |
|---|---|---|
| Rotation refusal was raised too deep in the flow | Added an up-front check returning HTTP 400 |
Daemon repeatedly attempted rotation | Managed secrets with box-level rotation were automatically considered | Skip GCP token-only secrets |
File-based GCP key failed parsing | PASM File secret content is base64 encoded | Support raw and base64 JSON |
Admin could not checkout | Vault User role is required | Add policy and log in again |
5. Approach 2: AWS-style Key Rotation
Approach 2 extends the token functionality by allowing the Vault to manage the complete GCP service account key lifecycle.
5.1 How it works
Create
- Validate the supplied service account key.
- Create a new GCP service account key.
- Wait until Google fully accepts the new key.
- Store the new key in the Vault.
- Delete the original supplied key.
Checkout
- Read the Vault-owned key.
- Generate a short-lived access token.
- Return the token, expiry, and scope.
Rotation
- Read the existing key.
- Create a new key in Google.
- Verify that the new key works.
- Save the new key in the Vault.
- Delete the previous key.
This ordering ensures that the old working key is not deleted before the new key is safely stored.
5.2 Google Key Propagation Issue
One important issue was found during implementation.
A newly created GCP service account key is not necessarily accepted by all Google services at the same time.
For example:
- New key is created.
- Token generation succeeds.
- IAM request using the same key still fails.
- Google returns:
invalid_grant: Invalid JWT Signature
Because of this, checking only token generation was not sufficient.
Fix
A new key is now considered ready only when:
- token generation succeeds, and
- IAM key read succeeds.
The retry delays are:
2s
4s
8s
15s
15s
15s
15s
15s
Maximum wait is approximately 89 seconds.
5.3 Required GCP Permission
Approach 2 requires:
roles/iam.serviceAccountKeyAdmin
The role must be granted on the service account itself.
Example:
gcloud iam service-accounts add-iam-policy-binding <SERVICE_ACCOUNT_EMAIL> </span>
--project=<PROJECT_ID> </span>
--member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" </span>
--role="roles/iam.serviceAccountKeyAdmin"
Without this permission, Vault cannot create or delete service account keys.
5.4 Rotation
Approach 2 supports:
- Manual rotation
- Secret-level duration rotation
- Box-level rotation
Minimum rotation duration:
30 minutes
Example:
"rotation": {
"duration": "PT30M"
}
Lease is still not supported.
5.5 Scheduled Rotation Test
Automatic rotation was tested using:
PT30M
Result:
- Daemon automatically rotated the secret at approximately 31.3 minutes
- Previous key was removed
- New key became active
- Checkout worked after rotation
- New access token was accepted by Google
- Cleanup completed successfully
Result: PASSED
5.6 End-to-End Testing
17 of 17 checks passed.
The flow validated:
- Login
- Initial key exists in Google
- Secret creation
- Creation of Vault-managed key
- Removal of supplied key
- Checkout
- Token validity
- Manual rotation
- New key activation
- Removal of previous Vault key
- Checkout after rotation
- Lease rejection
- Invalid-key rejection
- Secret deletion
- GCP key cleanup
5.7 Issues Found and Fixes
# | Issue | Cause | Fix |
|---|---|---|---|
1 | Key creation denied | Service account had no permission on itself | Granted Service Account Key Admin on itself |
2 |
| New key was not propagated to all Google services | Added retry logic |
3 | New key remained after failure | Cleanup used the new key before it was fully working | Cleanup now uses old key first |
4 | Token check passed but IAM request failed | Token and IAM services accept new keys at different times | New key must pass both token and IAM checks |
6. Key Findings
Google Cloud
- Service accounts do not automatically have Key Admin permission on themselves.
- A newly created key may not become usable across all Google services at the same time.
- Token service may accept a key before IAM accepts it.
- Deleting a key does not revoke previously generated access tokens.
- A service account can have a maximum of 10 keys.
Vault
- Cleanup order is important.
- The existing working key should be preserved until the new key is verified and stored.
- A failure during key creation should leave the original key usable.
- A failure while saving a rotated key removes the newly created Google key.
- Failure to delete the previous key after successful rotation is logged but does not fail the rotation.
- Create and rotation can take roughly 90 seconds in the worst case.
7. Security Considerations
Both Approaches
- Access tokens inherit the service account permissions allowed through their OAuth scope.
- Apply least-privilege permissions to the service account.
- Use a narrower
gcp_checkout_scopewhere possible. - Service account key files should not remain on shared machines.
- Remove local copies after they are added to PASM.
Approach 1
A leaked private key remains usable until somebody manually removes it from Google Cloud.
Approach 2
- Always provide a dedicated key during create because Vault deletes the supplied key.
- Grant Key Admin only on the required service account.
- Keep capacity below Google's 10-key limit so rotation has room to create the replacement key.
iam.disableServiceAccountKeyCreationprevents this approach from working.- If deletion of an old key fails, manual cleanup may be required.
8. Final Recommendation
For the final PASM implementation, Approach 2 is the stronger option when automated credential lifecycle management is required.
It provides:
- Short-lived checkout tokens
- Automatic GCP key rotation
- Manual and scheduled rotation
- Key cleanup
- Recovery/rollback handling
- No unchanged long-lived key sitting indefinitely