Version: v2

PayeeConfig Encryption & Key Rotation (Internal)

Overview

Payee configuration data (address, tax ID, patient account number, payer 835 ID, etc.) contains sensitive PII/PHI. To protect this data at rest, Convenient Checkout encrypts the serialized payee details using AES-256-GCM before persisting them in the database.

Keys are stored in Azure Key Vault (never in source code or app config). Each encrypted payee record also stores the Key Vault secret version used to encrypt it, enabling zero-downtime key rotation.

Data Model

Where the encrypted data lives

Payee configs are stored on the merchant aggregate:

VendorMerchant.payeeConfigs (type: PayeeConfigs, from wallet-event-commons)
- defaultPayeeId
- payees: List<PayeeConfig>

Sensitive payload is stored per payee in:

PayeeConfig.encryptedPayeeDetails (ciphertext or legacy plaintext)
PayeeConfig.keyVersion (Key Vault secret version)

`PayeeConfig` (wallet-event-commons)

Stored per-payee on VendorMerchant.payeeConfigs.

Field	Type	Description
`payeeId`	`UUID`	Unique identifier for the payee entry.
`encryptedPayeeDetails`	`String`	AES-256-GCM ciphertext prefixed with `enc:v1:`, or legacy plain text for pre-encryption rows.
`keyVersion`	`String`	Azure Key Vault secret version used to encrypt this row; required for decryption.

`PayeeConfigs` (wallet-event-commons)

Top-level container stored on VendorMerchant.

Field	Type	Description
`defaultPayeeId`	`UUID`	The payee used by default when none is specified on a payment.
`payees`	`List<PayeeConfig>`	All payee entries for this vendor-merchant.

`PayeeConfigDTO` (wallet-merchant-service)

Inbound API representation validated before encryption.

Field	Constraints	Description
`payeeName`	Required, max 35 chars, `[A-Za-z0-9 !@#$%&()_+=\-;\",'./?]`	Payee name.
`payeeAddressLine1`	Required, max 30 chars, `[A-Za-z0-9 !@#$%&()_+=\-;\",'./?]`	Street address line 1.
`payeeAddressLine2`	Optional	Street address line 2.
`payeeCity`	Required, max 24 chars, letters & spaces only	City.
`payeeState`	Required, max 2 chars, letters only	US state code.
`payeeZip`	Required, max 10 chars, alphanumeric	ZIP / postal code.
`payeeCountry`	Required	Country code.
`payeePatientAccountNumber`	Required, max 38 chars	Patient account number used in payment routing.
`payeeTaxId`	Required, max 15 chars, alphanumeric	Payee tax identifier.
`payer835Id`	Required, max 5 chars, alphanumeric	835 payer ID used for claim routing.

🔒 InternalInternal information — not visible in the public (merchant) site.

Architecture

+-----------------------------------------------------------------------+
| wallet-merchant-service |
| |
| Write path (encrypt-on-save) |
| Merchant API -> MerchantService |
| -> PayeeConfigEncryptionHelper.encryptAll(...) |
| -> PayeeEncryptionService.encrypt(...) |
| -> KeyVersionedSupplier (Azure Key Vault) |
| |
| Rotation path (re-encrypt stale versions) |
| KeyRotationService.rotate(trigger) |
| -> PostgreSQL advisory lock (cluster-wide singleton) |
| -> Re-encrypt stale payees to latest key version |
| -> Publish MerchantEvent (MERCHANT_UPDATED_EVENT) |
+-----------------------------------+-----------------------------------+
|
| MerchantEvent (PayeeConfigs encrypted)
v
+-----------------------------------------------------------------------+
| wallet-payment-service |
| |
| Ingest path |
| MerchantConsumer persists PayeeConfigs as received (encrypted) |
| |
| Runtime path (decrypt-on-use) |
| CreateUserPaymentCommandHandler |
| -> PayeeEncryptionService.decrypt(..., keyVersion) |
| -> KeyVersionedSupplier (Azure Key Vault) |
+-----------------------------------------------------------------------+

--- wallet-event-commons (shared library) ---
PayeeEncryptionService (core logic)
AesGcmUtils (AES-256-GCM)
KeyVersionedSupplier (KV abstraction)
PayeeConfig / PayeeConfigs (POJOs)

Key Components

`PayeeEncryptionService` (wallet-event-commons — framework-agnostic core)

Located at com.optum.wallet.common.util.payment.vendor.crypto.PayeeEncryptionService.

Pure-Java library class with no Spring dependency. Both services wrap it with a thin @Service adapter that supplies the KeyVersionedSupplier from their own Azure Key Vault client.

Method	Description
`encrypt(String plainPayeeDetails)`	Encrypts the plain text using the current key version; returns ciphertext prefixed `enc:v1:` and the `keyVersion` used.
`decrypt(String storedValue, String keyVersion)`	Decrypts an `enc:v1:` prefixed value with the specified key version. Returns the value unchanged for legacy plain-text rows (backward compatibility).
`isEncrypted(String value)`	Returns `true` if the value starts with `enc:v1:`.
`currentKeyVersion()`	Returns the current (latest) Key Vault secret version identifier.

Versioned-key design

Every encrypted row stores the keyVersion (Azure Key Vault secret version ID) alongside the ciphertext. On decryption the service fetches that specific version from Key Vault. This enables zero-downtime key rotation: new rows are encrypted with the new version; old rows remain decryptable as long as the old version exists in the vault.

Backward compatibility

Rows that pre-date encryption (plain-text encryptedPayeeDetails) are detected by the absence of the enc:v1: prefix and are returned as-is by decrypt(). Plaintext payee details are encrypted when the merchant is created or updated via the Merchant API. Encrypted records with a null or blank keyVersion require verified backfill or explicit re-encryption before key rotation to ensure they can be safely re-keyed.

`AesGcmUtils` (wallet-event-commons)

Low-level utility that performs AES-256-GCM operations.

Algorithm: AES/GCM/NoPadding
IV: 12 random bytes generated per encryption via SecureRandom
GCM tag: 128 bits
Key size: exactly 256 bits (32 bytes); enforced at runtime
Wire format: Base64( IV[12] || ciphertext+tag )

`KeyVersionedSupplier` (wallet-event-commons)

Interface that decouples the core encryption logic from any specific key-store SDK.

    public interface KeyVersionedSupplier {  
        /** Returns the Base64-encoded AES-256 key for the given version. */  
        String getKey(String version);  
  
        /** Returns the current (latest) version identifier from the key store. */  
        String currentVersion();  
    }  

Both services implement this using their Key Vault client.

Service Responsibilities

wallet-merchant-service

Responsibility: encrypt payee details on write.

Merchant API receives a VendorMerchant payload containing a PayeeConfigs object with plain-text payee details in encryptedPayeeDetails.
MerchantService delegates to PayeeConfigEncryptionHelper.encryptAll(PayeeConfigs).
The helper iterates each PayeeConfig:
- Skips entries where details are blank.
- Skips entries already prefixed with enc:v1: (idempotent re-saves).
- Calls PayeeEncryptionService.encrypt(details) for all other entries.
Stores the encrypted VendorMerchant and publishes a MerchantEvent containing encrypted PayeeConfigs.

Spring adapter (merchant-service)

Property: merchant.payee.encryption.secret-name
Strips trailing whitespace/newlines from Key Vault secret values before Base64 decoding.

wallet-payment-service

Responsibility: store encrypted payee configs and decrypt on payment processing.

MerchantConsumer receives MerchantEvent and persists the PayeeConfigs (already encrypted) onto the Merchant entity — no decryption at this stage.
CreateUserPaymentCommandHandler decrypts payee details at payment-processing time via PayeeEncryptionService.decrypt(encryptedPayeeDetails, keyVersion).

Key Vault Configuration

Vault names & secret names (by environment)

Non-prod

Key Vault: fcc-comn-chkt-kv-dev
Secrets (enabled):
- ccg-payment-payee-encryption-key-dev
- ccg-payment-payee-encryption-key-perf
- ccg-payment-payee-encryption-key-reg
- ccg-payment-payee-encryption-key-stage
- ccg-payment-payee-encryption-key-test

Prod

Key Vault: ccg-comn-chkt-kv-prod
Secret (enabled):
- ccg-payment-payee-encryption-key

App property

Both services reference the same Key Vault secret per environment, but use different Spring property paths:

wallet-merchant-service (and Event Grid webhook): merchant.payee.encryption.secret-name
wallet-payment-service: payment.payee.encryption.secret-name

Secret format

The Key Vault secret value must be a Base64-encoded 256-bit (32-byte) AES key.

Generate:

openssl rand -base64 32

Store the output directly as the secret value.

Local development

For local dev profiles (LocalKeyVaultClient), use a static Base64 key by implementing KeyVersionedSupplier with fixed values:

    KeyVersionedSupplier localSupplier = new KeyVersionedSupplier() {  
        @Override  
        public String getKey(String version) { return localBase64Key; }  
  
        @Override  
        public String currentVersion() { return "local"; }  
    };  
  
    new com.optum.wallet.common.util.payment.vendor.crypto.PayeeEncryptionService(localSupplier);  

Sequences

Sequence: Merchant Save (Encrypt)

    Client -> MerchantController  
      -> MerchantService.saveMerchant(vendorMerchant)  
        -> PayeeConfigEncryptionHelper.encryptAll(payeeConfigs)  
          -> [for each PayeeConfig]  
            -> PayeeEncryptionService.isEncrypted(details) -> false  
            -> PayeeEncryptionService.encrypt(details)  
              -> KeyVault currentVersion() -> version  
              -> KeyVault getKey(version) -> base64Key  
              -> AesGcmUtils.encrypt(details, aesKey) -> ciphertext  
              -> store PayeeConfig(enc:v1:<payload>, version)  
        -> save encrypted VendorMerchant  
        -> publish MerchantEvent (encrypted)  

Sequence: Payment Processing (Decrypt)

    PaymentCommand -> CreateUserPaymentCommandHandler  
      -> merchantRepository.findMerchant(...)  
      -> PayeeEncryptionService.decrypt(encryptedPayeeDetails, keyVersion)  
        -> isEncrypted? -> yes (enc:v1: prefix)  
        -> KeyVault getKey(keyVersion) -> base64Key  
        -> AesGcmUtils.decrypt(cipherPayload, aesKey) -> plainText  
      -> use plain-text payee details for payment routing

Key Rotation (merchant-service)

Purpose

Rotation rewrites payee configs encrypted with stale Key Vault secret versions so the dataset converges to the latest version.

Implementation:

com.optum.wallet.merchant.service.KeyRotationService

What “stale” means

A merchant needs rotation if any payee has:

payee.keyVersion != latestKeyVersion

Concurrency control (cluster-wide singleton)

Rotation is guarded by a PostgreSQL advisory lock (one rotation at a time across the cluster):

lock name/resource: merchant-service:key-rotation

If lock is already held, rotation is skipped.

Safety rules during rotation

Rotation is non-destructive:

If a payee cannot be safely rotated, the original PayeeConfig is retained unchanged.

Common cases:

Legacy plaintext (encryptedPayeeDetails missing enc:v1:): not rotated (logged as not encrypted).
Encrypted but missing keyVersion: rotation may probe decryption; if it can’t decrypt reliably, it leaves the payee unchanged.

Persistence and downstream propagation

Merchants are saved only if at least one payee was re-encrypted.
After saving, merchant-service publishes MerchantEvent of type MERCHANT_UPDATED_EVENT so downstream consumers ( payment-service) converge on the new keyVersion values.

Rotation Triggers

Rotation can be initiated via:

Manual REST call (internal endpoint)
Azure Event Grid webhook (Key Vault secret new version)
(Optional) scheduler trigger (if configured)

All triggers call: KeyRotationService.rotate("<trigger>")

Controllers / Endpoints (Internal)

Manual Trigger (REST)

Controller: com.optum.wallet.merchant.controller.KeyRotationController
Method: POST
Path: /v1/encryption/key-rotation
Behavior: triggers keyRotationService.rotate("REST")

Response mapping (controller behavior):

Condition	`KeyRotationResponse.status`	HTTP status
`result.latestKeyVersion() == null` (advisory lock already held by another rotation)	`SKIPPED`	`200 OK`
`result.rotated() > 0 && result.failed() == 0`	`ROTATED`	`202 ACCEPTED`
`result.failed() > 0 && result.rotated() == 0`	`FAILED`	`422 UNPROCESSABLE_ENTITY`
`result.failed() > 0` (and rotated > 0)	`PARTIAL`	`202 ACCEPTED`
else	`UP_TO_DATE`	`200 OK`

Event Grid Webhook (Key Vault -> merchant-service)

Controller: com.optum.wallet.merchant.controller.EventGridWebhookController
Base path: /api/eventgrid/v1.0
Route: POST /api/eventgrid/v1.0/keyvault-secret-event
Consumer: com.optum.wallet.merchant.event.consumer.keyvault.KeyVaultSecretVersionConsumer

Supported Event Grid event types:

Microsoft.EventGrid.SubscriptionValidationEvent
- returns 200 OK with body containing validationResponse from the event.
Microsoft.KeyVault.SecretNewVersionCreated
- triggers rotation only if the event's data.ObjectName matches ${merchant.payee.encryption.secret-name}
- responds quickly with 202 ACCEPTED and starts rotation asynchronously

Important: rotation does not trust the version in the event payload; it fetches the latest version directly from Key Vault.

Observability (Rotation Logs)

Rotation uses structured event= log keys (examples):

KEY_ROTATION_STARTED
KEY_ROTATION_SKIPPED
KEY_ROTATION_COMPLETED
KEY_ROTATION_FAILED
KEY_ROTATION_TRIGGERED
PAYEE_CONFIG_NOT_ENCRYPTED
PAYEE_CONFIG_NULL_KEY_VERSION
PAYEE_CONFIG_DECRYPT_FAILED
PAYEE_CONFIG_ENCRYPT_FAILED

Key versions should be masked in logs (e.g., only last 8 chars).

Operational Runbook

When to run rotation manually

Use POST /v1/encryption/key-rotation when:

a new Key Vault secret version was created but Event Grid delivery is delayed/missed
you need to force convergence after fixing encryption/config
you want immediate remediation before a release window closes

Example:

    curl -X POST https://<merchant-service-host>/v1/encryption/key-rotation

How to interpret the response

UP_TO_DATE (200): nothing needed rotation
ROTATED (202): one or more payees were rotated successfully
PARTIAL (202): some payees rotated; some failed (check logs)
FAILED (422): processed but none rotated successfully
SKIPPED (200): another rotation is already in progress (PostgreSQL advisory lock is held by another instance)

Error Handling

All encryption/decryption failures are wrapped in PayeeCryptoException (unchecked). Common causes:

Cause	Exception message
Blank payee details on encrypt	`"Payee details are empty"`
Missing `keyVersion` for encrypted row	`"Missing key version for encrypted payee details"`
Key Vault returns empty/null key	`"Encryption key is empty"`
Key Vault key is not valid Base64	`"Encryption key is not valid Base64"`
Key is not 256-bit	`"Invalid AES key length: N bytes"`
JCE encryption failure	`"Failed to encrypt payee details"`
JCE decryption failure	`"Failed to decrypt payee details"`

Troubleshooting

Rotation didn’t trigger after creating a new secret version

Check:

Event Grid subscription targets POST /api/eventgrid/v1.0/keyvault-secret-event.
Subscription validation succeeded (service returns the validationResponse).
data.ObjectName equals ${merchant.payee.encryption.secret-name}.
Logs for event=KEY_ROTATION_TRIGGERED trigger=EVENT_GRID ... or ignore logs for unrelated secrets.

Rotation ran but payees still show old key versions

Check:

completion log: event=KEY_ROTATION_COMPLETED ... rotated=<n> failed=<n>
per-payee failures (PAYEE_CONFIG_*_FAILED)
downstream consumption of MERCHANT_UPDATED_EVENT in payment-service

Testing

PayeeEncryptionServiceTest (commons, merchant-service, payment-service): round-trips, legacy pass-through, blank guards, invalid key detection.
PayeeConfigEncryptionHelperTest (merchant-service): idempotency guard, null/blank skip, full encryptAll behavior.

Overview​

Data Model​

Where the encrypted data lives​

PayeeConfig (wallet-event-commons)​

PayeeConfigs (wallet-event-commons)​

PayeeConfigDTO (wallet-merchant-service)​

Architecture​

Key Components​

PayeeEncryptionService (wallet-event-commons — framework-agnostic core)​

Versioned-key design​

Backward compatibility​

AesGcmUtils (wallet-event-commons)​

KeyVersionedSupplier (wallet-event-commons)​

Service Responsibilities​

wallet-merchant-service​

wallet-payment-service​

Key Vault Configuration​

Vault names & secret names (by environment)​

Non-prod​

Prod​

App property​

Secret format​

Local development​

Sequences​

Sequence: Merchant Save (Encrypt)​

Sequence: Payment Processing (Decrypt)​

Key Rotation (merchant-service)​

Purpose​

What “stale” means​

Concurrency control (cluster-wide singleton)​

Safety rules during rotation​

Persistence and downstream propagation​

Rotation Triggers​

Controllers / Endpoints (Internal)​

Manual Trigger (REST)​

Event Grid Webhook (Key Vault -> merchant-service)​

Observability (Rotation Logs)​

Operational Runbook​

When to run rotation manually​

How to interpret the response​

Error Handling​

Troubleshooting​

Rotation didn’t trigger after creating a new secret version​

Rotation ran but payees still show old key versions​

Testing​