I am working on an application running in/on a hosted Kubernetes cluster. All services are built using a Java based stack.
For a new feature it is required to temporarily store notifications (email address, notification text) in database.
We want to encrypt this data to prevent leaking in case of a data loss/attack. We plan to use a public key based approach. While this allows for easy encryption, handling/storing the private key for decryption seems quite difficult.
We plan to have a single instance of a "notification service" for sending these notifications. Only this service needs to decrypt this data.
Goal is to not store the private key in the cluster (at least not persistent; so, in memory only).
I researched various options for storing the private key and for now, came to these options:
1: Use of an external key management service (e.g., using a REST API)
Pros:
- Easy to use/implement.
- Key is not directly available in our system (only after requesting from the external service); key could be requested for each batch or kept in memory for longer time.
Cons:
- Additional, external service.
- We must store the credentials for accessing the service. An attacker could use these credentials to access the key (at least until the credentials are disabled/blocked).
- Key is kept in memory (memory dump).
2: Use of a secrets manager (e.g., Hashi Vault)
Pros:
- Runs within the cluster (no external service).
- Easy to use.
Cons:
- Additional container(s) (backup, configuration, updates, …).
- Part of the cluster.
- Key is kept in memory (memory dump; on secret manager and where data is decrypted).
- Needs to be (manually?) unlocked after restart.
3: Use of an (virtual) HSM
Pros:
- Probably the best and most secure option.
- Key is NOT kept in memory.
- Encryption and decryption using the HSM.
Cons:
- Company hosting our cluster does not offer HSM (however, moving to another company is not an option now).
- Probably quite expensive (at least real hardware).
4: Load keys from "outside" (e.g., using a REST API), keep them in memory only
Pros:
- Easy to implement.
- No additional tools/services.
Cons:
- After restarting, keys must be loaded.
- Key is kept in memory (memory dump, where data is decrypted).
- Attacker might intercept loading/injection of keys somehow.
- It is necessary to securely store the keys outside of the cluster.
Are there any other options available? are there any "best practices" I have missed so far?
