Profile Applicability:
 • Level 1

Description:
 Consider using an external secrets storage and management system instead of relying solely on Kubernetes Secrets when you have complex secret management needs. Ensure that the chosen solution provides secure access control, requires authentication to access secrets, supports auditing, and encrypts the secrets. Many solutions also facilitate the rotation of secrets, which can significantly improve security.

Rationale:
 Kubernetes supports secrets as first-class objects, but managing them within Kubernetes may not meet all organizational or security requirements. An external secrets management system can provide advanced features such as centralized management, cross-environment support (for both Kubernetes and non-Kubernetes environments), and better access control mechanisms. These systems typically offer encryption at rest, authentication, and auditing capabilities, which help in meeting compliance requirements and enhancing security.

Impact:
 Pros:

  • Easier to manage secrets across multiple environments, including Kubernetes and other infrastructure.

  • Enhanced security by integrating with authentication systems and logging access attempts.

  • Simplifies secret rotation and auditing, improving overall security posture.

Cons:

  • Additional complexity and overhead in managing external secrets storage systems.

  • Potential integration issues or complexities when transitioning from Kubernetes-native secrets to external solutions.

Default Value:
 By default, Kubernetes does not configure any external secret management systems, and secrets are stored and managed internally within Kubernetes.

Pre-requisites:

  • A solution for managing external secrets, either from a cloud provider or a third-party service.

  • Integration with Kubernetes to handle the retrieval of secrets from the external system securely.

  • The necessary permissions to configure external secret management systems.

Remediation

Test Plan:

Using AWS Console:

  1. Review your existing secrets management approach and check if any external secret storage solution is in use.

  2. Evaluate the use of services such as AWS Secrets Manager, HashiCorp Vault, or similar services to store and manage secrets outside of Kubernetes.

Using AWS CLI:

  1. Review your current configuration to see if any external secrets storage is integrated. For AWS services, you can check if AWS Secrets Manager is being used:

aws secretsmanager list-secrets

  1. If no external solution is configured, consider setting up an external secrets manager, such as AWS Secrets Manager or HashiCorp Vault, and integrate it with Kubernetes.

Implementation Plan

Using AWS Console:

  1. If using AWS Secrets Manager, ensure that you have the necessary IAM permissions to access secrets.

  2. Set up AWS Secrets Manager to store secrets, ensuring that secrets are encrypted and access is restricted to authorized users and services.

  3. Configure Kubernetes to access the external secrets store by using tools like the Kubernetes External Secrets project, which allows Kubernetes to retrieve secrets from AWS Secrets Manager, HashiCorp Vault, or similar solutions.

Using AWS CLI:

  1. Set up AWS Secrets Manager with your secrets by running:

aws secretsmanager create-secret --name <secret-name> --secret-string "<secret-value>"

  1. Install and configure the Kubernetes External Secrets operator to fetch secrets from AWS Secrets Manager into Kubernetes:

kubectl apply -f https://github.com/external-secrets/kubernetes-external-secrets/releases/download/v<version>/kubernetes-external-secrets-<version>.yaml

  1. Define an ExternalSecret resource to specify how Kubernetes should retrieve secrets from AWS Secrets Manager:

apiVersion: eksctl.io/v1alpha5
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: secretsManager
  data:
    - key: <secret-name>
      name: <k8s-secret-name>

  1. Apply the configuration to Kubernetes:

kubectl apply -f external-secret-definition.yaml

Backout Plan

Using AWS Console:

  1. If the external secret management configuration causes issues, revert the application back to using Kubernetes-native secrets.

  2. Modify the configurations in Kubernetes to retrieve secrets directly from Kubernetes Secret objects, if necessary.

Using AWS CLI:

  1. If you need to revert to Kubernetes-native secrets, delete the external secret configuration and modify the pod/deployment definitions to use Kubernetes secrets.

kubectl delete -f external-secret-definition.yaml

  1. Ensure that your applications revert to using Kubernetes secrets instead of external storage.

References:

  1. Kubernetes Documentation on Secrets

  2. HashiCorp Vault Documentation
  3. AWS Secrets Manager Documentation