Controlling access to sensitive information, or secrets, required by your applications is a ubiquitous architectural requirement. Your applications need information like passwords, API keys, and certificates, and as the application owner you need to ensure this information is only accessed by the correct application. You also need to know when this information was accessed and by which entity.
In an organization pursuing DevOps best practices -- with a cultural view that everything needs to be committed into code, and there can’t be any steps requiring human intervention -- this problem seems closer to a paradox. Especially, with the adoption of microservices in which your application is split into many small components, the number of secrets can increase exponentially. This propagation can lead to difficulty in efficiently managing your secrets and if you are an organization that has stringent compliance requirements, then this can seem especially daunting.
In this article, we examine the use case of a healthcare organization that sought an enterprise grade solution that could help automate and streamline the process of secrets management while also ensuring its data met HIPAA compliance regulations in order to protect patient healthcare records.
Introducing HashiCorp Vault
Vault is an excellent choice for automated secret management for a number of reasons:
- Vault provides an interface to manage encrypted static secrets, as well an interface for generating dynamic secrets
- The Vault “SSH backend” can dynamically generate SSH credentials for remote hosts, removing the need to share private keys with all users needing access to infrastructure
- Vault generates detailed audit logs capturing which secrets were used by which operator, at what time and on which system(s). This is a must-have to meet HIPAA audit requirements.
Containers help create high-availability immutable environments which make them a suitable choice for the secure introduction of a security tool like Vault. To implement Vault with containers, Flux7 uses AWS ECS.
We deploy two ECS clusters: Cluster 1 and Cluster 2. Vault runs as an ECS service in the ECS Cluster 1. Its storage backend, Consul, runs in Cluster 2. Vault nodes connect to the Consul backend on the ECS cluster 2 to retrieve credentials before returning them to the user. The diagram above depicts the architect and the path that inbound Vault requests follow.
While this architecture is sufficient for Vault, we use the strengths of ECS and HashiCorp Consul to provide an extensible cluster that can also run additional security and management services in the future. To convert this setup into a general purpose microservices architecture, we add Nginx as reverse proxy configured as an API gateway. It forwards incoming traffic to the appropriate containers based on the URL pattern. Nginx needs to have up-to-date information about where the containers of a given services are running (their IP and port). For this purpose we leverage Registrator, Consul, and Consul template. When a new container is launched anywhere on the ECS cluster, Registrator is aware of the container launch and checks it into Consul, specifying important information about the container like IP, service port, and health status. Consul sends a notification to all nodes when a particular key/value pair change. Consul Template maintains a “service file” on all nodes on the cluster with details for each API, and automatically updates this file when data in Consul changes, and reloads Nginx configuration.
To complete the picture, we also run consul-snapshot as a service on Cluster 1 to periodically snapshot the contents of Consul and upload them to AWS S3 for backups.
By using ECS, Vault, and Consul, Flux7 is able to to achieve a balance between security and agility. The healthcare organization is able to maintain strict governance over its credentials in a way that allows it to meet strict HIPAA compliance, but still easily retrieve important forensics about credentials, enforce policies in an automated way, and ultimately reduce day-to-day management for both the security and development teams. All of which allows the customer to focus more on its business: providing services that help take care of patient health. Find out more of the story in the full case study.