Print is the new digital and no one proves that more than one of our most recent customers who is a brand-name publishing house. Embracing a digital-first business model, this publisher’s IT, development and security teams sought to significantly reduce their on-premise footprint, moving to Docker and AWS. As part of its move, the need for a secret management solution was identified and we were happy to get the call to help address the company’s need with the Flux7 SmartStart for HashiCorp Vault.
The firm was using a solution for its usernames and passwords which was not programmable, and it was without a secure storage or secret store for its SSH keys, and KV pairs. As a result, it sought to switch to HashiCorp’s Vault a full-featured secret management solution of choice when building highly secure and highly available systems.
The first step of any Flux7 AWS Premier Consulting Partner engagement is an assessment in which we examine the technical, business and process needs. In doing so for this firm, we identified three clear steps for the project:
- Create a workflow using HashiCorp Vault and a custom security broker
- Deliver AWS CloudFormation templates to deploy HashiCorp Vault and HashiCorp Consul clusters
- Use AWS CloudFormation templates for AWS serverless application model (SAM) to deploy security broker
To start, the DevOps team at Flux7 developed the AWS CloudFormation templates that would allow it to deploy HashiCorp Consul -- a highly available and distributed service discovery and KV store -- and Vault in the publisher’s AWS environment. Flux7 used the Consul AWS auto-discovery feature to bootstrap the Consul cluster. We created a Vault Auto-Scaling Group (ASG) and added an ELB (Elastic Load Balancer) in front of Consul ASG and Vault ASG to ensure high system availability.
We secured the HashiCorp Consul - Vault communication with TLS (transport layer security) encryption and used Consul's snapshot endpoints for Backups and Disaster Recovery (DR). For accountability, and an audit trail, we configured AWS CloudWatch to capture logs which were in turn forwarded to Splunk as the company’s central log service.
Security Broker Approach
While Vault authentication backend is not enough, we used role ID and secret ID to authenticate to Vault. However getting this role ID and secret ID to the instances are a problem. This problem is referred to as the ‘Secret 0’ problem and it means that a third identity that can help establish trust between the client (containers and instances) and secret IDs is needed.
To resolve this issue, AWS Lambda was used as the security broker to serve role and secret IDs; the security broker client is set to authenticate against the API Gateway and fetch credentials. It should be noted that while this problem can be solved using the AWS IAM authentication method, we could not deploy this approach as this customer has a hybrid (Google and AWS) cloud architecture.
In the customer’s AWS Docker-based microservices environment, the container calls a function that asks for credentials to Vault. The request goes through an automated validation process, where tags on the infrastructure are looked at, access is given and when approved, the container is given one-time credentials which it then hands back.
This best practices layered security approach was supplemented with a skills training and knowledge transfer that assured the publisher’s technology teams would be able to apply and extend the tools and processes moving forward. Now the publishing house has a highly available federated installation of Vault and Consul that the team can actively manage themselves.
By proactively building in a cloud security architecture as it moved its systems to AWS, this firm has decreased risk from manual management while introducing automation that streamlines processes. Moreover, a highly secure and high availability design for Consul and Vault means that the system has zero downtime for applications and users alike.