For assured success, it is important to monitor your systems for ongoing operational efficiency, security and compliance to internal policies. In June we shared with you our Enterprise DevOps Framework in which inspectors, like logs, play a critical role in analyzing services in the pipeline and landing zone to ensure compliance with operational, security, and regulatory requirements. At Flux7 we universally recommend customers use Amazon CloudWatch Logs for this purpose -- even if you are using Splunk or another log solution, we recommend CloudWatch Logs as a first stop for your logs as it is a more robust solution as we will discuss. First, let’s review Amazon CloudWatch, and CloudWatch Logs and then we’ll discuss why they should be the first stop for your AWS system logs.
CloudWatch monitors AWS resources and the applications you run on AWS in real time. It is reliable, scalable and easy to use as there is minimal infrastructure set up to do. Within minutes, you can begin monitoring your AWS architecture -- all in near real-time. Amazon CloudWatch can monitor a host of Amazon services including Amazon EC2 instances, Amazon EBS volumes, Elastic Load Balancers, and Amazon RDS DB instances.
With Amazon CloudWatch, you can access up-to-the-minute statistics, view graphs, and set alarms for your metric data to help you troubleshoot, spot trends, and take automated action based on the state of your cloud environment. Amazon CloudWatch functionality is accessible via API, command-line tools, the AWS SDK, and the AWS Management Console. Metrics such as CPU utilization, latency, and request counts are provided automatically for these AWS resources. You can also supply your own custom application and system metrics, such as memory usage, transaction volumes, or error rates, and Amazon CloudWatch will record these too.
CloudWatch has two primary features: CloudWatch Events and CloudWatch Logs.
Amazon CloudWatch Events
Events watch for operational changes as they occur and let you respond to them by activating functions, making changes, and capturing state information. CloudWatch Events can also be used to schedule automated actions that self-trigger at a given time.
Amazon CloudWatch Logs
Amazon CloudWatch Logs let you monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, Lambda functions, VPC flow logs, or other sources. With CloudWatch Logs, you can troubleshoot your systems and applications using your existing system, application, and custom log files from your applications.
Moreover, with CloudWatch Logs, you can monitor your logs in near real-time for specific phrases, values, or patterns (metrics). For example, you could set an alarm for the number of errors that occur in your system logs or view graphs of web request latencies from your application logs. And, you can view the original log data to see the source of the problem if needed.
Why we Recommend CloudWatch Logs
We recommend CloudWatch logs to customers to monitor, store, and access log files from Amazon EC2 instances, AWS CloudTrail, VPC Flow logs, Lambda logs and other sources. At the heart of our recommendation is that CloudWatch Logs:
- Are a managed service which can be easily provisioned from within your AWS account with no extra purchases. They are easy to work with via AWS console or CLI. You can search the logs or filter them easily via one or more metric filters. You can even tag CloudWatch Log Groups, making this task even easier.
- Have deep integration with many AWS services, e.g., Lambda and ECS.
- Can forward data to a Kinesis Stream, Lambda function, or to an ELK cluster created via the AWS ElasticSearch service.
- Can monitor not just logs from the instances but also from other source, e.g., security logs from AWS CloudTrail logs.
- Can trigger alerts on certain patterns occurring in logs, e.g., when root account credentials are used to login to an AWS account or if there are too many failed login attempts.
- Are stored in the highly durable S3 service. Note that the first 5GB of ingested log volume and first 5GB of archived log data is free every month as a part of the free tier. By default, log data is stored in CloudWatch Logs indefinitely. However, you can configure how long to store log data in a log group. Any data older than the current retention setting is automatically deleted.
- Do not require any extra network connectivity as they are available in AWS.
- Have integration with common tools where they can scoop data from CloudWatch Logs natively. For example, Splunk recently introduced the Lambda blueprint for CloudWatch Logs, which makes streaming AWS CloudWatch Logs via AWS Lambda and into Splunk incredibly easy. We will discuss this point further below.
Integrating CloudWatch with other logging solutions
If you are using Splunk or another logging solution, we highly recommend CloudWatch Logs as the first stop for your logs, i.e., forward logs from AWS resources like EC2 instances to CloudWatch Logs first and then forward them from CloudWatch Logs to your log aggregator. Customers do this to have a single pane of glass view for all their on-premise and AWS logs.
This architectural approach is helpful as it:
- Allows you to take advantage of centralized alerting and provides you the ability to trigger CloudWatch Events based on these alerts.
- Gives you one source of IAM control for your logs, which helps you more securely control access to your logs and associated data.
For example, at a Fortune 500 manufacturing organization, we used CloudWatch Logs as part of a serverless monitoring and notification pipeline wherein we used Lambda functions to log data to AWS CloudWatch Logs. From these logs, we used Lambda to trigger and copy the data into the customer’s existing in-house ELK cluster where the customer setup a dashboard to view and use the audit information.
Similarly, for another customer, we are forwarding their logs to CloudWatch Logs from not just CloudTrail, VPC flow logs but also from EC2 instances, (that is, both security logs and application logs). From there, the logs are forwarded to a SIEM solution on-premise. We’ve also had customers that use CloudWatch Logs as their primary logging solution, using the service as their primary DevOps inspector.
AWS CloudWatch Logs are a powerful mechanism for continuous monitoring to ensure that companies maintain their desired system state. Whether you have an external logging solution or not, the DevOps team at Flux7 recommend CloudWatch Logs as the first stop for all logging data.
For additional reading on best practice AWS architectures, please check out our AWS case studies or subscribe to our blog below.