While reviewing AWS Security and Monitoring in Linuxacademy.com's CSA-Pro course, I heard the instructor say that CloudWatch could be used to monitor application logs. I've only ever used CloudWatch to monitor metrics like CPUUtilization and DatabaseConnections, and only recently implemented custom metrics like DiskUtilization. Monitoring application logs looks like some next level sorcery! I'm in, let's do this!

I used this blog post as the basis of my exploration (when I grow up, I want to be just like Jeff Barr, he's my hero!):
https://aws.amazon.com/blogs/aws/cloudwatch-log-service/

Rather than create a new policy, I appended the sample policy into an existing cloudwatch policy. It took a little fiddling with a custom Stmt ID and tracking all the brackets and parentheses, but finally the policy validated.

Instead of doing the manual install, I followed the instructions here and used the awslogs package.

Setting up the log group and log streams was a bit confusing. Using CloudWatch's Logs interface, I settled on building a log set for my school webserver, cleverly named webServer. I created two streams, messages and and ypics.org.access_log. I didn't realize until much later that this step was completely unnecessary. ("They really should put the warnings before the spells"...)

In the /etc/awslogs/awslogs.conf file, I set up the stream definitions, leveraging the messages example and built one for the access_log. Turned on the server, it should all work, right?

Not so fast...

The dreaded "No Events Found" notice in "be calm, don't hurt anyone" blue filled warning rectangle. Stop, it's troubleshooting time... (Woah, Woah Woah)

In the /var/log/awslogs.log it looks like everything is fine:

2017-09-26 16:35:08,341 - cwlogs.push.publisher - INFO - 2572 - Thread-3 - Log group: webServer, log stream: webServer_messages, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1506443701000, 'start_position': 240553L, 'end_position': 240661L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1506443701000, 'start_position': 240553L, 'end_position': 240661L}, 'source_id': '6536545901daa6a75722ce388afbd37d', 'num_of_events': 1, 'batch_size_in_bytes': 133}
2017-09-26 16:36:56,721 - cwlogs.push.publisher - INFO - 2572 - Thread-5 - Log group: ypics.org.access_log, log stream: webServer_messages, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1506443811357, 'start_position': 2848118L, 'end_position': 2848201L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1506443811357, 'start_position': 2849033L, 'end_position': 2849183L}, 'source_id': 'aeb5ce683400f407f5adfd0cbdfe07bd', 'num_of_events': 8, 'batch_size_in_bytes': 1265}

At the command line, I checked to see of maybe the policy was messed up:

]$ aws cloudwatch list-metrics

Could not connect to the endpoint URL: "https://monitoring.us-west-2b.amazonaws.com/"

That doesn't look right.

I think I broke the policy. I guess that means valid JSON doesn't always mean correct JSON... Back to IAM! Time to build a custom policy.... No, that doesn't feel right. Instead, I ran sudo aws configure and set the region, but no keys. That seemed to fix the issue with aws cloudwatch list-metrics, but still no data was showing up in the stream. Hmmm....

This is what I get for not reading the instructions. The issue was that the /etc/awslogs/awscli.conf was pointing at us-east-1. As soon as I corrected the region and restarted, the logs started pouring it.

Too easy! Time to put the config on the rest of my servers for my log viewing pleasure!