CloudWatch for Server Logs, or what magic is this?

While reviewing AWS Security and Monitoring in Linuxacademy.com’s CSA-Pro course, I heard the instructor say that CloudWatch could be used to monitor application logs. I’ve only ever used CloudWatch to monitor metrics like CPUUtilization and DatabaseConnections, and only recently implemented custom metrics like DiskUtilization. Monitoring application logs looks like some next level sorcery! I’m in, let’s do this!

I used this blog post as the basis of my exploration (when I grow up, I want to be just like Jeff Barr, he’s my hero!):
https://aws.amazon.com/blogs/aws/cloudwatch-log-service/

Rather than create a new policy, I appended the sample policy into an existing cloudwatch policy. It took a little fiddling with a custom Stmt ID and tracking all the brackets and parentheses, but finally the policy validated.

Instead of doing the manual install, I followed the instructions here and used the awslogs package.

Setting up the log group and log streams was a bit confusing. Using CloudWatch’s Logs interface, I settled on building a log set for my school webserver, cleverly named webServer. I created two streams, messages and and ypics.org.access_log. I didn’t realize until much later that this step was completely unnecessary. (“They really should put the warnings before the spells”…)

In the /etc/awslogs/awslogs.conf file, I set up the stream definitions, leveraging the messages example and built one for the access_log. Turned on the server, it should all work, right?

Not so fast…

The dreaded “No Events Found” notice in “be calm, don’t hurt anyone” blue filled warning rectangle. Stop, it’s troubleshooting time… (Woah, Woah Woah)

In the /var/log/awslogs.log it looks like everything is fine:

2017-09-26 16:35:08,341 – cwlogs.push.publisher – INFO – 2572 – Thread-3 – Log group: webServer, log stream: webServer_messages, queue size: 0, Publish batch: {‘skipped_events_count’: 0, ‘first_event’: {‘timestamp’: 1506443701000, ‘start_position’: 240553L, ‘end_position’: 240661L}, ‘fallback_events_count’: 0, ‘last_event’: {‘timestamp’: 1506443701000, ‘start_position’: 240553L, ‘end_position’: 240661L}, ‘source_id’: ‘6536545901daa6a75722ce388afbd37d’, ‘num_of_events’: 1, ‘batch_size_in_bytes’: 133}
2017-09-26 16:36:56,721 – cwlogs.push.publisher – INFO – 2572 – Thread-5 – Log group: ypics.org.access_log, log stream: webServer_messages, queue size: 0, Publish batch: {‘skipped_events_count’: 0, ‘first_event’: {‘timestamp’: 1506443811357, ‘start_position’: 2848118L, ‘end_position’: 2848201L}, ‘fallback_events_count’: 0, ‘last_event’: {‘timestamp’: 1506443811357, ‘start_position’: 2849033L, ‘end_position’: 2849183L}, ‘source_id’: ‘aeb5ce683400f407f5adfd0cbdfe07bd’, ‘num_of_events’: 8, ‘batch_size_in_bytes’: 1265}

At the command line, I checked to see of maybe the policy was messed up:

]$ aws cloudwatch list-metrics

Could not connect to the endpoint URL: “https://monitoring.us-west-2b.amazonaws.com/”

That doesn’t look right.

I think I broke the policy. I guess that means valid JSON doesn’t always mean correct JSON… Back to IAM! Time to build a custom policy…. No, that doesn’t feel right. Instead, I ran sudo aws configure and set the region, but no keys. That seemed to fix the issue with aws cloudwatch list-metrics, but still no data was showing up in the stream. Hmmm….

This is what I get for not reading the instructions. The issue was that the /etc/awslogs/awscli.conf was pointing at us-east-1. As soon as I corrected the region and restarted, the logs started pouring it.

Too easy! Time to put the config on the rest of my servers for my log viewing pleasure!

 

Rescheduled the Solutions Architect Pro Exam

In order to prepare for the SysOps Associate and the Solutions Architect Professional certifications, I intended to take two weeks vacation and spend every available minute studying and practicing. However, I was told that I could not take vacation, but had to be on call at least two hours a day. Still, a generous proposal, but I ended up spending much more time at work than I had thought, and the SysOps exam was more challenging that I expected. As I began to prepare for the SA-Pro, the linuxacademy.com lecturer was adamant that I don’t schedule until 2-3 weeks after I complete the course so I could practice and do the additional reading and practice tests. I had already booked the test, so the best compromise is to reschedule for a month from now. That gives me this week to finish the videos, and the rest of the time to practice. This blog is part of the reflection and practice process. I hope it might be interesting to some out there who are pursuing AWS certifications.

Here’s the count down (no stress!): https://daycalc.appspot.com/10/28/2017

Breadth of Knowledge at this point

So many AWS services and features, and more coming out every year at re:invent. Here’s a tally on what I know and what I want to know. The analogy of an iceberg’s above / below water ratio comes to mind.

Use Regularly

  • EC2 – Elastic Cloud Compute
  • S3 – Simple Storage Service
  • EFS – Elastic File Service (NFS)
  • RDS – Relational Database Service
  • VPC – Virtual Private Cloud
  • CloudFront (CDN)
  • Route 53 (DNS)
  • CodeCommit
  • CloudWatch
  • IAM
  • Simple Notification Service
  • WorkDocs
  • WorkSpaces

Experimented With

  • LightSail
    Elastic Beanstalk
    Glacier
  • DynamoDB
  • CloudFormation
  • CloudTrail
  • Config
  • OpsWorks
  • Trusted Advisor
  • Inspector
  • Directory Service
  • AppStream 2.0

Want to Learn

  • EC2 Container Service (Docker)
  • Lambda
  • Storage Gateway
  • ElastiCache
  • Direct Connect
  • Certificate Manager
  • WAF & Shield
  • Kinesis
  • Step Functions
  • SWF
  • API Gateway
  • Elastic Transcoder
  • Simple Queue Service
  • Simple Email Service
  • WorkMail
  • Amazon Chime

For Fun

  • Glacier
  • Redshift
  • CodeStar
  • CodeBuild
  • CodeDeploy
  • CodePipeline
  • Lex
  • Amazon Polly
  • Rekognition

Extra

  • Batch
  • AWS Migration Hub
  • Application Discovery Service
  • Database Migration Service
  • Server Migration Service
  • Snowball
  • Service Catalog
  • Managed Services
  • Artifact
  • Amazon Macie
  • CloudHSM
  • Athena
  • EMR
  • CloudSearch
  • Elasticsearch Service
  • Data Pipeline
  • QuickSight
  • AWS Glue
  • Machine Learning
  • AWS IoT
  • AWS Greengrass
  • Amazon Connect
  • Amazon Gamelift
  • Mobiler Hub
  • Cognito
  • Device Farm
  • Mobile Analytics
  • Pinpoint

My first VPC, my little buddy

While studying for the last of the associate certifications, I realized that if I was ever asked during an interview to log in and show what I have done, my AWS accounts would look like a kid’s toy chest, much played with but never organized. Creating a Virtual Private Cloud (VPC) is such a core skill, and it was finally time to rein in my personal account. I’ll tackle work on Monday.

I’ve used the VPC wizard before, but today was kind of a test for me, could I do it from memory? I have an adapted CloudFormation template for a private class C network, which is all I really need. However, I watched a re:invent video on VPC design and figured “why not use 65k IPs if they are free?” You never know, I might need a bigger network space one day, and it would be a pain to renumber it all.

So, the plan:
My region has three AZs, so three public subnets, for the day I might actually add a load balancer and scaling group. Fun for another day. I also created 3 private subnets:

10.0.1.0/24 – public-2a
10.0.2.0/24 – public-2b
10.0.3.0/24 – public-2c
10.0.100.0/24 – private-2a
10.0.101.0/24 – private-2b
10.0.102.0/24 – private-2c

I did have a moment of fun and created a 10.0.0.0/22 and revelled in all the room, but figured I could always redo the subnets later. 251 IPs is enough when I have one server instance and one database. I can dream big later.

First step, go to the console and create my first VPC. I didn’t realize until I saw a re:invent video on how the network works on the actual machine that I can have separate VPCs with the same CIDR blocks. Coming from a Cisco switching understanding of lans and vlans, it blew my mind that the VPC ID is just a tag added to the traffic to keep it separate from the rest of the bits and bytes. Still cleaning up bits of brain. Whoa.

Next step was to create the Internet GateWay (IGW) and attach it to my new VPC.

Next step was to create the routing table and associate the subnets with it. Right out of the gate, adding the 0.0.0.0/0 route and pointing it at the IGW. Wait a minute, now all the subnets can route out to the internet, how is that even private? This is where our friends the NAT instance and the NAT gateway come in. Ideally, creating the NAT gateway is the right answer, since it scales and doesn’t need managing or securing. However, it costs money, and I don’t see a need to patch servers in the private subnets right now, because I don’t have any. So, a NAT instance from the community AMI worked fine, I spun it up, changed the Source Destination Checks and gave it an Elastic IP. So far, so good.

Next, I created a new routing table with the NAT instance as the gateway. It popped right up as an option, didn’t have to look for it. Once I had the two routing tables, I assigned the public subnets to the main gateway and the private subnets to the NAT instance gateway, and then shut it down.

Almost there! I created a webDMZ security group that tightened down the access to the web server instance, and created a privateToPublic group that allowed only traffic from the webDMZ. Next, I spun up a test instance in a private subnet and installed mysql to test connectivity. Of course, I had to restart the NAT instance to allow the install.

I had a little trouble figuring out why the mysql traffic wasn’t working between the public and private subnets. I understood that subnets in the same VPC could communicate with each other, but I had to explicitly add mysql to the  the security group rules for each security group to reach each other, as well as explicitly add the public subnets to the the private subnet Access Control List (ACL). I need to go back and understand that better.

The fun part was getting the database in RDS to work, as AWS introduced a new interface and I had some fun with Subnet Groups and Security Groups, but that is another story for another day.

So, my personal account is divested from the default VPC, so I feel like my toy box is all cleaned up and put away properly.

Building a Personal Brand, step 1

The online job application process is broken, and everyone knew it but me.

Every day you get inundated with all these sites promising to connect you with the job of your dreams, help you write your resume, build your skills, and give you lots of helpful advice…

And your carefully crafted resume goes into the bucket with all the others, to be key word searched by bots and drones, and maybe, if you are lucky, you can get an interview. But it will take a lot of app submissions and cover letter repurposing to get there.

The painful truth is revealed; you can’t get a job unless you know someone already, or they know of you.

So, how do I get the word out about just how amazing and awesome and humble I am to all the world?

Build a personal brand.

Huh?

I’ve heard this before, about leveraging social media to market yourself, but I never gave it much thought. I can’t imagine anyone wanting to hear about my trials and mistakes. All my sins for the world to Google, that is just scary. I always worry that if people google my email address, they will find that one stupid question I asked to a linux board back in the late 90s. Please don’t hold that against me, I didn’t know what I didn’t know…

So, here we go, documenting the process to build a personal brand, so that if I am successful, others may have a path to follow that isn’t spread all over a million job boards and social media sites.

My first step has been to dust off my domain name, and start blogging again. I’m going to keep it strictly technology related, no politics or work stuff. My goal is to post something each day. I can write, I just don’t know if what I write is something others would read, or am I writing to my self, a journal with an expensive carbon foot print. We’ll see.

So, I published my first story and have started drafting a couple of others. I got my first retweets a few minutes ago, so at least a couple people found them interesting. It is also possible a bot just passed it along based on the keyword “AWS”, but hey, I’ll take it.

My next goal is to build out my consulting site. I’ve done a fair bit of work over the years, just need to document the projects and put them up. I haven’t had the need to do this before, so the site wont win any prizes for a while. The useful part was that I had a chance to put in my first VPC with public and private subnets, and get my websites all playing nice on the same instance. It gave me a chance to learn a bit about Apache’s ServerAlias directive. I’ve been putting duplicate entries with ServerName changes in my v-hosts.conf file all these years. Could have saved me some typing!

See, the stories just come to me, so the challenge is writing and publishing them. One step at a time.

AWS SysOps Associate testing experience – The Waiting Game

So, I passed the SysOps Associate exam on Saturday. At least, I think I did. I have an email that gave me my score, but I haven’t received my pdf with the cool logo so I can stick it in my email header. The new testing vendor is doing things a bit different than the last, and though the facilities were much more professional, there are a couple of hiccups that are giving me pause.

Previously, I took my certification tests at a Webassessor vendor on Topanga Blvd. It was a couple of cramped office cubicle spaces in a retired cop’s private investigator / training office on the third floor of a rather uninspired office building. I seem to only test on Saturday mornings, and the testing center was run by a lovely lady who always had her little girls under foot, running errands and keeping we test takers from getting too anxious. When I would complete a test, I would immediately get an email with my score, and then a second email with my certificate and badge / logo assets.

All that went away once AWS changed testing vendors to PSI and are running it all through the cool new certification portal. It is annoying to have to sign in a couple of times to get to my test info, but I imagine this is just growing pains, it will get simpler after a while.

The testing facility was a bit further away in Agoura Hills, but in Los Angeles terms it was conveniently close for me. There was some very loud construction going on in the lobby, which added to the anxiety of the testing experience. The testing facility was on the third floor, and when I walked in, it was clear that they were in the testing business, not just as a side line.

Once in the door, the proctor team was no nonsense. They take your primary and secondary IDs and use them as tokens to keep track of you and your stuff in the testing process. The primary ID is attached on your test scratch paper, and your secondary matched to your locker key, with which you secure all of your electronics and pocket contents immediately, no chance to cram in a little bit more studying before getting in line. If you are in the door, you are ready to test. I had rather counted on standing in line as people get signed in, but not this time!

The intake process was very efficient, much the same look and feel as being checked into a jail or medical facility, all white walls and cameras. The place could use some institutional happy colors , but otherwise you felt like they knew what they were doing.

Until I sat down to take the test…

The testing station was great, they gave me two pencils, earplugs and hearing protection such as you might get at the firing range. I had already signed off that the construction noise didn’t bother me, but I might have been able to reschedule if I wished to make an issue of it. Nice chair, well lit, cameras everywhere. It was a bummer that I couldn’t take off my shoes, but I can see why some folks might get bent out of shape about that. All ready to go, clicked Next, and waited.

And waited.

And waited.

And waited.

Finally I summoned the proctor, who told me that they had gotten an email that morning saying that there would be a delay between questions, but that the delay would not count against the test time limit.

ok….

And so it went, after each question, I had to stare at the animated circle of life for 25 to 45 seconds until the next question page loaded. I don’t know if it was a network issue, caching issue or just how the test is going to go from now on, but I had a mandatory moment of zen between each question. If you went back to review an answer, you also got a delay. Luckily, there was no penalty to the count down timer (I watched that sucker like a hawk) for the delay, but I nearly fell asleep between questions a couple of times. Lots of deep breathing.

Once I complete my test and submit it for grading, I am expecting the dreaded pass or fail page. Heart in my throat, I wait 30 seconds for it to load… and a feedback survey page pops up. Gah! I have to wade through a bunch of survey question pages, not knowing if I passed or not, and wait 30 seconds between questions to contemplate all my life’s bad decisions that led up to this moment!

After the survey, I got the feedback that I had passed, and I would get an email. Normally I take about a third of the allocated time for these tests, but on this occasion I was forced to take the whole thing. I was rehearsing my excuse to PSI asking for a refund if I didn’t pass, blaming my lack of preparation on the testing scenario…. Lucky I didn’t have to dance to that tune.

So, I get my score sheet, which tells me how I did in each section. There is one section I need to practice, but the rest held up enough for me to pass. However, I don’t see my certification PDF with the cert number, so I can’t update the resume. What is going on?

Looking at the certification portal, I have to wait 5 business days before I can expect any update to the website for proof that I passed. So, I get to practice patience and wait again.

I did pass, didn’t I? If a pdf fails to be created on the internet, does anyone hear it? In about 5 days someone is going to hear about it, for sure.

Anyway, I am grateful to the folks at A Cloud Guru, Linux Academy and Whizlabs for helping me prepare for the certification. I am also grateful for my parent organization, YPI Charter Schools, who gave me time off to prepare and let me experiment on their dime.

 

Update: I kid you not, the minute I published this article, I got an email directing me to look at my certification in the portal. Yep, there it is, 24 or so hours later. So much for instant gratification, but glad to have the proof at long last!