Learning python with boto3: Client vs Resource

I’ve only recently spent any serious time learning Python, and usually just to solve a specific problem. I am not a programmer by training, just used to hacking out a script, one line at a time. Frequently, I really feel my lack of proper programming training, so I am using the bang-head-on-desk method of programming, figuring it out with the language reference close at hand, and by buying some video tutorials and books to eat up all of my sleep time.

Previously, I built a script that would automate the back of AMIs and copying them to the disaster recovery region, but that was really just substituting python for bash. Now, I have a problem to solve at the new gig that requires I learn Boto3, the AWS sdk for python. 

The issue is that I need to:

  1. interrogate each AWS account by ID,
  2. list the resources created,
  3. write those resources into a DynamoDB table so we can can see what is in each account,
  4. maintain state info when something changes.

I have figured out how to get ec2 instance data the hard way:

import boto3
ec2 = boto3.client('ec2')
response = ec2.describe_instances()

This gives you a json dump of all of your ec2 instances and all their information. Useful, but more than a little messy. However, this script calls the the default credentials from the ~/.aws/credentials file, and I am working with many profiles. So, I need to do that for each account, which introduced me to the idea of sessions. I need to add in my list of accounts, then go through each one. Creating the session was not difficult:

profiles = ['default','foo1','foo2']
for profile in profiles:
    session = boto3.Session(region_name='us-east-1', profile_name=profile)
# need to set the region_name, or will err

However, if you use the ec2 = boto3.client(‘ec2’) code here, you will not get any change of results. This introduced the idea of Resources and Clients.

I had trouble finding a web page that explained it clearly, and at this point, I’m not all that clear myself, but using the session variable and calling for a resource, I was able to get the info I needed:

ec2 = session.resource('ec2')
for instance in ec2.instances.all():

This gives me specific information, rather than the json object, and it changes with each new session, as expected. 

So, I need to get really familiar with the methods and attributes available to each service, and while it is a really thick read, it is all there in the docs. My next step is to start writing to the database. 



Mission Complete: Got that AWS job!

It took about 4 months after I completed the AWS Solutions Architect Professional exam, but I got hired on with a large consulting company as a member of the infrastructure group, and I am working with an information security group with a large media company. Luckily, I was able to impress during my in-person technical interview, so I got hired even though I didn’t get the role for which I interviewed at first. A month later, and I am fully immersed in augmenting and tuning a business application that leverages several AWS technologies. The learning curve is nearly vertical, so much to learn now that I know that I really know nothing, when it comes to applying AWS. My humble four EC2 instances and a database are nothing compared to the scale I am working in now. So, hitting the tutorials hard to learn the details of the parts I only knew at a 10,000 foot level. Lots of late nights and early mornings studying before I go to work, and then an hour or two of work before my head hits the pillow. Still, nice to work on one problem instead of several, so I can focus. I’ve been using the pomodoro method to stay focused and take breaks to keep my brain fresh. This is the most I’ve had to use my brain in while, and I turn off all the distractions (except my new company email, can’t escape the leash!).

Got to get some rest before I attack my DynamoDB problem, my first!