CodeCommit : example

This is a visualizer snapshot of the main repo my current organization uses to organize their code, showing the recent commits and branch merges.

lambda : python3 : log filter and transport serverless function

This lambda function is deployed via cloud formation template. It is composed of the following parts:

  • A yaml config file that is sourced from an s3 bucket at time of execution. It includes instructions on how to build job rules to extend the function’s capacity.
  • A library of filtering, parsing, and messaging functions
  • A simple lambda_function handler that is triggered by SQS messages posted when a log file is posted to an s3 bucket.
  • cloudformation template

 The function handles 10 messages at a time, filters out files we don’t want, identifies file types we do want, wraps them as syslog messages, then compresses the file and posts it to a bucket, which will get pulled on premise by a local worker script.

The yaml config file

# This document is an abstraction of the pattern match and job function logic of the aws_logs_filter.py lambda function.
#
# The 'job' rules define how the aws-logs-filter.py function will parse logs.
#
# Each job is a log type that is received from AWS accounts that have implemented log aggregation and forwarding
#
# Each log is tested for gzip compression, unzipped if necessary, and loaded into memory. All files are treated as
# plain text files for the purposes of job definition.
#
# The data structure is as follows:
# job##: a simple name for the rule. Must be unique, but not necessarily sequential
# name: a plain language name for the job that is not processed by the function
# match_variable: a pattern that the function will use to identify the log
# service_variable: a service that will be added to the syslog message to determine the log type.
# action variables: an ordered list of steps the function will take to process the log.
#
# The callable methods work as follows:
#   json: treats the file as a json object on each line
#   firehose: json objects on a single line. The method looks for touching braces "}{" and breaks them into lines "}\n{"
#   text_wrap: handles text logs regardless of form, wraps them as a syslog message
#
# all logs that meet the job rules get parsed, gzipped and sent to the filtered s3 bucket.
# all logs that fail to get parsed are written to the region's DynamoDB table, AWSLogsFilterExceptions
#
# Periodic review of the table will show logs that are not being processed, and a new set of job rules can be
# added to this file. The configuration will be updated on the next execution of the lambda function.
#
# Periodic review of the dead letter queue will also be useful to identify logs that are not being processed, but these
# be a rare exception. We will need to ascertain why the log was not written to the database table.
#
# This file is kept in the s4://<redacted> bucket, which supports versioning. If a change is made that
# breaks the lambda functions, you can restore the previous version using the AWS S3 management console.
#
jobs:
    job01 :
        name: cloudtrail
        match_variable: cloudtrail/
        service_variable: 'cloudtrail.amazonaws.com'
        action_variables:
           - json

    job02 :
        name: rds
        match_variable: rds/
        service_variable: 'rds.amazonaws.com'
        action_variables:
           - text_wrap

    job03 :
        name: firehose
        match_variable: firehose/
        service_variable: 'cloudwatch.amazonaws.com'
        action_variables:
           - firehose
           - json

    job04 :
        name: config
        match_variable: config/
        service_variable: 'config.amazonaws.com'
        action_variables:
           - json

    job05 :
        name: flowlog
        match_variable: flowlog/
        service_variable: 'vpc.amazonaws.com'
        action_variables:
           - text_wrap

    job06 :
        name: elb
        match_variable: elasticloadbalancing/
        service_variable: 'elb.amazonaws.com'
        action_variables:
           - text_wrap

    job07 :
        name: serveraccesslogs
        match_variable: serveraccesslogs/
        service_variable: 'ec2.amazonaws.com'
        action_variables:
           - text_wrap

    job08 :
        name: mq
        match_variable: mq/
        service_variable: 'mq.amazonaws.com'
        action_variables:
           - text_wrap

    job09 :
        name: redshift
        match_variable: redshift/
        service_variable: 'redshift.amazonaws.com'
        action_variables:
           - text_wrap

    job10 :
        name: cloudfront
        match_variable: cloudfront/
        service_variable: 'cloudfront.amazonaws.com'
        action_variables:
           - text_wrap

    job11 :
        name: ec2
        match_variable: ec2/
        service_variable: 'ec2.amazonaws.com'
        action_variables:
           - json

    job12 :
        name: emr
        match_variable: elasticmapreduce/
        service_variable: 'elasticmapreduce.amazonaws.com'
        action_variables:
           - text_wrap

    job13 :
        name: elb2
        match_variable: elb/
        service_variable: 'elb.amazonaws.com'
        action_variables:
           - text_wrap

    job14 :
        name: etl
        match_variable: etl/
        service_variable: 'glue.amazonaws.com'
        action_variables:
           - text_wrap

    job15 :
        name: s3
        match_variable: s3/
        service_variable: 's3.amazonaws.com'
        action_variables:
           - text_wrap

    job16:
        name: aurora
        match_variable: aurora/
        service_variable: 'rds.amazonaws.com'
        action_variables:
        - text_wrap

# filter variable. These are key path patterns that we drop to avoid redundant logs and files that lock up the parser
FILTERS:
    - CloudTrail-Digest
    - ConfigWritabilityCheckFile
    - DS_Store
    - .trm
    - .trc
    - <example key path>
    - <rest removed for privacy>

# aggregate_accounts variable for those that combine their logs in one account before replication. Note in key path where
# actual account number shows, split by '/'
aggregate_accounts:
    agg_acct01:
        name: <redacted1>
        agg_account: '<example1>'
        match_position: '1'
    agg_acct02:
        name: <redacted2>
        agg_account: '<redacted2>'
        match_position: '2'
    agg_acct03:
        name: <redacted3>
        agg_account: '<redacted3>'
        match_position: '2'

The Library File

#!/usr/local/bin/python3
"""
DESCRIPTION
Lambda function to process sqs events, filter messages based on key paths,
load filtered messages into memory,parse formats to syslog,
and write syslog messages to a gzipped s3 object, and drop the event generating sqs message.
"""

import re
import ast
import gzip
import json
from datetime import datetime
import urllib
import boto3
import botocore
import os
import yaml
import math
import uuid


REGION = os.environ.get('AWS_REGION')


# setup vars
config_bucket = '<redacted>'
yaml_config = 'config.yaml'
s3 = boto3.client('s3')
config_handle = s3.get_object(Bucket=config_bucket, Key=yaml_config)
config_data = body = config_handle.get('Body').read()
yaml_data = yaml.load(config_data)
FILTERS = yaml_data['FILTERS']
aggregate_accounts = yaml_data['aggregate_accounts']


class Message:
    """ Pull objects and variables plus event data. """

    def __init__(self, sqs_object, record, in_queue, out_queue):
        """ receives a single record from event. """
        self.sqs = sqs_object
        self.message_id = record["messageId"]
        self.receipt_handle = record["receiptHandle"]
        # not in testing
        body = ast.literal_eval(record["body"])
        self.msg_key_path = body["Records"][0]["s3"]["object"]["key"]
        # self.msg_key_path = record["body"]["Records"]["s3"]["object"]["key"]
        self.in_queue = in_queue
        self.out_queue = out_queue
        self.tag = 'message: initialized'
        #print('key: ', self.msg_key_path)

    def filter_message(self):
        """
        filters keypath against list to drop files we don't want
        """
        return_value = ''
        global FILTERS
        for my_filter in FILTERS:
            if re.search(my_filter, self.msg_key_path):
                #print('matched: ', self.msg_key_path)
                self.drop_message()
                self.tag = 'message: filtered'
                return_value = True
        if return_value:
            return True
        else:
            #print('file not matched: ', self.msg_key_path)
            return False

    def drop_message(self):
        """
        removes sqs msg from queue after successful file processing
        """
        try:
            self.sqs.delete_message(
                QueueUrl=self.in_queue,
                ReceiptHandle=self.receipt_handle
            )
            return True

        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not drop message:', exception_string)
            return False

    def write_sqs_message_to_out_queue(self, sqs_object, out_queue, data):
        """
        Writes a message to specified / dlq queue. Have to replace single quotes to meet json formatting
        """

        try:
            replace_single_quotes = re.sub('\'', '\"', data)
            sqs = sqs_object
            sqs.send_message(
                QueueUrl=out_queue,
                MessageBody=replace_single_quotes,
            )
            return True

        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not write message to out_queue:', exception_string)
            return False


class File:

    """ Processes s3 object from msg_key_path. """
    def __init__(self, s3_object, msg_key_path, in_bucket, out_bucket):
        self.s3 = s3_object
        self.msg_key_path = msg_key_path
        self.in_bucket = in_bucket
        self.out_bucket = out_bucket
        self.service_name = ''
        self.owner_id = ''
        self.tag = 'file: initialized'

# check these
    def parse_body_json_to_syslog(self):
        output_data = ''
        accountID = 'account-' + self.owner_id
        my_time = datetime.now().strftime('%Y-%m-%dT%H:%M:%S%Z')
        service_name = self.service_name
        my_data = str(self.body).split('\n')
        self.datetime = my_time

        for data in my_data:
            my_data = json.loads(data)
            message = my_time + ' ' + str(accountID) + ' ' + str(service_name) + ' ' + str(my_data) + '\n'
            output_data += message
        self.body = output_data
        return True

    def determine_if_agg_account(self):
        for key in aggregate_accounts.keys():
            match = re.search(self.msg_key_path, aggregate_accounts[key]['agg_account'])
            split_path = self.msg_key_path.split('/')

            if match:
                self.owner_id = split_path[int(aggregate_accounts[key]['match_position'])]
            else:
                self.owner_id = split_path[0]
        return self.owner_id

    def determine_service_name(self, service_name):
        self.service_name = service_name
        return True

    def fix_body_firehose(self):
        data = re.sub('}{', '}\n{', self.body)
        self.tag = 'body: each line json'
        self.body = data
        return True

    def parse_ascii_body_to_syslog(self):
        output_data = ''
        accountID = 'account-' + self.owner_id
        my_time = datetime.now().strftime('%Y-%m-%dT%H:%M:%S%Z')
        service_name = self.service_name
        self.datetime = my_time
        output_data = my_time + ' ' + str(accountID) + ' ' + str(service_name) + ' ' + str(self.body)
        self.body = output_data
        return True

# these are reliable
    def load_s3_object_to_body(self):
        """
       Loads the file's contents to memory for manipulation
        """
        self.msg_key_path = urllib.parse.unquote(self.msg_key_path)
        file_handle = self.s3.get_object(Bucket=self.in_bucket, Key=self.msg_key_path)
        try:
            self.body = file_handle.get('Body').read()
            self.tag = 'body: loaded'
            return True
        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not load file from path: ', self.msg_key_path, ' ', exception_string)
            return False

    def put_s3_object(self):
        """ Write loaded data to filtered bucket. """
        try:
            self.s3.put_object(Bucket=self.out_bucket, Key=self.msg_key_path, Body=self.body)
            return True
        except:
            print('could not put: ', self.msg_key_path)
            return False

    def check_body_is_gzip(self):
        try:
            test_gzip = gzip.decompress(self.body)
            self.tag = 'body: gzip'
            return True
        except:
            return False

    def check_body_is_ascii(self):
        try:
            data = self.body.decode()
            test_ascii = isinstance(data, data)
            if test_ascii:
                self.tag = 'body: ascii'
                return True
        except:
            return False

    def gunzip_body(self):
        data = gzip.decompress(self.body)
        self.body = data.decode('utf-8')
        self.tag = 'body: unzipped'
        return True

    def gzip_body(self):
        data = self.body
        encoded_data = data.encode('utf-8')
        self.body = gzip.compress(encoded_data)
        self.tag = 'body: compressed to gz'
        return True

    def add_syslog_suffix_to_path(self):
        self.msg_key_path = self.msg_key_path + '.syslog'
        self.tag = 'file: .syslog'
        return True

    def add_gz_suffix_to_path(self):
        self.tag = 'file: .gz'
        self.msg_key_path = self.msg_key_path + '.gz'
        return True

    def strip_path_of_suffix(self):
        try:
            self.msg_key_path = re.sub('.gz', '', self.msg_key_path)
            self.msg_key_path = re.sub('.json', '', self.msg_key_path)
            self.msg_key_path = re.sub('.syslog', '', self.msg_key_path)
            return True
        except:
            return False

    def buffer_msg_size(self):
        content_length = len(self.body)
        content = self.body.split('\n')
        owner_id = self.owner_id
        service_name = self.service_name
        my_timestamp = self.datetime
        main_body = ''

        for line in content:
            line_length = len(line)
            if line_length < 8000:
                main_body += line + "\n\n"
            else:
                my_uuid = self.my_random_sequence()
                part_count = math.ceil(line_length / 8000)
                print('part_count: ', part_count)
                for x in range(0, part_count):
                    if x == part_count:
                        submessage = line[(x * 8000):0]
                        submessage = str(my_timestamp) + " account-" + str(owner_id) + " " + service_name + " " + submessage + " sequence " + str(my_uuid) + " part " + str(x + 1) + " of " + str(part_count) + "\n\n"
                        main_body += submessage
                    else:
                        submessage = line[(x * 8000):((x + 1) * 8000)]
                        submessage = str(my_timestamp) + " account-" + str(owner_id) + " " + service_name + " " + submessage + " sequence " + str(my_uuid) + " part " + str(x + 1) + " of " + str(part_count) + "\n\n"
                        main_body += submessage
        self.body = main_body
        try:
            self.transport_syslog_body_to_gz_s3()
        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not write message to out_queue:', exception_string)

    def parse_exception(self):
        # can let message process 4 times and fail to dlq, or copy directly to dlq, &/or write to ddb table
        line_split = self.msg_key_path.split("/")
        accountID = line_split[0]
        now = datetime.now().today().strftime("%Y-%m-%dT%H:%M:%S:%f")
        ddb = boto3.client('dynamodb')
        table_name = 'AwsLogsFilterExceptions'
        if self.check_body_is_gzip() == True:
            self.gunzip_body()
        # limiting body message to < 400kb, max limit of ddb entry
        self.body = str(self.body[0:3000])
        item = {
            'accountID': {'S': accountID},
            'eventDateTime': {'S': now},
            'region': {'S': REGION},
            'msg_key_path': {'S': self.msg_key_path},
            'tag': {'S': self.tag},
            'body_data': {'S': self.body},
        }

        try:
            ddb.put_item(
                TableName=table_name,
                Item=item
            )
            return True
        except botocore.exceptions.ClientError as error:
            print('can not write to ddb' + 'error: {0}'.format(error))
            return False

    def my_random_sequence(self,string_length=5):
        sequence = str(uuid.uuid4()).upper()
        return sequence[0:string_length]

    def transport_syslog_body_to_gz_s3(self):
        self.add_syslog_suffix_to_path()
        self.gzip_body()
        self.add_gz_suffix_to_path()
        self.put_s3_object()
        return True

The Lambda Handler

def lambda_handler(event, context):
    """
    Flow control for aws-logs-filter.
    """
    import aws_logs_filter
    import boto3
    import botocore
    import re
    import os
    import yaml


    # setup vars
    config_bucket = '<redacted>'
    yaml_config = 'config.yaml'
    s3 = boto3.client('s3')
    config_handle = s3.get_object(Bucket=config_bucket, Key=yaml_config)
    config_data = config_handle.get('Body').read()
    yaml_data = yaml.load(config_data)

    REGION = os.environ.get('AWS_REGION')
    FILTERS = yaml_data['FILTERS'] # TODO see if this use of FILTERS is necessary, called as function in aws_logs_filter
    jobs = yaml_data['jobs']
    aggregate_accounts = yaml_data['aggregate_accounts']

    in_bucket = '<redacted>' + REGION
    out_bucket = '<redacted>-filtered-' + REGION

    sqs = boto3.client('sqs')
    in_queue = 'https://sqs.' + REGION + '.amazonaws.com/<redacted>/Log-Aggregation-Queue'
    out_queue = 'https://sqs.' + REGION + '.amazonaws.com/<redacted>/Log-Aggregation-Dead-Letter-Queue'

    ddb = boto3.client('dynamodb')
    table_name = 'AwsLogsFilterExceptions'

    sent_flag = 'no'

    for record in event['Records']:

        try:
            message = aws_logs_filter.Message(sqs, record, in_queue, out_queue)
            # print(message.tag)
        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not retrieve message:', exception_string)

        try:
            filtered_result = message.filter_message()
            # print(message.tag)
        except botocore.exceptions.ClientError as error:
            exception_string = 'Raised exception: error: {0}'.format(error) + '\n'
            print('can not filter:', exception_string)

        # simple filter function, mostly relevant to us-west-2, but some global entries
        if filtered_result is True:
            print('dropped message: ', message.msg_key_path)
            message.drop_message()

        # main processing flow control
        if filtered_result is False:
            print('loading message:', message.msg_key_path)
            try:
                file = aws_logs_filter.File(s3, message.msg_key_path, in_bucket, out_bucket)
                file.load_s3_object_to_body()
            except:
                print('could not load file into memory')
                break
                # could result in dlq messages from unprocessed from break.
                # TODO fix so this fails gracefully

            try:
                if file.check_body_is_gzip() == True:
                    file.gunzip_body()
                    file.strip_path_of_suffix()
                elif file.check_body_is_gzip() == False:
                    file.check_body_is_ascii()

            except:
                print('file is unparsable: ', file.msg_key_path)
                file.parse_exception()

            for key in jobs.keys():
                file.determine_if_agg_account()
                #print('account id: ',file.owner_id)
                match_variable = jobs[key]['match_variable']
                action_variables = jobs[key]['action_variables']
                service_variable = jobs[key]['service_variable']
                # TODO make sure ownerID variables are in place


                match = re.search(match_variable, message.msg_key_path, re.IGNORECASE)

                if match:
                    file.determine_service_name(service_variable)
                    print("service: ", file.service_name)
                    action_count = len(action_variables)
                    #print('action_count: ', action_count)
                    for i in range(0, action_count):
                        #print(action_variables[i])

                        if action_variables[i] == 'firehose':
                            file.strip_path_of_suffix()
                            file.fix_body_firehose()

                        elif action_variables[i] == 'json':
                            file.strip_path_of_suffix()
                            file.parse_body_json_to_syslog()

                        elif action_variables[i] == 'text_wrap':
                            file.parse_ascii_body_to_syslog()
                        #print('end of action #', i)
                    file.buffer_msg_size()
                    sent_flag = "yes"

            if sent_flag == "yes":
                print('sent: ', sent_flag)
            else:
                print('write to ddb as error')
                result = file.parse_exception()
                if result is True:
                    message.drop_message()
                # if false, no db entry, message goes to dead letter for further processing

Cloudformation template

AWSTemplateFormatVersion: 2010-09-09
Description: >-
    Creates an instance of aws_logs_filter.py for filtering, wrapping and transporting
    log files so they can be pulled on-premise for further processing.

Parameters:
    LambdaCodeBucket:
        Description: The bucket where you uploaded the zip file
        Type: String
    LambdaCodeFile:
        Description: the name of the zip file you uploaded
        Type: String
    FunctionInstanceName:
        Description: Name of this instance of the function (must be unique).
        Type: String
Resources:
    AWSLogsFilter:
        Type: AWS::Lambda::Function
        Properties:
            Code:
                S3Bucket:
                    Ref: LambdaCodeBucket
                S3Key:
                    Ref: LambdaCodeFile
            FunctionName:
                Ref: FunctionInstanceName
            Handler: lambda_function.py
            MemorySize: 512
            Role: arn:aws:iam::<redacted>:role/lambda_filter_log_agg_sqs_queue
            Timeout: 900
            Description: >-
                pulls log file, wraps as syslog, segments and sequences long messages
                and puts compressed file to filter bucket
            Runtime: python3.6
    LambdaFunctionEventSourceMapping:
        Type: AWS::Lambda::EventSourceMapping
        Properties:
            BatchSize: 10
            Enabled: true
            #change above to true for production
            EventSourceArn:
                Fn::Join:
                    - ''
                    - 
                      - 'arn:aws:sqs:'
                      - Ref: 'AWS::Region'
                      - ':<redacted>:Log-Aggregation-Queue'
            FunctionName: 
                Ref: AWSLogsFilter           

lambda : python3 : backupRotation

This is a simple function that rotates the backup files in an s3 bucket to maintain a 30 day archive. Some variables have obfuscated.

!/usr/bin/python3
import boto3
from datetime import datetime, timedelta
today_date = str(datetime.now().today().strftime("%y%m%d"))
one_month_delta = datetime.now().today() - timedelta(days=31)
one_month_date = str(one_month_delta.strftime("%y%m%d"))
s3 = boto3.client('s3')
my_bucket = 'mybucket'
object_list = s3.list_objects(Bucket = my_bucket)
file_list = []
for object in object_list['Contents']:
    file_list.append(object['Key'])
for file in file_list:
    if one_month_date in file:
        s3.delete_object(Bucket= my_bucket, Key = file)

cloudformation : yaml : simple load balancer

We are in the process of rebuilding our instrastructure stack as code from the original configurations that were all hand crafted in the console. The first and easiest was the student web server load balancer. Some variables have been changed to avoid tampering.

Resources:
  studentServerELB:
    Type: AWS::ElasticLoadBalancing::LoadBalancer
    Properties:
AvailabilityZones:
- "us-west-2a"
- "us-west-2b"
- "us-west-2c"
  Instances:
     - i-myinstance
  Listeners:
  - LoadBalancerPort: '80'
    InstancePort: '80'
    Protocol: HTTP
  HealthCheck:
    Target: HTTP:80/healthy.html
    HealthyThreshold: '2'
    UnhealthyThreshold: '5'
    Interval: '10'
    Timeout: '5'
  Subnets:
    - subnet-1
    - subnet-2
    - subnet-3
  SecurityGroups:
    - sg-mysecgroup

terrafom: managing dns

I use terraform and CodeCommit to manage our dns changes for each of my current organization’s zones. The specific variables are changed for publishing, and I have edited for brevity by showing a single example of each record type. This has significantly simplified our dns management processes. We use an s3 bucket to maintain state, so each of the site techs can make changes without stepping on each other.

/*  180129 - soops@ucla.edu
    Deploy myzone.org for Route53
    */
sets the region
provider "aws" {
    region = "us-west-2"
}
makes state document an s3 object
terraform {
  backend "s3" {
    bucket  = "myzone.terraform.state"
    encrypt = true
    key     = "global/dns/myzone.org.state"
    region  = "us-west-2"
  }
}
variable "ttl" {
    default = "60" # for rapid propagation of changes
}
variable "zone_id" {
    default = "myzoneid"
}

resource "aws_route53_record" "apex" {
    zone_id = "myzoneid"
    name = "myzone.org"
    type = "A"
    ttl = "${var.ttl}"
    records = ["myip"]
    }
resource "aws_route53_record" "print" {
    zone_id = "${var.zone_id}"
    name = "print.myzone.org"
    type = "CNAME"
    ttl = "${var.ttl}"
    records = ["print.otherzone.org"]
}

terraform : instantiating a webserver

This script is an example of applying infrastructure as code principles to managing cloud servers. This script code be used as a step to pre-baking an AMI for production use, however it is currently used to rebuild the server as needed, rather than interactively patching one.


/*  180228 - soops@ucla.edu
    Deploy new web server
    into prodVPC 
    for testing 
    */

# sets the region
provider "aws" {
    region = "us-west-2"
}

#variables changed for publishing
resource "aws_instance" "webServerDev" {
    ami = "ami-standardAMI" 
    instance_type = "m4.large" # pubProd-b
    subnet_id = "subnet-mysubnet"
    vpc_security_group_ids = ["sg-mysecgroup"]
    associate_public_ip_address = true
    key_name = "mykey"
    iam_instance_profile = "myrole"

    ebs_block_device {
        device_name             = "/dev/xvda"
        delete_on_termination   = true
        volume_type             = "gp2"
        #volume_type            ="io1"
        volume_size             = "20"
#        iops                    = "1000"
    }

    volume_tags = {
        Name = "webServerDev"
    }
    # bootstraps the server
    user_data = <<-EOF
        #!/bin/bash
        # add apache 2.2
        sudo yum install httpd -y
        sudo mkdir -p /tmp/php7
        cd /tmp/php70
        # abb webstatic and load modules
        wget https://mirror.webtatic.com/yum/el6/latest.rpm
        sudo yum install latest.rpm -y
        sudo yum install --enablerepo=webtatic php70w -y
        sudo yum install php70w-opcache php70w-xml php70w-ldap php70w-pdo php70w-mysqlnd php70w-gd php70w-pecl-apcu php70w-mbstring php70w-imap geoip mod_geoip -y
        # sudo echo "<?php echo phpinfo(); ?>" > /var/www/html/index.php
        sudo yum install openldap-clients -y
        sudo yum update -y
        cd /tmp
        wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
        sudo chmod +x /tmp/awslogs-agent-setup.py
        sudo /tmp/awslogs-agent-setup.py --region us-west-2 --non-interactive --configfile s3://mys3bucket/awslogs/awslogs.conf
        sudo mkdir /etc/awslogs
        sudo aws s3 cp s3://mys3bucket/webServer/etc/awslogs/awslogs.conf /etc/awslogs/.
        sudo service awslogs stop
        sudo service awslogs start
        cd ~
        # adds prebuilt v-hosts.conf
        sudo aws s3 cp s3://mys3bucket/webServer/etc/httpd/conf.d/v-hosts.conf /etc/httpd/conf.d/v-hosts.conf
        sudo aws s3 cp s3://mys3bucket/webServer/etc/httpd/conf.d/geoip.conf /etc/httpd/conf.d/geoip.conf
       
        sudo sed -i "s/DirectoryIndex\ index.html\ index.html.var/DirectoryIndex\ index.html\ index.php/g" /etc/httpd/conf/httpd.conf
        #set NameVirtualHost to work in httpd.conf
        sudo sed -i "s/\#NameVirtualHost\ \*\:80/NameVirtualHost\ \*\:80/g" /etc/httpd/conf/httpd.conf
        sudo sed -i "s/memory_limit\ =\ 128M/memory_limit\ =\ 256M/g" /etc/php.ini
        sudo sed -i "s/upload_max_filesize\ =\ 2M/upload_max_filesize\ =\ 10M/g" /etc/php.ini
        #load the /home/www directory
        sudo mkdir /home/www
        #sudo aws s3 cp --recursive s3://mys3bucket/webServer/wwwSync /home/www/
        sudo chown -R apache:apache /home/www
        sudo service httpd start
        # adding dump scripts, ldap & mysql dependencies
        # need to make sure security group access for ldap and db
        mkdir /home/ec2-user/scripts
        echo "aws s3 sync /etc s3://mys3bucket/webServer/etc/; aws s3 sync /home s3://mys3bucket/webServer/home/" > /home/ec2-user/scripts/awsSync.sh
        sudo chmod +x /home/ec2-user/awsSync.sh
        echo "aws s3 sync /var/log s3://mys3bucket/webServer/" > /home/ec2-user/scripts/logSync.sh
        sudo chmod +x /home/ec2-user/logSync.sh
        sudo chmod 777 /etc/crontab
        sudo echo "*/5 * * * * root /home/ec2-user/scripts/awsSync.sh" >> /etc/crontab
        sudo echo "*/5 * * * * root /home/ec2-user/scripts/logSync.sh" >> /etc/crontab
        sudo chmod 644 /etc/crontab
        # loading postfix, killing sendmail, starting servics
        sudo chkconfig sendmail off
        sudo service sendmail stop
        sudo yum install postfix -y
        sudo aws s3 cp s3://mys3bucket/webServer/etc/postfix/main.cf /etc/postfix/main.cf
        sudo chkconfig postfix on
        sudo service postfix start
        sudo chkconfig httpd on
        sudo service httpd start
        EOF
    



# names the instance
    tags {
        Name = "webServerTesting"
    }

}

# attach to elb
resource "aws_elb_attachment" "webServerAttachment" {
    elb = "webServer"
    instance = "{aws_instance.webServerDev.id}"
}

# sets dns A record
resource "aws_route53_record" "testServer" {
    zone_id = "myzoneid"
    name = "testwebserver.mydomain"
    type = "A"
    ttl = "300"
    records = ["${aws_instance.webServerDev.public_ip}"]
}


# sends public ip to terminal STDOUT
output "public ip" {
    value = "${aws_instance.webServerDev.public_ip}"
}

python: boto3 : multi-threading

This script takes a list of AWS account numbers and scans each one for deployed AWS services, then writes the data to a table. The icstools.py is the library outlined here. This is my first multi-threaded script, which cut down the task of scanning 275 AWS accounts from 4 hours to 12 minutes.

!/usr/local/bin/python3
""" flow control for AWS account service interrogation, using multithreading to speed up the process. """
from datetime import datetime
from multiprocessing import Process
import time
import boto3
import icstools
tempDateTime = datetime.now().strftime("%Y-%m-%d")
dynamodb = boto3.client('dynamodb')
tablename = 'awsAccountAudit' + '-' + str(tempDateTime)
table = icstools.createTable(dynamodb=dynamodb, tablename=tablename)
time.sleep(30) # give time for table to instantiate
open removed-for-privacy/has-readOnly-role.txt as list
s3 = boto3.resource('s3')
obj = s3.Object('removed-for-privacy', 'has-readOnly-role.txt')
body = obj.get()['Body'].read()
hasRoleList = body.decode().split('\n')

def accountIterate(session, id):
    question = icstools.AwsAccount(session, id)
    try:
        result = question.ec2Services()
        icstools.writeDB(dynamodb, tablename, result)
    except:
        print('no ec2 results in ', id)

--- truncated for brevity. each service in library is called and output written to table

# multiprocess array
procs = []
for newID in hasRoleList:
    time.sleep(1)
    session = icstools.assumeRoleSession(newID)
    # create processes for each account iteration
    proc = Process(target=accountIterate, args=(session, newID))
    procs.append(proc)
    proc.start()
run each proc in list of procs
for proc in procs:
    proc.join()

python: boto3: interrogating AWS services library

#!/usr/local/bin/python3
""" Class and methods for interrogating AWS accounts for deployed services. """

import boto3
import botocore
from random import randint
from datetime import datetime, timedelta
import collections
from pprint import pprint


def nested_dict_iter(nested):
    for key, value in nested.items():
        if isinstance(value, collections.Mapping):
            yield from nested_dict_iter(value)
        else:
            yield key, value


def printException(exceptionString: str, exceptionWriteString: str) -> set:
    """Write to variable for later s3 put."""
    exceptionWriteString += exceptionString + '\n'

    return exceptionWriteString


def assumeRoleSession(idNumber: str) -> set:
    """Create session with the assumed read only security role in the idNumber account."""
    region = 'us-west-2'

    client = boto3.client('sts')
    role_arn = "arn:aws:iam::" + idNumber + ":role/Security-Readonly"
    # Use the assertion to get an AWS STS token using Assume Role with SAML
    client.get_session_token()
    token = client.assume_role(RoleArn=role_arn,
                               RoleSessionName="AssumeRoleSession")

    creds = token['Credentials']
    session = boto3.Session(region_name=region,
                            aws_access_key_id=creds['AccessKeyId'],
                            aws_secret_access_key=creds['SecretAccessKey'],
                            aws_session_token=creds['SessionToken'])

    return session


def createTable(dynamodb, tablename):
    """ Initializes daily table to write entries. """
    table = dynamodb.create_table(
        TableName=tablename,
        KeySchema=[
            {
                'AttributeName': 'accountID',
                'KeyType': 'HASH'
            },
            {
                'AttributeName': 'datetimekey',
                'KeyType': 'RANGE'
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'accountID',
                'AttributeType': 'S'
            },
            {
                'AttributeName': 'datetimekey',
                'AttributeType': 'S'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 10,
            'WriteCapacityUnits': 150,

        }
    )
    return table


def destroyTable(dynamodb, tablename):
    table = dynamodb.delete_table(
        TableName=tablename
    )
    return True


def destroyWeekOldTable(dynamodb):
    currentDate = datetime.now().today()
    lastWeek = currentDate - timedelta(days=7)
    thisWeekDB = 'awsAccountAudit-' + str(currentDate.strftime("%Y-%m-%d"))
    lastWeekDB = 'awsAccountAudit-' + str(lastWeek.strftime("%Y-%m-%d"))
    listDbs = dynamodb.list_tables()
    # print('lastWeekDB: ', lastWeekDB)
    # pprint(listDbs)
    if lastWeekDB in listDbs['TableNames']:
        try:
            result = dynamodb.delete_table(TableName=lastWeekDB)
            return result
        except:
            print('unable to delete table: ', lastWeekDB)
            return False


def downShiftTable(dynamodb, tablename):
    """ Reduces read / write capacity after the table is completed, to save costs. """
    response = dynamodb.update_table(
        TableName=tablename,
        ProvisionedThroughput={
            'ReadCapacityUnits': 10,
            'WriteCapacityUnits': 10,
        },
    )


def writeDB(dynamodb, tablename, result):
    tempDateTime = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    for key in result:
        nestedResult = nested_dict_iter(result[key])
        for key1, value1 in nestedResult:
            myDateTimeKey = tempDateTime + str(randint(0000, 9999))
            if key1 == 'service':
                service = value1
            if key1 == 'region':
                region = value1
            if key1 == 'resourceID':
                resourceID = value1
            if key1 == 'accountID':
                accountID = value1
            if key1 == 'accountID':
                accountID = value1
                resourceData = str(result[key].get('resourceData'))  # works as string 'S'
        item = {
            'accountID': {'S': accountID},
            'datetimekey': {'S': myDateTimeKey},
            'dateandtime:': {'S': tempDateTime},
            'service': {'S': service},
            'region': {'S': region},
            'resourceID': {'S': resourceID},
            'resourceData': {'S': resourceData}
        }
        dynamodb.put_item(
            TableName=tablename,
            Item=item
        )


def getRegions(session: object) -> set:
    """Get list of regions from one account and set as global list."""
    ec2 = session.client('ec2')
    regions = ec2.describe_regions()
    for region in regions['Regions']:
        # pprint(region)
        regions = [region['RegionName'] for region in ec2.describe_regions()['Regions']]
        global count
        count = len(regions)
    return regions


class AwsAccount:
    """ Initialize an account object to query available services. """

    def __init__(self, session, id):
        self.session = session
        self.id = id


    def returnSession(self):
        return self.session


    def returnID(self):
        return self.id


    # functions that return region specific results ----------------------------------


    def ec2Services(self):
        """Get list of all deployed EC2 instances, including stopped."""
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        regionsLength = len(regions)
        # print('regionsLength: ',regionsLength)
        for x in range(0, len(regions)):
            try:
                region_name = regions[x]
                # print('x = ', x)
                ec2 = session.client('ec2', region_name=region_name)
                # print('tested region: ', region_name)
                instances = ec2.describe_instances()
                for key in instances['Reservations']:
                    for instance in key['Instances']:
                        # print('tested instance in: ', region_name)
                        results[instance['InstanceId']] = {
                            'accountID': newID,
                            'resourceID': instance['InstanceId'],
                            'region': region_name,
                            'service': 'ec2.amazonaws.com',
                            'resourceData': instance
                        }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
            x += 1
        return results

    def publicIps(self):
        """Get list of all assigned public IPs"""
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        regionsLength = len(regions)
        # print('regionsLength: ',regionsLength)
        for x in range(0, len(regions)):
            try:
                region_name = regions[x]
                # print('x = ', x)
                ec2 = session.client('ec2', region_name=region_name)
                response = ec2.describe_network_interfaces(Filters=[{'Name':'addresses.association.public-ip', 'Values':['*']}])
                for interface in response['NetworkInterfaces']:
                    results[interface['Association']['PublicIp']] = {
                        'accountID': newID,
                        'resourceID': interface['Association']['PublicIp'],
                        'region': region_name,
                        'service': 'ec2.amazonaws.com',
                        'resourceData': interface
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
            x += 1
        return results


    def vpcServices(self):
        """Get list of all VPCs in all accounts."""
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            # print(region_name)
            ec2 = session.client('ec2', region_name=region_name)

            vpc = ec2.describe_vpcs()
            try:
                for vpc in vpc['Vpcs']:
                    results[vpc['VpcId']] = {
                        'accountID': newID,
                        'resourceID': vpc['VpcId'],
                        'region': region_name,
                        'service': 'vpc.amazonaws.com',
                        'resourceData': vpc
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def elbServices(self):
        """ Method to list all deployed load balancers. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            # print(region_name)
            elb = session.client('elb', region_name=region_name)
            response = elb.describe_load_balancers()
            try:
                for x in response['LoadBalancerDescriptions']:
                    results[x['LoadBalancerName']] = {
                        'accountID': newID,
                        'resourceID': x['LoadBalancerName'],
                        'region': region_name,
                        'service': 'elb.amazonaws.com',
                        'resourceData': response['LoadBalancerDescriptions'],
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def configServices(self):
        """ Pull config rule names for all regions. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            config = session.client('config', region_name=region_name)
            try:
                response = config.describe_compliance_by_config_rule()
                for item in response['ComplianceByConfigRules']:
                    results[item['ConfigRuleName']] = {
                        'accountID': newID,
                        'resourceID': item['ConfigRuleName'],
                        'region': region_name,
                        'service': 'config.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def snsServices(self):
        """ List all sns topics in all regions. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            sns = session.client('sns', region_name=region_name)
            try:
                response = sns.list_topics()
                # print('fetching sns topics for acct: ', newID)
                for item in response['Topics']:
                    results[item['TopicArn']] = {
                        'accountID': newID,
                        'resourceID': item['TopicArn'],
                        'region': region_name,
                        'service': 'sns.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)

        return results


    def codedeployServices(self):
        """ List all applications in all regions. Useful for identifying necessary app logs. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for x in range(0, len(regions)):
            try:
                region_name = regions[x]
                codedeploy = session.client('codedeploy', region_name=region_name)
                response = codedeploy.list_applications()
                # pprint(response)

                for item in response['applications']:
                    results[item] = {
                        'accountID': newID,
                        'resourceID': item,
                        'region': region_name,
                        'service': 'codedeploy.amazonaws.com',
                        'resourceData': 'null',  # response['applications'][item].get('resourceData'),
                    }
                # print(results)

            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def cloudfrontServices(self):
        """ List all cloudfront distributions. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            cloudfront = session.client('cloudfront', region_name=region_name)
            try:
                response = cloudfront.list_distributions()
                #results = response
                for x in response['DistributionList']['Items']:
                    results[x['Id']] = {
                        'accountID': newID,
                        'resourceID': x['Id'],
                        'region': region_name,
                        'service': 'cloudfront.amazonaws.com',
                        'resourceData': x,
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def acmServices(self):
        """ List all certificates in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            acm = session.client('acm', region_name=region_name)
            try:
                response = acm.list_certificates()
                for item in response['CertificateSummaryList']:
                    results[item['CertificateArn']] = {
                        'accountID': newID,
                        'resourceID': item['CertificateArn'],
                        'region': region_name,
                        'service': 'acm.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def cloudtrailServices(self):
        """ List all cloudtrails in all regions. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            cloudtrail = session.client('cloudtrail', region_name=region_name)
            try:
                response = cloudtrail.describe_trails()
                for item in response['trailList']:
                    results[item['Name']] = {
                        'accountID': newID,
                        'resourceID': item['Name'],
                        'region': region_name,
                        'service': 'cloudtrail.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def elasticacheServices(self):
        """ List all cache clusters in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            elasticache = session.client('elasticache', region_name=region_name)
            try:
                response = elasticache.describe_cache_clusters()
                for x in response['CacheClusters']:
                    results[x['CacheClusterId']] = {
                        'accountID': newID,
                        'resourceID': x['CacheClusterId'],
                        'region': region_name,
                        'service': 'elasticache.amazonaws.com',
                        'resourceData': x,
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def kmsServices(self):
        """ List all customer encryption keys in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            kms = session.client('kms', region_name=region_name)
            try:
                response = kms.list_keys()
                for item in response['Keys']:
                    results[item['KeyId']] = {
                        'accountID': newID,
                        'resourceID': item['KeyId'],
                        'region': region_name,
                        'service': 'kms.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def lambdaServices(self):
        """ List all lambda functions in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            lambdaFunctions = session.client('lambda', region_name=region_name)
            try:
                response = lambdaFunctions.list_functions()
                for item in response['Functions']:
                    results[item['FunctionName']] = {
                        'accountID': newID,
                        'resourceID': item['FunctionName'],
                        'region': region_name,
                        'service': 'lambda.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def dynamodbServices(self):
        """ List all tables in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            ddb = session.client('dynamodb', region_name=region_name)
            response = ddb.list_tables()
            try:
                for x in response['TableNames']:
                    results[x] = {
                        'accountID': newID,
                        'resourceID': x,
                        'region': region_name,
                        'service': 'dynamodb.amazonaws.com',
                        'resourceData': x
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def efsServices(self):
        """ List all Elastice File Systems (NFS) in each region """
        session = self.session
        newID = self.id
        # overriding region due to limited offerings, avoid connection failure error
        regions = ['us-east-1','us-east-2','us-west-1','us-west-2','eu-west-1','eu-west-1','eu-central-1','ap-southeast-2','ap-northeast-2']
        results = {}
        for region_name in regions:
            efs = session.client('efs', region_name=region_name)
            try:
                response = efs.describe_file_systems()
                for x in response['FileSystems']:
                    # pprint(x)
                    results[x['Name']] = {
                        'accountID': newID,
                        'resourceID': x['Name'],
                        'region': region_name,
                        'service': 'elasticfilesystem.amazonaws.com',
                        'resourceData': x,
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def autoscalingServices(self):
        """ List all autoscaling groups in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            autoscaling = session.client('autoscaling', region_name=region_name)
            try:
                response = autoscaling.describe_auto_scaling_groups()
                for x in response['AutoScalingGroups']:
                    # pprint(x)
                    results[x['AutoScalingGroupName']] = {
                        'accountID': newID,
                        'resourceID': x['AutoScalingGroupName'],
                        'region': region_name,
                        'service': 'autoscaling.amazonaws.com',
                        'resourceData': x,
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def cloudformationServices(self):
        """ List all stacks loaded in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            cloudformation = session.client('cloudformation', region_name=region_name)
            try:
                response = cloudformation.describe_stacks(StackStatusFilter=['CREATE_COMPLETE','UPDATE_COMPLETE'])
                for item in response['Stacks']:
                    results[item['StackName']] = {
                        'accountID': newID,
                        'resourceID': item['StackName'],
                        'region': region_name,
                        'service': 'cloudformation.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def rdsServices(self):
        """ List all database instances in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            rds= session.client('rds', region_name=region_name)
            try:
                response = rds.describe_db_instances()
                for item in response['DBInstances']:
                    results[item['DBName']]={
                        'accountID': newID,
                        'resourceID': item['DBName'],
                        'region': region_name,
                        'service': 'rds.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def elbv2Services(self):
        """ List all v2 elbs in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            elbv2= session.client('elbv2', region_name=region_name)
            try:
                response = elbv2.describe_load_balancers()
                for item in response['LoadBalancers']:
                    results[item['LoadBalancerName']]={
                        'accountID': newID,
                        'resourceID': item['LoadBalancerName'],
                        'region': region_name,
                        'service': 'elbv2.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def ebsServices(self):
        """ List all elastic beanstalk environments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            ebs= session.client('ebs', region_name=region_name)
            try:
                response = ebs.describe_environments()
                for item in response['Environments']:
                    results[item['ApplicationName']]={
                        'accountID': newID,
                        'resourceID': item['ApplicationName'],
                        'region': region_name,
                        'service': 'ebs.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def lightsailServices(self):
        """ List all lightsail deployments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            lightsail= session.client('lightsail', region_name=region_name)
            try:
                response = lightsail.get_instances()
                for item in response['instances']:
                    results[item['name']]={
                        'accountID': newID,
                        'resourceID': item['name'],
                        'region': region_name,
                        'service': 'lightsail.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def emrServices(self):
        """ List all elastic map reduce deployments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            emr = session.client('emr', region_name=region_name)
            try:
                response = emr.list_clusters()
                for item in response['Clusters']:
                    results[item['Name']]={
                        'accountID': newID,
                        'resourceID': item['Name'],
                        'region': region_name,
                        'service': 'elasticmapreduce.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def workspacesServices(self):
        """ List all workspace deployments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            workspaces = session.client('workspaces', region_name=region_name)
            try:
                response = workspaces.list_clusters()
                for item in response['Workspaces']:
                    results[item['ComputerName']]={
                        'accountID': newID,
                        'resourceID': item['ComputerName'],
                        'region': region_name,
                        'service': 'workspaces.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def guarddutyServices(self):
        """ List all guard duty detectors in each region. TODO Need to get findings. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            guardduty = session.client('guardduty', region_name=region_name)
            try:
                response = guardduty.list_detectors()
                for item in response['DetectorIds']:
                    results[item]={
                        'accountID': newID,
                        'resourceID': item,
                        'region': region_name,
                        'service': 'guardduty.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def redshiftServices(self):
        """ List all redshift deployments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            redshift = session.client('redshift', region_name=region_name)
            try:
                response = redshift.describe_clusters()
                for item in response['Clusters']:
                    results[item['DBName']]={
                        'accountID': newID,
                        'resourceID': item['DBName'],
                        'region': region_name,
                        'service': 'redshift.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results

    def elasticsearchServices(self):
        """ List all elasticsearch deployments in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            es = session.client('es', region_name=region_name)
            try:
                response = es.list_domain_names()
                for item in response['DomainNames']:
                    results[item]={
                        'accountID': newID,
                        'resourceID': item,
                        'region': region_name,
                        'service': 'es.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    def firehoseServices(self):
        """ List all firehose in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            firehose = session.client('firehose', region_name=region_name)
            try:
                response = firehose.list_delivery_streams()
                for item in response['DeliveryStreamNames']:
                    results[item]={
                        'accountID': newID,
                        'resourceID': item,
                        'region': region_name,
                        'service': 'firehose.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results

    def kinesisServices(self):
        """ List all firehose in each region. """
        session = self.session
        newID = self.id
        regions = getRegions(session)
        results = {}
        for region_name in regions:
            kinesis = session.client('kinesis', region_name=region_name)
            try:
                response = kinesis.list_streams()
                for item in response['StreamNames']:
                    results[item]={
                        'accountID': newID,
                        'resourceID': item,
                        'region': region_name,
                        'service': 'kinesis.amazonaws.com',
                        'resourceData': item
                    }
            except botocore.exceptions.ClientError as error:
                exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
                print(exceptionString)
        return results


    # functions that return global results--------------------------------------------

    def route53Services(self):
        """ List all hosted zones. """
        session = self.session
        newID = self.id
        route53 = session.client('route53')
        results = {}
        try:
            response = route53.list_hosted_zones()
            for item in response['HostedZones']:
                results[item['Name']] = {
                    'accountID': newID,
                    'resourceID': item['Name'],
                    'region': 'global',
                    'service': 'route53.amazonaws.com',
                    'resourceData': item
                }
        except botocore.exceptions.ClientError as error:
            exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
            print(exceptionString)
        return results


    def s3Services(self):
        """ List all s3 buckets. TODO Needs to be improved to add region. """
        session = self.session
        newID = self.id
        s3 = session.client('s3')
        results = {}
        try:
            buckets = s3.list_buckets()
            for x in buckets['Buckets']:
                results[x['Name']] = {
                    'accountID': newID,
                    'resourceID': x['Name'],
                    'region': 'global',
                    'service': 's3.amazonaws.com',
                    'resourceData': x,
                    }
        except botocore.exceptions.ClientError as error:
            exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
            print(exceptionString)
        return results


    def cloudwatchServices(self):
        """ List all dashboards. """
        session = self.session
        newID = self.id
        cloudwatch = session.client('cloudwatch')
        results = {}
        try:
            response = cloudwatch.list_dashboards()
            for item in response['DashboardEntries']:
                results[item['DashboardName']] = {
                    'accountID': newID,
                    'resourceID': item['DashboardName'],
                    'region': 'global',
                    'service': 'cloudwatch.amazonaws.com',
                    'resourceData': item
                }
        except botocore.exceptions.ClientError as error:
            exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
            print(exceptionString)
        return results

    def iamServices(self):
        """ List all iam users in the account. """
        session = self.session
        newID = self.id
        iam = session.client('iam')
        results = {}
        try:
            response = iam.list_users()
            for item in response['Users']:
                results[item['UserName']] = {
                    'accountID': newID,
                    'resourceID': item['UserName'],
                    'region': 'global',
                    'service': 'iam.amazonaws.com',
                    'resourceData': item
                }
        except botocore.exceptions.ClientError as error:
            exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
            print(exceptionString)
        return results

    def shieldServices(self):
        """ List all shield deployments in the account. """
        session = self.session
        newID = self.id
        shield = session.client('shield')
        results = {}
        try:
            response = shield.list_attacks()
            for item in response['AttackSummaries']:
                results[item['AttackId']] = {
                    'accountID': newID,
                    'resourceID': item['AttackId'],
                    'region': 'global',
                    'service': 'shield.amazonaws.com',
                    'resourceData': item
                }
        except botocore.exceptions.ClientError as error:
            exceptionString = 'account id: ' + newID + ' error: {0}'.format(error) + '\n'
            print(exceptionString)
        return results

AWS DevOps Pro test date approaching!

With the exam expiring on February 17th, I decided to schedule the DevOps Pro certification test on January 26th, giving me a bit over two weeks to get ready. I followed this same practice and got my Solutions Architect Pro cert, so it is time to get down and study hard for this test! Shooting for that 5/5!

#EDIT: pushed it to 2/2, last chance before they change the test!

Running dhcpd as a container on Mac OS X: the problems

Well, this has been challenging.

My goal was to implement dhcpd on each school site’s time machine server to provide failover and reduce latency. I am using Mac OS X Minis as the server platform, as we invested in them as Time Machine backups for critical staff laptops, and they also serve to manage printer and copier queues for the print accounting service. I hit a couple of road blocks that are forcing me to seriously consider investing a couple of grand in 4 linux boxes and just move the whole container concept back to linux. Here are a couple of the issues I’ve been working through:

  • Docker for Mac is not for production. I am finding out that the app is designed for single user, and doesn’t respond to tinkering with the storage-driver or other options, and I can’t run a swarm using Macs or Windows as nodes, so no docker services or that very cool mesh routing.
  • Running Docker for Mac as a production service. The app requires an interactive window session to run, so each time machine has to be set to automatically log in.
  • Freeing up port 67 & 68 is difficult. I moved the network interface to a static IP, but there were some legacy services from the Server app that kept taking over the ports I to run dhcpd from a container. I have to remove the Dhcp scope reference from the Dhcp service, even though the service was turned off, and I had to disable NetBoot.  This worked fine on OS X 1.10 – 13, but 10.14 will not release control of those ports no matter what I do, even after removing the Server app entirely. May have to take a hammer to it…
  • The container may not be handling all the traffic. I am testing a set up where the site’s router has two ip helper-address directives for each vlan, forwarding layer 2 traffic to the time machine and a central dhcpd server at a remote site, which is running on a trusty snowflake centos 6 installation, a treasured pet for many years. In reviewing the logs, the centos service appears to be handling all of the dhcpd traffic, while the container is only handling a fraction. I haven’t ruled out the time machine entirely, as I need to confirm how the switches and wifi controllers are handling the ip helper-address directives, but it is possible that the high volume of traffic (each site has over 500 devices over 5-7 subnets) may be too much for either the time machine’s interface or the container’s bridge network interface.

So, it has not been as easy as I had thought to implement Docker with Macs as nodes. However, I think I can use this proof of concept to perhaps fund some small linux boxes and build a proper cluster. We’ll see what happens…