Examples

Getting Started

Testing with a local DynamoDB

DynamoDB has a Docker Image that allows you to run a local, persistent DynamoDB. This provides us with a cheap and easy way to start trying out CloudWanderer.

$ docker run -p 8000:8000 -v $(pwd):/data amazon/dynamodb-local \
    -Djava.library.path=./DynamoDBLocal_lib \
    -jar DynamoDBLocal.jar \
    -sharedDb -dbPath /data/

This starts a DynamoDB docker image on your local machine and tells it to persist data into the current directory in a shared database file shared-local-instance.db. This allows the data to persist even if you stop the container.

>>> from cloudwanderer.storage_connectors import DynamoDbConnector
>>> local_storage_connector=DynamoDbConnector(
...     endpoint_url='http://localhost:8000'
... )

This creates an alternative storage connector that points at your local DynamoDB

>>> wanderer = cloudwanderer.CloudWanderer(storage_connectors=[local_storage_connector])

This passes the storage connector that points at your local DynamoDB into a new wanderer and now all subsequent CloudWanderer operations will occur against your local DynamoDB!

Testing With the Memory Connector

If you don’t mind that your data is thrown away as soon as your python executable stops you can test CloudWanderer using the Memory Storage Connector!

>>> import cloudwanderer
>>> wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()]
... )

It’s wise to do this in an interactive environment otherwise you may spend an inordinate amount of time re-querying your AWS environment!

Writing Resources

Writing all Resources from all Regions

Writing all Supported Resources in all regions is as simple as using the write_resources() method.

>>> import cloudwanderer
>>> storage_connector = cloudwanderer.storage_connectors.DynamoDbConnector()
>>> storage_connector.init()
>>> wanderer = cloudwanderer.CloudWanderer(storage_connectors=[storage_connector])
>>> wanderer.write_resources()

In that block we are:

  1. Creating a storage connector (in this case DynamoDB)

  2. Initialising the storage connector (in this case creating a dynamodb table called cloud_wanderer

  3. Creating a wanderer and using write_resources() to get all resources in all regions.

Important: This will create DynamoDB table in your AWS account and write a potentially large number of records to it which may incur some cost. See earlier examples for how to test against a local DynamoDB or memory.

Retrieving all VPCs from all Regions

>>> vpcs = storage_connector.read_resources(service='ec2', resource_type='vpc')
>>> for vpc in vpcs:
...     print('vpc_region:', vpc.urn.region)
...     vpc.load()
...     print('vpc_state: ', vpc.state)
...     print('is_default:', vpc.is_default)
vpc_region: eu-west-2
vpc_state:  available
is_default: True
vpc_region: us-east-1
vpc_state:  available
is_default: True

You’ll notice here we’re calling a property urn in order to print the region. AwsUrns are CloudWanderer’s way of uniquely identifying a resource.

You can also see we’re printing the vpc’s state and is_default attributes. It’s very important to notice the load() call beforehand which loads the resource’s data. Resources returned from any read_resources() call on DynamoDbConnector are lazily loaded unless you specify the urn= argument. This is due to the sparsely populated global secondary indexes in the DynamoDB table schema.

Once you’ve called load() you can access any property of the AWS resource that is returned by its describe method. E.g. for VPCs see EC2.Client.describe_vpcs. These attributes are stored as snake_case instead of the APIs camelCase, so isDefault becomes is_default.

Writing Secondary Resource Attributes

Some resources require additional API calls beyond the initial list or describe call to retrieve all their metadata. For example, let’s say we want to get the value of enableDnsSupport for a VPC. We can get this one of two ways, either by looping over the dictionaries in secondary_attributes on cloudwanderer_metadata, or by calling get_secondary_attribute() with a JMESPath.

>>> first_vpc = next(storage_connector.read_resources(service='ec2', resource_type='vpc'))
>>> first_vpc.cloudwanderer_metadata.secondary_attributes
[]
>>> first_vpc.get_secondary_attribute('[].EnableDnsSupport.Value')
[]

However, when we try, we find it empty!

This is because this value isn’t captured when we write the VPC using write_resources() as it’s not returned by a standard describe_vpcs() call.

Instead we have to find a way to call describe_vpc_attribute(). We do this with our own custom secondary attribute definitions in order to allow us to standardise the process of:

  1. Retrieving that additional information from methods like describe_vpc_attribute()

  2. Putting it in storage

  3. Returning it in our CloudWandererResource

These secondary attributes are written using write_secondary_attributes().

>>> wanderer.write_secondary_attributes()

Now we have to call load() to pull the new data into the object after calling write_secondary_attributes().

>>> first_vpc.load()
>>> first_vpc.cloudwanderer_metadata.secondary_attributes[0]['EnableDnsSupport']
{'Value': True}
>>> first_vpc.get_secondary_attribute('[].EnableDnsSupport.Value')
[True]

Reading Resources

Let’s say we want to get a list of role policies. We can start by getting the role

>>> role = next(storage_connector.read_resources(service_name='iam', resource_type='role'))
>>> role.load()

Next we need to find out what policies are attached.

>>> role.get_secondary_attribute('[].PolicyNames[0]')
['test-role-policy']

Then we can lookup the inline policy

>>> inline_policy_urn = cloudwanderer.AwsUrn(
...     account_id = role.urn.account_id,
...     region=role.urn.region,
...     service='iam',
...     resource_type='role_policy',
...     resource_id=f"{role.urn.resource_id}:test-role-policy"
... )
>>> inline_policy = storage_connector.read_resource(urn=inline_policy_urn)
>>> inline_policy.policy_document
{'Version': '2012-10-17', 'Statement': {'Effect': 'Allow', 'Action': 's3:ListBucket', 'Resource': 'arn:aws:s3:::example_bucket'}}

Deleting Stale Resources

CloudWanderer deletes resources which no longer exist automatically when you run: write_resources_of_service_in_region().

Individual Resources

Deleting individual resources (if necessary), can be done by calling delete_resource() directly on the storage connector.

e.g.

>>> vpc = next(storage_connector.read_resources(
...     service='ec2',
...     resource_type='vpc',
... ))
>>> str(vpc.urn)
'urn:aws:123456789012:eu-west-2:ec2:vpc:vpc-11111111'
>>> storage_connector.delete_resource(urn=vpc.urn)
>>> vpc = storage_connector.read_resource(
...     urn=vpc.urn
... )
>>> print(vpc)
None