CloudWanderer Storage Connectors

These storage connector classes allow CloudWanderer to push data to an extensible list of storage backends.

DynamoDB Connector

Allows CloudWanderer to store resources in DynamoDB.

class DynamoDbConnector(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)

CloudWanderer Storage Connector for DynamoDB.

Parameters
  • table_name (str) – The name of the table to store resources in.

  • endpoint_url (str) – Optional override endpoint url for DynamoDB.

  • boto3_session (boto3.session.Session) – Optional boto3 session to use to interact with DynamoDB. Useful if your DynamoDB table is in a different account/region to your configured defaults.

  • number_of_shards (int) – The number of shards to break records across low-cardinality indices. Prevents hot-partitions. If you don’t know what this means, ignore this setting.

  • client_args (dict) – Arguments to pass into the boto3 client. See: boto3.session.Session.client()

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.DynamoDbConnector(
...         endpoint_url='http://localhost:8000'
...     )]
... )
__init__(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)

Initialise the DynamoDbConnector.

Parameters
  • table_name (str) – The name of the table to read/write from/to.

  • endpoint_url (str) – Optional endpoint url (useful primarily to write to a local DynamoDB).

  • boto3_session (str) – Optional boto3 session to use to persist params.

  • client_args (dict) – Optional dictionary of arguments to be passed to the boto3 dynamodb client.

  • number_of_shards (int) – Optional specification of the number of shards to create for low-cardinality indexes.

Return type

None

delete_resource(urn)

Delete the resource and all its resource attributes from DynamoDB.

Parameters

urn (URN) – The URN of the resource to delete from Dynamo

Return type

None

delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete

Return type

None

init()

Create the DynamoDB Database.

Return type

None

read_all()

Return raw data from all DynamoDB table records (not just resources).

Return type

Iterator[dict]

read_resource(urn)

Return the resource with the specified cloudwanderer.urn.URN.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

read_resources(account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None

Memory Connector

class MemoryStorageConnector

Storage connector to place data in memory.

Useful for testing.

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()]
... )
delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (URN) – The URN of the resource to delete

Return type

None

delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete

Return type

None

init()

Do nothing. Dummy method to fulfil interface requirements.

Return type

None

read_all()

Return the raw dictionaries stored in memory.

Return type

Iterator[dict]

read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

read_resources(account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None

Base Connector

Module containing abstract classes for CloudWanderer storage connectors.

class BaseStorageConnector

Abstract class for specification of the CloudWanderer storage connector interface.

abstract delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (URN) – The URN of the resource to delete

Return type

None

abstract delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete

Return type

None

abstract init()

Initialise the storage backend whatever it is.

Return type

None

abstract read_all()

Return all records from storage.

Return type

Iterator[dict]

abstract read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

abstract read_resources(account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

abstract write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None