Storage Connector Reference

DynamoDB Connector

Allows CloudWanderer to store resources in DynamoDB.

class DynamoDbConnector(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)

CloudWanderer Storage Connector for DynamoDB.

Parameters
  • table_name (str) – The name of the table to store resources in.

  • endpoint_url (str) – Optional override endpoint url for DynamoDB.

  • boto3_session (boto3.session.Session) – Optional boto3 session to use to interact with DynamoDB. Useful if your DynamoDB table is in a different account/region to your configured defaults.

  • number_of_shards (int) – The number of shards to break records across low-cardinality indices. Prevents hot-partitions. If you don’t know what this means, ignore this setting.

  • client_args (dict) – Arguments to pass into the boto3 client. See: boto3.session.Session.client()

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.DynamoDbConnector(
...         endpoint_url='http://localhost:8000'
...     )]
... )
delete_resource(urn)

Delete the resource and all its resource attributes from DynamoDB.

Parameters

urn (AwsUrn) – The URN of the resource to delete from Dynamo

Return type

None

delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account id unless in list of URNs.

Parameters
  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • urns_to_keep (List[cloudwanderer.aws_urn.AwsUrn]) – A list of resources not to delete

Return type

None

init()

Create the DynamoDB Database.

Return type

None

read_all()

Return raw data from all DynamoDB table records (not just resources).

Return type

Iterator[dict]

read_resource(urn)

Return the resource with the specified cloudwanderer.aws_urn.AwsUrn.

Parameters

urn (AwsUrn) – The AWS URN of the resource to return

Return type

cloudwanderer.cloud_wanderer_resource.CloudWandererResource

read_resources(**kwargs)

Return the resources matching the arguments.

All arguments are optional, though some will fallback to performing a table scan.

Parameters
  • urn (cloudwanderer.aws_urn.AwsUrn) – The AWS URN of the resource to return

  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(urn, resource)

Write the specified resource to DynamoDB.

Parameters
Return type

None

write_secondary_attribute(urn, attribute_type, secondary_attribute)

Write the specified resource attribute to DynamoDb.

Parameters
  • urn (AwsUrn) – The resource whose attribute to write.

  • attribute_type (str) – The type of the resource attribute to write (usually the boto3 client method name)

  • secondary_attribute (boto3.resources.base.ServiceResource) – The resource attribute to write to storage.

Return type

None

Memory Connector

class MemoryStorageConnector(*args, **kwargs)

Storage connector to place data in memory.

Useful for testing.

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()]
... )
delete_resource(urn)

Delete the resource and all its resource attributes from memory.

Parameters

urn (cloudwanderer.aws_urn.AwsUrn) –

Return type

None

delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account id unless in list of URNs.

Parameters
  • service (str) –

  • resource_type (str) –

  • account_id (str) –

  • region (str) –

  • urns_to_keep (Optional[List[cloudwanderer.aws_urn.AwsUrn]]) –

Return type

None

init()

Dummy method to fulfil interface requirements.

Return type

None

read_all()

Return the raw dictionaries stored in memory.

Return type

Iterator[dict]

read_resource(urn)

Return the resource with the specified cloudwanderer.aws_urn.AwsUrn.

Parameters

urn (cloudwanderer.aws_urn.AwsUrn) – The AWS URN of the resource to return

Return type

cloudwanderer.cloud_wanderer_resource.CloudWandererResource

read_resources(**kwargs)

Return the resources matching the arguments.

All arguments are optional

Parameters
  • urn (cloudwanderer.aws_urn.AwsUrn) – The AWS URN of the resource to return

  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(urn, resource)

Write the specified resource to memory.

Parameters
Return type

None

write_secondary_attribute(urn, attribute_type, secondary_attribute)

Write the specified resource attribute to DynamoDb.

Parameters
  • urn (AwsUrn) – The resource whose attribute to write.

  • attribute_type (str) – The type of the resource attribute to write (usually the boto3 client method name)

  • secondary_attribute (boto3.resources.base.ServiceResource) – The resource attribute to write to storage.

Return type

None

Base Connector

Module containing abstract classes for CloudWanderer storage connectors.

class BaseStorageConnector

Abstract class for specification of the CloudWanderer storage connector interface.

abstract delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (cloudwanderer.aws_urn.AwsUrn) –

Return type

None

abstract delete_resource_of_type_in_account_region(service, resource_type, account_id, region, urns_to_keep=None)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • service (str) –

  • resource_type (str) –

  • account_id (str) –

  • region (str) –

  • urns_to_keep (Optional[List[cloudwanderer.aws_urn.AwsUrn]]) –

Return type

None

abstract init()

Initialise the storage backend whatever it is.

Return type

None

abstract read_all()

Return all records from storage.

Return type

Iterator[dict]

abstract read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (cloudwanderer.aws_urn.AwsUrn) –

Return type

cloudwanderer.cloud_wanderer_resource.CloudWandererResource

abstract read_resources(urn, account_id, region, service, resource_type)

Yield a resource matching the supplied urn from storage.

Parameters
Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

abstract write_resource(urn, resource)

Persist a single resource to storage.

Parameters
Return type

None

abstract write_secondary_attribute(urn, attribute_type, secondary_attribute)

Write the specified resource attribute to DynamoDb.

Parameters
  • urn (AwsUrn) – The resource whose attribute to write.

  • attribute_type (str) – The type of the resource attribute to write (usually the boto3 client method name)

  • secondary_attribute (boto3.resources.base.ServiceResource) – The resource attribute to write to storage.

Return type

None