CloudWanderer Storage Connectors¶
These storage connector classes allow CloudWanderer to push data to an extensible list of storage backends.
DynamoDB Connector¶
Allows CloudWanderer to store resources in DynamoDB.
-
class
DynamoDbConnector
(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)¶ CloudWanderer Storage Connector for DynamoDB.
- Parameters
table_name (str) – The name of the table to store resources in.
endpoint_url (str) – Optional override endpoint url for DynamoDB.
boto3_session (boto3.session.Session) – Optional boto3 session to use to interact with DynamoDB. Useful if your DynamoDB table is in a different account/region to your configured defaults.
number_of_shards (int) – The number of shards to break records across low-cardinality indices. Prevents hot-partitions. If you don’t know what this means, ignore this setting.
client_args (dict) – Arguments to pass into the boto3 client. See:
boto3.session.Session.client()
Example
>>> import cloudwanderer >>> cloud_wanderer = cloudwanderer.CloudWanderer( ... storage_connectors=[cloudwanderer.storage_connectors.DynamoDbConnector( ... endpoint_url='http://localhost:8000' ... )] ... )
-
__init__
(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)¶ Initialise the DynamoDbConnector.
- Parameters
table_name (str) – The name of the table to read/write from/to.
endpoint_url (str) – Optional endpoint url (useful primarily to write to a local DynamoDB).
boto3_session (str) – Optional boto3 session to use to persist params.
client_args (dict) – Optional dictionary of arguments to be passed to the boto3 dynamodb client.
number_of_shards (int) – Optional specification of the number of shards to create for low-cardinality indexes.
- Return type
None
-
delete_resource
(urn)¶ Delete the resource and all its resource attributes from DynamoDB.
- Parameters
urn (URN) – The URN of the resource to delete from Dynamo
- Return type
None
-
delete_resource_of_type_in_account_region
(service, resource_type, account_id, region, urns_to_keep=None)¶ Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete
- Return type
None
-
init
()¶ Create the DynamoDB Database.
- Return type
None
-
read_all
()¶ Return raw data from all DynamoDB table records (not just resources).
- Return type
Iterator[dict]
-
read_resource
(urn)¶ Return the resource with the specified
cloudwanderer.urn.URN
.- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
-
read_resources
(account_id=None, region=None, service=None, resource_type=None, urn=None)¶ Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
kwargs – Optional arguments narrowing the scope of the resources returned.
account_id (Optional[str]) –
region (Optional[str]) –
service (Optional[str]) –
resource_type (Optional[str]) –
urn (Optional[cloudwanderer.urn.URN]) –
- Keyword Arguments
urn (URN) – The AWS URN of the resource to return
account_id (
str
) – AWS Account IDregion (
str
) – AWS region (e.g.'eu-west-2'
)service (
str
) – Service name (e.g.'ec2'
)resource_type (
str
) – Resource Type (e.g.'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
-
write_resource
(resource)¶ Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
Memory Connector¶
-
class
MemoryStorageConnector
¶ Storage connector to place data in memory.
Useful for testing.
Example
>>> import cloudwanderer >>> cloud_wanderer = cloudwanderer.CloudWanderer( ... storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()] ... )
-
delete_resource
(urn)¶ Delete this resource and all its resource attributes.
- Parameters
urn (URN) – The URN of the resource to delete
- Return type
None
-
delete_resource_of_type_in_account_region
(service, resource_type, account_id, region, urns_to_keep=None)¶ Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete
- Return type
None
-
init
()¶ Do nothing. Dummy method to fulfil interface requirements.
- Return type
None
-
read_all
()¶ Return the raw dictionaries stored in memory.
- Return type
Iterator[dict]
-
read_resource
(urn)¶ Return a resource matching the supplied urn from storage.
- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
-
read_resources
(**kwargs)¶ Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
kwargs – Optional arguments narrowing the scope of the resources returned.
- Keyword Arguments
urn (URN) – The AWS URN of the resource to return
account_id (
str
) – AWS Account IDregion (
str
) – AWS region (e.g.'eu-west-2'
)service (
str
) – Service name (e.g.'ec2'
)resource_type (
str
) – Resource Type (e.g.'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
-
write_resource
(resource)¶ Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
-
Base Connector¶
Module containing abstract classes for CloudWanderer storage connectors.
-
class
BaseStorageConnector
¶ Abstract class for specification of the CloudWanderer storage connector interface.
-
abstract
delete_resource
(urn)¶ Delete this resource and all its resource attributes.
- Parameters
urn (URN) – The URN of the resource to delete
- Return type
None
-
abstract
delete_resource_of_type_in_account_region
(service, resource_type, account_id, region, urns_to_keep=None)¶ Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)urns_to_keep (List[cloudwanderer.urn.URN]) – A list of resources not to delete
- Return type
None
-
abstract
init
()¶ Initialise the storage backend whatever it is.
- Return type
None
-
abstract
read_all
()¶ Return all records from storage.
- Return type
Iterator[dict]
-
abstract
read_resource
(urn)¶ Return a resource matching the supplied urn from storage.
- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
-
abstract
read_resources
(**kwargs)¶ Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
kwargs – Optional arguments narrowing the scope of the resources returned.
- Keyword Arguments
urn (URN) – The AWS URN of the resource to return
account_id (
str
) – AWS Account IDregion (
str
) – AWS region (e.g.'eu-west-2'
)service (
str
) – Service name (e.g.'ec2'
)resource_type (
str
) – Resource Type (e.g.'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
-
abstract
write_resource
(resource)¶ Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
-
abstract