CloudWanderer Storage Connectors

These storage connector classes allow CloudWanderer to push data to an extensible list of storage backends.

Neptune GraphDB Connector

Storage Connector for Gremlin databases.

class GremlinStorageConnector(endpoint_url, supports_multiple_labels=False, test_prefix='', **kwargs)

Storage Connector for Gremlin databases.

Parameters
  • endpoint_url (str) –

  • test_prefix (str) –

Return type

None

__init__(endpoint_url, supports_multiple_labels=False, test_prefix='', **kwargs)

Create a GremlinStorageConnector.

Parameters
  • endpoint_url (str) – The url of the gremlin endpoint to connect to (e.g. ws://localhost:8182)

  • supports_multiple_labels – Some GraphDBs (Neptune/Neo4J) support multiple labels on a single vertex.

  • test_prefix (str) – A prefix that will be prepended to edge and vertex ids to allow scenario separation.

  • **kwargs – Any unspecified args will be pased to the DriverRemoteConnection object.

Return type

None

close()

Close the connection to the backend storage.

Return type

None

delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (URN) – The URN of the resource to delete

Return type

None

delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • cloud_name (str) – The name of the cloud in question (e.g. aws)

  • account_id (str) – Cloud Account ID (e.g. 111111111111)

  • region (str) – Cloud region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • cutoff (Optional[datetime.datetime]) – Delete any resource discovered before this time

Return type

None

generate_edge_id(source_urn, destination_urn)

Generate a primary edge id.

Parameters
Return type

str

generate_vertex_id(urn)

Generate a vertex id.

Parameters

urn (cloudwanderer.urn.PartialUrn) – The URN of the vertex to create.

Return type

str

init()

Initialise the storage backend whatever it is.

Return type

None

open()

Open the connection to the backend storage.

Return type

gremlin_python.driver.driver_remote_connection.DriverRemoteConnection

read_all()

Return all records from storage.

Return type

Iterator[dict]

read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • cloud_name (Optional[str]) – The name of the cloud in question (e.g. aws)

  • urn (Optional[Union[cloudwanderer.urn.URN, cloudwanderer.urn.PartialUrn]]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – Cloud Account ID (e.g. 111111111111)

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None

DynamoDB Connector

Allows CloudWanderer to store resources in DynamoDB.

class DynamoDbConnector(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)

CloudWanderer Storage Connector for DynamoDB.

Parameters
  • table_name (str) – The name of the table to store resources in.

  • endpoint_url (str) – Optional override endpoint url for DynamoDB.

  • boto3_session (boto3.session.Session) – Optional boto3 session to use to interact with DynamoDB. Useful if your DynamoDB table is in a different account/region to your configured defaults.

  • number_of_shards (int) – The number of shards to break records across low-cardinality indices. Prevents hot-partitions. If you don’t know what this means, ignore this setting.

  • client_args (dict) – Arguments to pass into the boto3 client. See: boto3.session.Session.client()

Return type

None

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.DynamoDbConnector(
...         endpoint_url='http://localhost:8000'
...     )]
... )
__init__(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)

Initialise the DynamoDbConnector.

Parameters
  • table_name (str) – The name of the table to read/write from/to.

  • endpoint_url (str) – Optional endpoint url (useful primarily to write to a local DynamoDB).

  • boto3_session (str) – Optional boto3 session to use to persist params.

  • client_args (dict) – Optional dictionary of arguments to be passed to the boto3 dynamodb client.

  • number_of_shards (int) – Optional specification of the number of shards to create for low-cardinality indexes.

Return type

None

close()

Close the connection to the backend storage.

Return type

None

delete_resource(urn)

Delete the resource and all its resource attributes from DynamoDB.

Parameters

urn (URN) – The URN of the resource to delete from Dynamo

Return type

None

delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • cloud_name (str) – The name of the cloud.

  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.

Return type

None

init()

Create the DynamoDB Database.

Return type

None

open()

Open the connection to the backend storage.

Return type

None

read_all()

Return raw data from all DynamoDB table records (not just resources).

Return type

Iterator[dict]

read_resource(urn)

Return the resource with the specified cloudwanderer.urn.URN.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • cloud_name (Optional[str]) – The name of the cloud.

  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None

Memory Connector

class MemoryStorageConnector

Storage connector to place data in memory.

Useful for testing.

Example

>>> import cloudwanderer
>>> cloud_wanderer = cloudwanderer.CloudWanderer(
...     storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()]
... )
Return type

None

close()

Do nothing. Dummy method to fulfil interface requirements.

Return type

None

delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (URN) – The URN of the resource to delete

Return type

None

delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • cloud_name (str) – The name of the cloud.

  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.

Return type

None

init()

Do nothing. Dummy method to fulfil interface requirements.

Return type

None

open()

Do nothing. Dummy method to fulfil interface requirements.

Return type

None

read_all()

Return the raw dictionaries stored in memory.

Return type

Iterator[dict]

read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • cloud_name (Optional[str]) – The name of the cloud.

  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None

Base Connector

Module containing abstract classes for CloudWanderer storage connectors.

class BaseStorageConnector

Abstract class for specification of the CloudWanderer storage connector interface.

abstract close()

Close the connection to the backend storage.

Return type

None

abstract delete_resource(urn)

Delete this resource and all its resource attributes.

Parameters

urn (URN) – The URN of the resource to delete

Return type

None

abstract delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)

Delete resources of type in account and region unless in list of URNs.

This is used primarily to clean up old resources.

Parameters
  • cloud_name (str) – The name of the cloud.

  • account_id (str) – AWS Account ID

  • region (str) – AWS region (e.g. 'eu-west-2')

  • service (str) – Service name (e.g. 'ec2')

  • resource_type (str) – Resource Type (e.g. 'instance')

  • cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.

Return type

None

abstract init()

Initialise the storage backend whatever it is.

Return type

None

abstract open()

Open the connection to the backend storage.

Return type

None

abstract read_all()

Return all records from storage.

Return type

Iterator[dict]

abstract read_resource(urn)

Return a resource matching the supplied urn from storage.

Parameters

urn (URN) – The AWS URN of the resource to return

Return type

Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

abstract read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)

Yield a resource matching the supplied urn from storage.

All arguments are optional.

Parameters
  • cloud_name (Optional[str]) – The name of the cloud.

  • urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return

  • account_id (Optional[str]) – AWS Account ID

  • region (Optional[str]) – AWS region (e.g. 'eu-west-2')

  • service (Optional[str]) – Service name (e.g. 'ec2')

  • resource_type (Optional[str]) – Resource Type (e.g. 'instance')

Return type

Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]

abstract write_resource(resource)

Persist a single resource to storage.

Parameters

resource (CloudWandererResource) – The CloudWandererResource to write.

Return type

None