CloudWanderer Storage Connectors
These storage connector classes allow CloudWanderer to push data to an extensible list of storage backends.
Neptune GraphDB Connector
Storage Connector for Gremlin databases.
- class GremlinStorageConnector(endpoint_url, supports_multiple_labels=False, test_prefix='', **kwargs)
Storage Connector for Gremlin databases.
- Parameters
endpoint_url (str) –
test_prefix (str) –
- Return type
None
- __init__(endpoint_url, supports_multiple_labels=False, test_prefix='', **kwargs)
Create a GremlinStorageConnector.
- Parameters
endpoint_url (str) – The url of the gremlin endpoint to connect to (e.g.
ws://localhost:8182
)supports_multiple_labels – Some GraphDBs (Neptune/Neo4J) support multiple labels on a single vertex.
test_prefix (str) – A prefix that will be prepended to edge and vertex ids to allow scenario separation.
**kwargs – Any unspecified args will be pased to the
DriverRemoteConnection
object.
- Return type
None
- close()
Close the connection to the backend storage.
- Return type
None
- delete_resource(urn)
Delete this resource and all its resource attributes.
- Parameters
urn (URN) – The URN of the resource to delete
- Return type
None
- delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)
Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
cloud_name (str) – The name of the cloud in question (e.g.
aws
)account_id (str) – Cloud Account ID (e.g.
111111111111
)region (str) – Cloud region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)cutoff (Optional[datetime.datetime]) – Delete any resource discovered before this time
- Return type
None
- generate_edge_id(source_urn, destination_urn)
Generate a primary edge id.
- Parameters
source_urn (cloudwanderer.urn.PartialUrn) – The URN of the resource we’re generating an edge from.
destination_urn (cloudwanderer.urn.PartialUrn) – The URN of the resource we’re generating an edge to.
- Return type
str
- generate_vertex_id(urn)
Generate a vertex id.
- Parameters
urn (cloudwanderer.urn.PartialUrn) – The URN of the vertex to create.
- Return type
str
- init()
Initialise the storage backend whatever it is.
- Return type
None
- open()
Open the connection to the backend storage.
- Return type
gremlin_python.driver.driver_remote_connection.DriverRemoteConnection
- read_all()
Return all records from storage.
- Return type
Iterator[dict]
- read_resource(urn)
Return a resource matching the supplied urn from storage.
- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)
Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
cloud_name (Optional[str]) – The name of the cloud in question (e.g.
aws
)urn (Optional[Union[cloudwanderer.urn.URN, cloudwanderer.urn.PartialUrn]]) – The AWS URN of the resource to return
account_id (Optional[str]) – Cloud Account ID (e.g.
111111111111
)region (Optional[str]) – AWS region (e.g.
'eu-west-2'
)service (Optional[str]) – Service name (e.g.
'ec2'
)resource_type (Optional[str]) – Resource Type (e.g.
'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- write_resource(resource)
Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
DynamoDB Connector
Allows CloudWanderer to store resources in DynamoDB.
- class DynamoDbConnector(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)
CloudWanderer Storage Connector for DynamoDB.
- Parameters
table_name (str) – The name of the table to store resources in.
endpoint_url (str) – Optional override endpoint url for DynamoDB.
boto3_session (boto3.session.Session) – Optional boto3 session to use to interact with DynamoDB. Useful if your DynamoDB table is in a different account/region to your configured defaults.
number_of_shards (int) – The number of shards to break records across low-cardinality indices. Prevents hot-partitions. If you don’t know what this means, ignore this setting.
client_args (dict) – Arguments to pass into the boto3 client. See:
boto3.session.Session.client()
- Return type
None
Example
>>> import cloudwanderer >>> cloud_wanderer = cloudwanderer.CloudWanderer( ... storage_connectors=[cloudwanderer.storage_connectors.DynamoDbConnector( ... endpoint_url='http://localhost:8000' ... )] ... )
- __init__(table_name='cloud_wanderer', endpoint_url=None, boto3_session=None, client_args=None, number_of_shards=10)
Initialise the DynamoDbConnector.
- Parameters
table_name (str) – The name of the table to read/write from/to.
endpoint_url (str) – Optional endpoint url (useful primarily to write to a local DynamoDB).
boto3_session (str) – Optional boto3 session to use to persist params.
client_args (dict) – Optional dictionary of arguments to be passed to the boto3 dynamodb client.
number_of_shards (int) – Optional specification of the number of shards to create for low-cardinality indexes.
- Return type
None
- close()
Close the connection to the backend storage.
- Return type
None
- delete_resource(urn)
Delete the resource and all its resource attributes from DynamoDB.
- Parameters
urn (URN) – The URN of the resource to delete from Dynamo
- Return type
None
- delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)
Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
cloud_name (str) – The name of the cloud.
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.
- Return type
None
- init()
Create the DynamoDB Database.
- Return type
None
- open()
Open the connection to the backend storage.
- Return type
None
- read_all()
Return raw data from all DynamoDB table records (not just resources).
- Return type
Iterator[dict]
- read_resource(urn)
Return the resource with the specified
cloudwanderer.urn.URN
.- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)
Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
cloud_name (Optional[str]) – The name of the cloud.
urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return
account_id (Optional[str]) – AWS Account ID
region (Optional[str]) – AWS region (e.g.
'eu-west-2'
)service (Optional[str]) – Service name (e.g.
'ec2'
)resource_type (Optional[str]) – Resource Type (e.g.
'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- write_resource(resource)
Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
Memory Connector
- class MemoryStorageConnector
Storage connector to place data in memory.
Useful for testing.
Example
>>> import cloudwanderer >>> cloud_wanderer = cloudwanderer.CloudWanderer( ... storage_connectors=[cloudwanderer.storage_connectors.MemoryStorageConnector()] ... )
- Return type
None
- close()
Do nothing. Dummy method to fulfil interface requirements.
- Return type
None
- delete_resource(urn)
Delete this resource and all its resource attributes.
- Parameters
urn (URN) – The URN of the resource to delete
- Return type
None
- delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)
Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
cloud_name (str) – The name of the cloud.
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.
- Return type
None
- init()
Do nothing. Dummy method to fulfil interface requirements.
- Return type
None
- open()
Do nothing. Dummy method to fulfil interface requirements.
- Return type
None
- read_all()
Return the raw dictionaries stored in memory.
- Return type
Iterator[dict]
- read_resource(urn)
Return a resource matching the supplied urn from storage.
- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)
Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
cloud_name (Optional[str]) – The name of the cloud.
urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return
account_id (Optional[str]) – AWS Account ID
region (Optional[str]) – AWS region (e.g.
'eu-west-2'
)service (Optional[str]) – Service name (e.g.
'ec2'
)resource_type (Optional[str]) – Resource Type (e.g.
'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- write_resource(resource)
Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None
Base Connector
Module containing abstract classes for CloudWanderer storage connectors.
- class BaseStorageConnector
Abstract class for specification of the CloudWanderer storage connector interface.
- abstract close()
Close the connection to the backend storage.
- Return type
None
- abstract delete_resource(urn)
Delete this resource and all its resource attributes.
- Parameters
urn (URN) – The URN of the resource to delete
- Return type
None
- abstract delete_resource_of_type_in_account_region(cloud_name, service, resource_type, account_id, region, cutoff)
Delete resources of type in account and region unless in list of URNs.
This is used primarily to clean up old resources.
- Parameters
cloud_name (str) – The name of the cloud.
account_id (str) – AWS Account ID
region (str) – AWS region (e.g.
'eu-west-2'
)service (str) – Service name (e.g.
'ec2'
)resource_type (str) – Resource Type (e.g.
'instance'
)cutoff (Optional[datetime.datetime]) – The date before which to delete resources of the specified type and account.
- Return type
None
- abstract init()
Initialise the storage backend whatever it is.
- Return type
None
- abstract open()
Open the connection to the backend storage.
- Return type
None
- abstract read_all()
Return all records from storage.
- Return type
Iterator[dict]
- abstract read_resource(urn)
Return a resource matching the supplied urn from storage.
- Parameters
urn (URN) – The AWS URN of the resource to return
- Return type
Optional[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- abstract read_resources(cloud_name=None, account_id=None, region=None, service=None, resource_type=None, urn=None)
Yield a resource matching the supplied urn from storage.
All arguments are optional.
- Parameters
cloud_name (Optional[str]) – The name of the cloud.
urn (Optional[cloudwanderer.urn.URN]) – The AWS URN of the resource to return
account_id (Optional[str]) – AWS Account ID
region (Optional[str]) – AWS region (e.g.
'eu-west-2'
)service (Optional[str]) – Service name (e.g.
'ec2'
)resource_type (Optional[str]) – Resource Type (e.g.
'instance'
)
- Return type
Iterator[cloudwanderer.cloud_wanderer_resource.CloudWandererResource]
- abstract write_resource(resource)
Persist a single resource to storage.
- Parameters
resource (CloudWandererResource) – The CloudWandererResource to write.
- Return type
None