Writing a Dependent Resource

In this guide we’re going to write a CloudWanderer dependent resource definition for lambda layer versions.

A CloudWanderer dependent resource is a specific type of Boto3 resource.

CloudWanderer dependent resources are resources which depend on their parent for their identity. A resource is a dependent resource in CloudWanderer terms if you must supply the ID of its parent in order to fetch it (e.g. IAM Role inline policies).

In our case lambda layer versions are dependent resources because you cannot fetch metadata about them from the AWS API without supplying the name of the layer of which they are a version.

Note

This guide assumes you have read the Writing a Resource Definition guide as a pre-requisite. As it has a lot more detail on some of the steps outlined here. Look to that page for further clarification if need be.

Getting the test data

$ aws lambda list-layer-versions --layer-name test-layer
{
    "LayerVersions": [
        {
            "LayerVersionArn": "arn:aws:lambda:eu-west-1:123456789012:layer:test-layer:1",
            "Version": 1,
            "Description": "This is a test layer!",
            "CreatedDate": "2020-10-17T13:18:00.303+0000",
            "CompatibleRuntimes": [
                "nodejs10.x"
            ]
        }
    ]
}

Create the tests

Dependent resources are always discovered alongside their parents, so our tests for lambda layer versions will fit nicely alongside the tests for lambda layers.

 1{
 2    "service": "lambda",
 3    "mockData": {
 4        "get_paginator.side_effect": [
 5            {
 6                "paginate.return_value": [
 7                    {
 8                        "Layers": [
 9                            {
10                                "LayerName": "test-layer",
11                                "LayerArn": "arn:aws:lambda:eu-west-1:123456789012:layer:test-layer",
12                                "LatestMatchingVersion": {
13                                    "LayerVersionArn": "arn:aws:lambda:eu-west-1:123456789012:layer:test-layer:1",
14                                    "Version": 1,
15                                    "Description": "This is a test layer!",
16                                    "CreatedDate": "2020-10-17T13:18:00.303+0000",
17                                    "CompatibleRuntimes": [
18                                        "nodejs10.x"
19                                    ]
20                                }
21                            }
22                        ]
23                    }
24                ]
25            },
26            {
27                "paginate.return_value": [
28                    {
29                        "LayerVersions": [
30                            {
31                                "LayerVersionArn": "arn:aws:lambda:eu-west-1:123456789012:layer:test-layer:1",
32                                "Version": 1,
33                                "Description": "This is a test layer!",
34                                "CreatedDate": "2020-10-17T13:18:00.303+0000",
35                                "CompatibleRuntimes": [
36                                    "nodejs10.x"
37                                ]
38                            }
39                        ]
40                    }
41                ]
42            }
43        ]
44    }
45}

We’ve added our test payload as a second value in the existing mockData.get_paginator.side_effect list. It is the second in the list because dependent resources are discovered _after_ top level resources.

..tip:

We'll also need to add another dict to the end of the ``expectedResults`` key, but like before you can populate
that with the results the failing test spits out when you run it at the end!

Populate the definition

Visit Botocore’s specification data on GitHub and open the latest service-2.json for your service.

We’re here following much the same process as in Writing a Resource Definition. We’re going to add a LambdaLayerVersion resource specification to the aws_interface/resource_definitions/lambda/2015-03-31/resource-1.json file.

We’re following our ListLayerVersions through its return of ListLayerVersionsResponse to its shape of LayerVersionsList to its member of LayerVersionsListItem. Along the way we’ve identified the

  • Collection Request Operation (ListLayerVersions)

  • Resource Shape (LayerVersionsListItem)

  • Identifiers (LayerName and Version)

Dependent Resource Identifiers

Dependent resources always have two identifiers, one is the identifier of their parent, and the other is their identifier. In CloudWanderer’s definition this is what makes them a dependent resource, that they do not have an independent identity without their parent.

[
    {
        "target": "LayerName",
        "source": "identifier",
        "name": "LayerName"
    },
    {
        "target": "Version",
        "source": "response",
        "path": "LayerVersions[].Version"
    }
]

Dependent Resource Request Operation

Top level resource request operations (e.g. ListLayers) were pretty simple as they had no arguments. Dependent resources on the other hand need to supply the identity of their parent as an argument to whatever API method they’re calling.

{
    "operation": "ListLayerVersions",
    "params": [
        {
            "target": "LayerName",
            "source": "identifier",
            "name": "LayerName"
        }
    ]
}

This will take the LayerName from the parent resource, and submit it as a parameter called LayerName to the ListLayerVersions API method.

Bringing it all together

In the resources definiton we’ve added the highlighted lines.

 1{
 2    "resources": {
 3        "Layer": {
 4            "identifiers": [
 5                {
 6                    "name": "LayerName",
 7                    "memberName": "LayerName"
 8                }
 9            ],
10            "shape": "LayersListItem",
11            "hasMany": {
12                "LayerVersions": {
13                    "request": {
14                        "operation": "ListLayerVersions",
15                        "params": [
16                            {
17                                "target": "LayerName",
18                                "source": "identifier",
19                                "name": "LayerName"
20                            }
21                        ]
22                    },
23                    "resource": {
24                        "type": "LayerVersion",
25                        "identifiers": [
26                            {
27                                "target": "LayerName",
28                                "source": "identifier",
29                                "name": "LayerName"
30                            },
31                            {
32                                "target": "Version",
33                                "source": "response",
34                                "path": "LayerVersions[].Version"
35                            }
36                        ],
37                        "path": "LayerVersions[]"
38                    }
39                }
40            }
41        },
42        "LayerVersion": {
43            "identifiers": [
44                {
45                    "name": "LayerName",
46                    "memberName": "LayerName"
47                },
48                {
49                    "name": "Version",
50                    "memberName": "Version"
51                }
52            ],
53            "shape": "LayerVersionsListItem"
54        }
55    }
56}

You’ll notice we’ve added the collection specification inside the Layer resource instead of inside the service as we did in Writing a Resource Definition, this is what allows us to reference the LayerName of the parent resource when we call ListLayerVersions in our collection API call.

Writing the Service Map

The service map is CloudWanderer’s store for resource type metadata that does not fit into the Boto3 specification. It broadly follows the structure of Boto3’s to try and keep things simple and consistent. For our new Layer resource we just need to ensure that the following exists in aws_interface/resource_definitions/lambda/2015-03-31/resources-cw-1.json

 1{
 2    "service": {
 3        "globalService": false,
 4        "regionalResources": true
 5    },
 6    "resources": {
 7        "Layer": {
 8            "type": "baseResource"
 9        },
10        "LayerVersion": {
11            "type": "dependentResource"
12        }
13    }
14}

We added the LayerVersion key to resources to indicate that we’ve added a dependent resource whose parent resource type is Layer. This allows CloudWanderer to determine the proper relationship between these resources and properly generate URNs.

  • globalService indicates whether the service has a Global API or not (that is, whether all API requests go to a single region, e.g. us-east-1 for IAM).

  • regionalResources indicates whether the service has regional resources or not (most do, but some, like IAM do not).

  • type indicates what type of resource this is, either baseResource, subresource or secondaryAttribute.

Running the tests

Now you’ve put all the pieces together you need to run the tests. You can see the specifications on GitHub:

To run the tests:

# Install the pre-reqs
$ pip install -r requirements-test.txt -r requirements.txt
# Install the package in interactive mode
$ pip install -e .
# Run the tests
$ pytest tests/integration/custom_resources/ -k layer_multiple_resources
AssertionError: Dictionaries do not match
E           Second dict as json: {"urn": ...

You can then take the Second dict as json result and place it in your expectedResults list, validate it’s what you expect, and re-run the test.

Tip

Make sure to remove the key/value for discovery_time as this will change with every test execution!

Submit a PR!

Congratulations! You have successfully created a new resource! If you submit a Pull Request to https://github.com/CloudWanderer-io/CloudWanderer/ with your new resource we will get it merged in and released for everyone to use as quickly as we possibly can! If you find you’re not getting the attention you deserve for whatever reason, contact us on twitter.