AWS Setup for Snowplow Pipeline

Snowplow documentation

AWS Setup for Snowplow Pipeline¶

To setup Snowplow, we need you to create a sub-account dedicated to Snowplow, then set up a user within this sub-account with the appropriate permissions (using an IAM policy) to set up the pipeline. The process for this is as follows:

Create sub-account¶

1. From your main AWS account, set up an Organisation if you haven’t done so already.

2. Create an account in that organisation (sub-account)

3. Sign out and sign into the new sub-account. Everything Snowplow-related will take place within this account from here in.

Set up User and IAM permissions¶

1. Create the IAM policy

2. Create a group and assign them this policy

3. Create user in that group, and send the login details in the pipeline setup form in the Insights console.

Walkthrough¶

Step 1. Create an Organisation and sub-account¶

• From the console homepage, navigate to ‘AWS Organizations’ (via the search bar), and select Create Organization.

• Create an organization - Choose enable all features

• Add account - you will need to create this account with a separate email address. This will be the sub-account we use for Snowplow. You can invite an existing account, or create a new one, but make sure that this account is not used for anything but setting up the Snowplow Pipeline.

• Sign out, and sign in with that account - this is your Snowplow sub-account and everything to do with Snowplow happens here from now on.

Step 2. Create a Permissions Policy, User Group, and User¶

Create IAM policy:¶

• Navigate to ‘IAM’ (via the search bar) - Note that all steps from here take place from within this section

• Navigate to Policies (left panel) -> Create policy -> Create Your Own Policy

• Name the policy snowplow-policy-setup-infrastructure, copy and paste the below into the policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:*",
        "autoscaling:*",
        "aws-marketplace:Subscribe",
        "aws-marketplace:Unsubscribe",
        "aws-marketplace:ViewSubscriptions",
        "cloudformation:*",
        "cloudfront:*",
        "cloudwatch:*",
        "dynamodb:*",
        "ec2:*",
        "es:*",
        "elasticbeanstalk:*",
        "elasticloadbalancing:*",
        "elasticmapreduce:*",
        "iam:*",
        "kinesis:*",
        "logs:*",
        "rds:*",
        "redshift:*",
        "s3:*",
        "sns:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Create a Group with this policy:¶

• Navigate to Groups (left pane) -> Create New Group

• Name the Group snowplow-setup

• On the attach policy step, select the policy we have just created: snowplow-policy-setup-infrastructure. You can use the searchbar to find it easily.

• Review the details and create the group.

Create a User in this Group:¶

• Navigate to Users (left pane, from the IAM homescreen) -> Add User

• Name the user snowplow-setup

• Select both checkboxes under ‘Access type’

• Select Autogenerated Password under ‘Console Password’

• Make sure that the Require password reset box is checked

• On the next screen, add the check the box next to snowplow-setup to add the user to this group.

• Once you see the success message, copy the username, password and AWS Console login URL - we will need these to set up the pipeline.

User Creation¶

You will need to create users in order to upload schemas and implement custom tracking - we will create a usergroup SnowplowIgluAdmin with the minimum required permissions to do so - it is recommended that users are created within this group for these purposes.