Connecting to Amazon S3
You can find the AWS S3 Connection by navigating to Connections on the main side nav
AWS S3 provides simple object storage. S3 object storage manages data as objects, meaning all data types are stored in their native formats. For Caravel, it is useful for hosting data analytics in JSON format.
Use our AWS S3 connector to deliver Caravel insights to AWS S3 in JSON file format where you can read those files to get insight into recurring themes and sentiment and use in your apps.
This overview will cover best practice ways of setting up AWS S3 for secure access from Caravel and setting up authentication so you can get started with using AWS S3.
In order to authenticate with the S3 Connection, use the IAM console in AWS to set up a user with the correct permissions. The two steps required for this are:
- 1.Create a User
- 2.Create a Permissions Policy
To create a new user, click Add User in the IAM console. Select the Access key - Programmatic access option to ensure the new user has programmatic access and is granted the credentials Caravel needs for connection:
The first step after clicking "Add User" from the IAM Management Console in AWS
On the final step you will be issued an Access key ID and Secret access key which you'll need to enter when connecting Caravel to S3:
Final step in IAM Add User flow where you can get your Access key ID and Secret access key for the Caravel connection
Be sure to download or store your ID and Key somewhere safe so you have them handy for when you connect S3 to Caravel.
Before using the Caravel S3 connector, you will need to make sure the user is configured with the correct IAM permissions. S3 permission policies can be managed on two levels. You can manage all at the user level if desired but we're going to go ahead and set them at both the user and bucket level.
- 1.At the user level - we will enable the basic 'List Buckets' operation (which displays all buckets your user has access to).
- 2.At the individual bucket level - we will specify our user as a 'Principal' when defining bucket access permissions to grant our user access to write files to the Bucket.
You can divide the permissions between user-level and bucket-level policies, but you will always need to set 'List Buckets' operation at the user-level.
In the IAM console, when setting permissions for an AWS user you can create a policy to add the user to or you can add an 'inline policy' in the user's details screen. Let's add an inline policy. Select Add inline policy from the user's console:
Add the policy using the visual editor, search and select S3 from services, and select ListBucket in the actions list:
Next, we'll specify the bucket resource ARN for the ListBucket action. Open up the Resources section, ensure Specific is selected, and next to bucket, select Add ARN.
Insert the name of you bucket and click Add:
Select Review Policy. Give your new policy a name, and select Create Policy:
You will need two things to write the bucket-level policy. Your IAM User ARN and Bucket ARN. Your User ARN can be found on the Summary page of your User:
And your Bucket ARN can be found under the Properties tab of your S3 Bucket:
With these two values in hand, you can set an individual policy that grants your user access to write files your bucket.
Navigate to the S3 management console, and click on your bucket. Then select Permissions. Scroll down to Bucket policy and select Edit.
You can use Amazon's Policy Generator to generate your Bucket's policy. For Type of Policy select S3 Bucket Policy. Input your User ARN as Principal, under Actions select PutObject and PutObjectAcl, and under Amazon Resource Name (ARN) input your Bucket ARN and append it with
/*Actions, whose names include the word Object (GetObject, PutObject, DeleteObject) should be applied to resources inside of the bucket (arn:aws:s3:::my-bucket/*) instead of the bucket itself (arn:aws:s3:::my-bucket) Generate Your policy and copy and paste the resulting policy into your Bucket's policy editor, and select Save Changes. Below is an example of the resulting policy:
{
"Version": "2012-10-17",
"Id": "Policy1657921651068",
"Statement": [
{
"Sid": "Stmt1657921619518",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::your-user-arn-id:user/your-user"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::your-bucket/*"
}
]
}
That's it! Now your user, and only your user, has access to write files to your bucket. Next, we'll use those keys to authenticate your user with Caravel.
In Caravel, navigate to Connections in the main side nav and click Connect on the AWS S3 connection.
Last modified 8mo ago