Amazon QuickSight asset deployments using AssetBundle Export and Import APIs - Private Datasource Guidelines

This post is to provide a step by step guide lines on how to export your Analysis which is using a Redshift joined dataset in a different account.

Here we will be deploying our existing analysis in Source Account to another account (Target Account)

Prerequisites:

a) Redshift Cluster in Source and Target Account with required tables
b) Analysis in Source QuickSight account which is using a Redshift Dataset.
c) QuickSight Permissions to call the APIs:

The following IAM policy allows an IAM user to get access to these APIs:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [ 
 "quicksight:StartAssetBundleImportJob",
 "quicksight:DescribeAssetBundleImportJob",
 "quicksight:ListAssetBundleImportJobs",
 "quicksight:StartAssetBundleExportJob",
 "quicksight:DescribeAssetBundleExportJob",
 "quicksight:ListAssetBundleExportJobs"
 ],
 "Resource": "*"
 }
 ]
}

Steps to perform in SOURCE Account:

Note down the Analysis ID (Resource ID) that you want to deploy in the Target Account, frame the CLI commands as following to start the Export Job:

Step 1.1:

Call the StartAssetBundleExportJob API call:

aws quicksight start-asset-bundle-export-job --aws-account-id 1111111111 --asset-bundle-export-job-id job-1 --resource-arns arn:aws:quicksight:us-east-1:1234567895:analysis/e5f07a80-60db-4e2f-a535-f29165ca2577 --include-all-dependencies --export-format QUICKSIGHT_JSON

Step 1.2:

Call the DescribeAssetBundleExportJob API to get the .qs file.

aws quicksight describe-asset-bundle-export-job --aws-account-id 1111111111 --asset-bundle-export-job-id job-1

Step 1.3:

Click on the “DownloadUrl“ from the output of Step 1.2, a file will be automatically downloaded in your local machine (Please note the file extension will be .qs).

Steps to perform in TARGET Account:

Step 2.1:

Create a S3 Bucket in your Target account or use an already existing S3 Bucket, create a folder and upload the the .qs file which you downloaded in previous step (1.3).

Step 2.2:

Frame a CLI command like below to import your assets in the Target account. You have to call the StartAssetBundleImportJob API to import your resources.

Please note:

• I had already created a Redshift cluster in Target account
• The IAM role associated with the cluster should have Quicksight in Trust policy
• Redshift SG should allow connections if publicly accessible is set or you need to add VPC connection configuration in your CLI command

aws quicksight start-asset-bundle-import-job --aws-account-id 222222222222 --asset-bundle-import-job-id job-1 --region us-east-1 --asset-bundle-import-source "{\"S3Uri\": \"s3://S3BucketName/job1.qs\"}" --override-parameters '{"ResourceIdOverrideConfiguration":{"PrefixForAllResources":"Imported-" },"DataSources":[{"Credentials": { "CredentialPair": { "Username": "awsuser","Password": "Password"}},"DataSourceId": "Imported-1b344e51-4da2-4845-8cd9-8a0ad01c4fa2","DataSourceParameters":{"RedshiftParameters": { "ClusterId": "redshift-cluster-1", "Database": "dev", "Host":"redshift-cluster-1.cabcedef.us-east-1.redshift.amazonaws.com","IAMParameters": { "DatabaseUser": "awsuser","RoleArn": "arn:aws:iam::22222222222:role/service-role/AmazonRedshift-CommandsAccessRole-20231027T225850"}, "Port": 5439}}}],"DataSets":[{"DataSetId":"Imported-Dataset","Name":"Imported-target"}]}'

Notes:

• Here, S3Uri is the location where .qs file is present
• Credential Pair contains the credentials of my Redshift cluster in Target Account
• DataSourceId should be the same as the DatasourceID in your Source Account (Check QuickSight → Manage QuickSight → Manage Assets → Datasource)
• If your DataSource ID in Source Account is “abcd12345”
• Since I added the prefix “Imported-” the DataSourceId in AssetBundleImportJob will be “Imported-abcd12345”
• Otherwise, you will run into an error like “DataSourceCredentials field is incorrectly set”

Once you have framed the command like above, execute the StartAssetBundleImportJob

Step 2.3:

Now you need to call the DescribeAssetBundleImportJob API to check the whether your resources have been imported successfully or not.

aws quicksight describe-asset-bundle-import-job --aws-account-id 22222222222 --asset-bundle-import-job-id job-1 --region us-east-1

While importing your DescribeAssetBundleImportJob will show status as “IN_PROGRESS”, you can call the command again to check whether it is showing “SUCCESSFUL” or not.

In case it fails the job status will be as “FAILED_ROLLBACK_COMPLETED” and it will show you the corresponding error message on why it has failed.

A SUCCESSFUL import will look like the below:

Step 2.4:

Once, the import job is Successful, you have to login to your QuickSight in Target Account and share your resources to the users, otherwise it will not be visible (even for Admins).

Go to Manage QuickSight → Manage Assets → DataSources
Find your newly created Datasource prefixed with “Imported-“, click on share and share it with respective users.

Do the same for datasets and analysis. Once the sharing is done, you can see the Analysis in your Target account and access it accordingly.

In case your Redshift Cluster is Private, you need to configure the VPC Connection:

Most of our customers have their data in Private subnets and have their Redshift cluster setup such that the “PubliclyAccesible” option is set as “Disabled”.
In case, your Redshift Cluster is Private, you need to configure the VPC Connection.

Notes:

• Make note of the VPC connection Id from the Source Account - use the same VPC Connection ID in the below command. (Imported is the override prefix used)
• However, you have use the VPC ID and subnets and Security Groups present in the TARGET account - Go to your Redshift Cluster, check network properties and mention them accordingly.
• Use the DataSourceID same as the datasource id from Source Account
• Make sure your execution role for VPC connection in QuickSight has required permissions as listed below

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "ec2:CreateNetworkInterface",
 "ec2:ModifyNetworkInterfaceAttribute",
 "ec2:DeleteNetworkInterface",
 "ec2:DescribeSubnets",
 "ec2:DescribeSecurityGroups"
 ],
 "Resource": "*"
 }
 ]
}

Note down all the relevant subnets, VPC-id and execution role details in Target account and frame the CLI command as below:
Import Job Command:

aws quicksight start-asset-bundle-import-job --aws-account-id 222222222222 --asset-bundle-import-job-id job-2 --region us-east-1 --asset-bundle-import-source "{\"S3Uri\": \"s3://S3BucketName/asset-bundle-job/job-2.qs\"}" --override-parameters '{"ResourceIdOverrideConfiguration":{"PrefixForAllResources":"Imported-" },"DataSources":[{"Credentials": { "CredentialPair": { "Username": "awsuser","Password": "Password"}},"DataSourceId": "Imported-581fbf38-d828-4566-9fb3-9a1f7322cf2a","DataSourceParameters":{"RedshiftParameters": { "ClusterId": "redshift-cluster-2", "Database": "dev", "Host":"redshift-cluster-2.22222222222.us-east-1.redshift.amazonaws.com","IAMParameters": { "DatabaseUser": "awsuser","RoleArn": "arn:aws:iam::2222222222:role/service-role/AmazonRedshift-CommandsAccessRole-20231027T225850"}, "Port": 5439}}}],"DataSets":[{"DataSetId":"Imported-Dataset-2","Name":"Imported-target-vpc"}],"VPCConnections":[{"VPCConnectionId":"Imported-9783a14e-bc90-4f44-96ef-fd47dca5e38b","SubnetIds":["subnet-0153bf2ded39e4659","subnet-077f112368a3913cc"],"SecurityGroupIds":["sg-0838ab5e0ba494dd8"],"RoleArn":"arn:aws:iam::22222222222:role/service-role/aws-quicksight-service-role-v0"}]}'

Check the Import Job status:

aws quicksight describe-asset-bundle-import-job --aws-account-id 22222222222 --asset-bundle-import-job-id job-2 --region us-east-1

Once successful, make sure you SHARE the resources from “Manage QuickSight” in the TARGET Account

Useful links:

• https://docs.aws.amazon.com/quicksight/latest/developerguide/asset-bundle-ops.html
• https://docs.aws.amazon.com/quicksight/latest/APIReference/API_StartAssetBundleImportJob.html
• https://community.amazonquicksight.com/t/automate-and-accelerate-your-amazon-quicksight-asset-deployments-using-the-new-apis/14402