Troubleshooting Amazon FinSpace with Managed kdb Insights

A good place to start with Amazon FinSpace is to follow the Amazon FinSpace Foundation Workshop to setup a basic FinSpace managed kdb Insights environment.

Environment Creation Issues

Environment creation issues will generally be due to permissions of the Role used when creating the environment, be sure to review all the required permissions in the documentationPermissions required for Managed kdb.

Database Creation Issues

Common issues experienced by customers creating a new database have been related to using a name that already exists (database names must be unique to an environment) and using characters in the name that are not allowed you can read more in the documentation of the API CreateKxDatabase.

Permissions: Check that the role used when calling the CreateKxDatabase API is allowed to create a database in the environment.

Changeset Creation Issues

Changeset creation is an asynchronous operation triggered by the CreateKxChangeset API, one requests changeset creation and then waits for the operation to complete. Creation issues can occur synchronously (call to API returns an error) or asynchronously (API call succeeds but creation fails instead of succeeds). You can check the status of the cluster after calling the create API with the GetKxCluster API.

Permissions

Issues creating changesets are almost always due to insufficient permissions. First, the role used to create the changeset must be allowed to add changesets to the database. Second, the S3 location of the changeset data (as identified in the CreateKxChangeset API ) has allowed the environment’s FinSpace service principal to access the S3 location as well. If the S3 bucket uses a KMS key, be sure the FinSpace service principal also has access to use the key used to encrypt the S3 bucket (kms:Decrypt and kms:DescribeKey).

You can see example permissions in the FinSpace documentation: Permissions required for Managed kdb

Changeset List Limit

Changesets have a limit in the length of the list of items in the changeset request (changeRequests argument), be sure to keep the list under 32 items.

Multiple Changsets Added to Same Database at Same Time

One cannot have more than one changeset being added at a time to any given database. Once you have called CreateKxChangeset to add a changeset to a given database, poll the status of the changeset with GetKxChangeset and be sure it completes before adding another changeset.

Scaling Group Creation Issues

Scaling group creation using CreateKxScalingGroup is an asynchronous operation, one requests creation and then waits for the operation to complete. Creation issues can occur synchronously (API call to create returns an error) or asynchronously (creation fails instead of succeeds). Use the function GetKxScalingGroup to get the creation status of the scaling group.

Common scaling group creation issues will be quota or permission related. For quotas, request an increase in the quota of a scaling group node type. For permissions, be sure the role being used to create the scaling group has the permission to create scaling groups in the account’s FinSpace environment.

Volume Creation Issues

Volume creation using CreateKxVolume is an asynchronous operation, one requests creation and then waits for the operation to complete. Creation issues can occur synchronously (API call to create returns an error) or asynchronously (creation fails instead of succeeds). Use the function GetKxVolume to get the creation status of the volume.

Common scaling group creation issues will be quote or permission related. For quota related issues, request limit increase, such as the volume size. For permissions, be sure the role being used to create the volume has the permission to create the volume in the account’s FinSpace enviroment.

Cluster Creation Issues

Cluster creation using CreateKxCluster is an asynchronous operation, one requests creation and then waits for the operation to complete. Creation issues can occur synchronously (API call to create returns an error) or asynchronously (creation fails instead of succeeds). Use the function GetKxCluster to get the creation status of the cluster.

If a cluster uses a scaling group or volume, be sure those resources have successfully completed creation before using them in the CreateKxCluster call.

Synchronous Errors

Synchronous errors will return in the API call, the call will fail and return an error. Most of these errors should be self explanatory such as bad cluster name or unknown networking parameters. Since a KX environment can be created before a KX license has been linked to the account, cluster creation can also fail due to no KX license being linked to the account. See Activating your Managed kdb Insights license in the documentation for more information

Asynchronous Errors

Asynchronous errors occur after the create API call, where the cluster enters CREATING state before completing in a CREATE_FAILED state (if polling the status, one would stop polling if state reaches CREATE_FAILED or COMPLETED states).

Asynchronous errors can be due to multiple issues, and troubleshooting them should start with getting the cluster status with the GetKxCluster API and reviewing the statusReason in the response.

Common Failure Root Causes (see Permissions required for Managed kdb for details)

• S3 Access
  • Does FinSpace service principal have the s3:GetObject permission to access the S3 bucket and kms:Decrypt permission for the key used to encrypt the S3 bucket?
• Network resource access such as endpoints and subnets
  • Permisions such as ec2:CreatevpcEndpoint, ec2:DeleteVpcEndpoints, and ec2:DescribeSubnets
• CloudWatch resource access
  • Permisions such as logs:CreateLogDelivery and logs:GetLogDelivery
• Tag access
  • Permissions such as ec2:DescribeTags and ec2:CreateTags

Cluster Connection Issues

Cluster connections have two parts to them, getting the connection string (a signed URL) to the cluster’s Privatelink VPC endpoint using the GetKxConnectionString API and the network part, being able to reach the cluster endpoint from the q client initiating the connection to the cluster. It should also be noted that clusters can also connect to each other, where issues will also depend on permissions to successfully call the q functions .aws.get_connection_string.

GetKxConnectionString

Connecting to a cluster is covered in the documentation section Connecting to a cluster endpoint or node in a cluster. Common connection string issues will first involve permissions, does the caller of the API have sufficient permissions to get a connection string to the cluster’s Privatelink VPC endpoint?

Network Issues

If the client machine connecting to a running cluster (the one calling hopen from q for example) times out, this is likely a network routing issue between the client and the account’s VPC where the FinSpace cluster Privatelink VPC endpoint was created. When creating a cluster with CreateKxCluster you pass networking argument vpcConfiguration and these are used in creating a Privatelink VPC endpoint to the created cluster. Be sure the client can reach the given vpc over the network.

Cluster Permission Issues

Clusters need permission to perform service related operations. Functions on the cluster that call AWS services will use the cluster’s execution role calling AWS services.

Note the permissions tab for functions listed in the FinSpace q API Reference, ensure the cluster execution role has the identified permissions if q code run from a cluster is expected to use a FinSpace q API.

Example: .aws.s3.get_object

Be sure that an execution role identified when creating the cluster using the API CreateKxCluster by the executionRole argument has the s3:GetObject permission to access the object and kms:Decrypt permission for the key used to encrypt the S3 bucket.

Example: .aws.get_kx_node_connection_string and .aws.get_kx_connection_string

Be sure that an execution role identified when creating the cluster using the API CreateKxCluster by the executionRole argument has the finspace:GetKxConnectionString and finspace:ConnectKxCluster permissions for the FinSpace environment and cluster being connected to.