How to create a AWS Glue Connector for data sources in VMware Cloud on AWS

3 minute read
Content level: Intermediate
1

This article will help you setup a AWS Glue Connector with data sources that are running in VMware Cloud on AWS

In this article, I will show you how to create AWS Glue Connections that will provide connectivity between data sources (eg. databases) in VMware Cloud on AWS and AWS Glue.

This becomes useful when you want to use AWS Glue with VMware Cloud on AWS as a data source. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.

Enter image description here

Prerequisites

Method

  1. Open the AWS Console, open the AWS Glue Service
  2. . Select Connections from the AWS Glue menu

Enter image description here

  1. Select Create connection
  2. Fill in the Connection properties (only select "Require SSL" if you have configured SSL on the data source)

Enter image description here

  1. Fill in the Connection access properties. (If you are using AWS Secrets Manager you can can select the secret from the drop down, else you can use username / password)
  2. JDBC Driver Class and JDBC Driver S3 Path is optional.

Enter image description here

  1. Fill in the Network options, this says optional but it is essential if you are using VMware Cloud on AWS
  2. Select the correct VPC and Subnet (these will need to have connectivity / be able to route to your VMware Cloud on AWS data source)
  3. For the Security Group see step 10

Enter image description here

  1. To enable AWS Glue to communicate between its components, either select or create a security group with a self-referencing inbound rule for all TCP ports.
  2. Add a self-referencing inbound rule to allow AWS Glue components to communicate. Specifically, add or confirm that there is a rule of Type All TCP, Protocol is TCP, Port Range includes all ports, and whose Source is the same security group name as the Group ID

Enter image description here

  1. Add a rule to for outbound traffic also. Either open outbound traffic to all ports, or create a self-referencing rule of Type All TCP, Protocol is TCP, Port Range includes all ports, and whose Source is the same security group name as the Group ID
  2. Select Create connection
  3. One the connection is created, we need to test it for connectivity
  4. Select the Connection you have just created, select the Actions drop down and select Test connection

Enter image description here

  1. Select an IAM role, I have used the AWSGlueServiceRole, select Confirm

Enter image description here

  1. This should take 1 - 2 mins to test connectivity

Now that you have your AWS Glue Connection created, you can start to create AWS Glue Crawlers and AWS Glue Jobs.

AWS
EXPERT
published 4 months ago593 views