How do I manage and reduce the costs of an AWS Glue interactive session?

Short description

With AWS Glue interactive sessions, you can rapidly build, test, and run data preparation and analytics applications.

Interactive sessions are created when you run Glue notebooks and when you activate data preview in Visual ETL. However, cost increases from interactive sessions occur mostly when you don't stop or manage sessions correctly.

Resolution

Monitor the interactive sessions

Interactive sessions are charged according to the number of DPU hours. For more information, see Change the default timeout for all notebook sessions.

The session history shows in the Interactive Sessions tab in AWS Glue Studio. Follow these steps to check the session history:

1. Log in to your AWS Glue console.
2. In the left pane under Data Integration and ETL, select Interactive Sessions.
   This tab has the session history for sessions activated by Glue notebooks and data preview.
3. Use the following parameters to monitor a session in a session history console:
   Session duration
   Status such as stop or run
   Number of workers
   DPU hours

Note: Use the list of sessions in the history console to check all the sessions that are in a running state. You can open any session for additional details.

Monitor billing

Go to the AWS Billing console and check the Charges by service tab.

Complete the following steps:

1. Log in to your AWS Billing console.
2. Choose the Bills tab in the left pane.
3. Select Charges by service.
4. Choose AWS Glue service.
5. Select the Region name for the AWS Glue service you selected in the preceding step.
6. Choose AWS Glue Interactive Session. Check the Usage quantity and the total billing amount.

Troubleshooting tips

Stop inactive sessions

AWS charges Glue interactive sessions according to the time that the session is active and the number of DPUs incurred (default is 5). List the current sessions and delete unwanted active sessions periodically to save cost. You can stop a session using the magic %stop_session.

To stop unwanted sessions, run the following AWS Command Line Interface (AWS CLI) commands:

Note: Replace my_region with the Region where your account is located.

List current sessions
$ aws glue list-sessions --region my_region 

Delete the sessions
$ aws glue delete-session --id <session_id> --region my_region

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Use idle timeouts

Interactive sessions have configurable idle timeouts and the default timeout value is 2880 minutes (48 hours). You can use %idle_timeout magic in a cell for customizing the timeout value to save costs.

Modify the default session timeout for notebooks in AWS Glue Studio. In the notebook, enter the %idle_timeout magic in a cell and specify the timeout value in minutes. For example: %idle_timeout 15 changes the default timeout to 15 minutes. If the session isn't used in 15 minutes, the session automatically expires.

Automate a stop session

Automate the use case to stop the session. For example, use a rule to set up an AWS Lambda function that monitors the CreateSession API calls within an account. The function sends Amazon Simple Notification Service (Amazon SNS) notifications at set intervals to remind you about the running interactive session.

Reduce user requests

By default, you request five workers when you launch a session. If needed, use a lower number for your use case.

Use interactive sessions only for development purpose

Interactive sessions are useful for development and testing. Use AWS Glue jobs for production workloads.