Update boto3 version

You can upgrade the boto3 version with below steps.

1. Upload boto3 wheel file to your S3 bucket. Boto3 wheel file is available in pypi.org. (https://pypi.org/project/boto3/#files)
2. Configure your Glue Python shell job with specifying the wheel file S3 path in 'Python library path' in the job configuration.
3. Insert below codes at the beginning of your python script. (The print statements can obviously be omitted)

import sys
sys.path.insert(0, '/glue/lib/installation')
keys = [k for k in sys.modules.keys() if 'boto' in k]
for k in keys:
    if 'boto' in k:
       del sys.modules[k]

import boto3
print('boto3 version')
print(boto3.__version__)

4. Then you can import boto3 and start scripting with newer boto3.
   For example, you can use Athena ListDataCatalogs which is not available in default boto3 yet.

athena = boto3.client("athena")
res = athena.list_data_catalogs()

Edited by: NoritakaS-AWS on Oct 22, 2020 9:59 PM

In case anyone else ends up on this thread, Glue v2.0 simplified this process drastically with the addition of the --additional-python-modules parameter. See

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#aws-glue-programming-python-libraries-glue-20

https://docs.aws.amazon.com/glue/latest/dg/reduced-start-times-spark-etl-jobs.html

The following job parameter made the Athena client functions in boto3 v1.17.12 available without any need to add extra code modifying the python path.

"--additional-python-modules", "botocore>=1.20.12,boto3>=1.17.12"

This doesn't work for some reason...

After importing the newly updated boto3 library, and checking the version:

print(boto3.__version__)

"1.17.9" is printed

But when I try to access the list_data_catalogs() method:

res = athena.list_data_catalogs()

I receive the following error: "AttributeError: module 'boto3' has no attribute 'list_data_catalogs'"

Edited by: g-crmf on Feb 23, 2021 11:43 AM

Edited by: g-crmf on Feb 23, 2021 11:43 AM