AWS Glue: local dev container and useGlueParquetWriter not working

The issue you're experiencing is due to a limitation of local development with AWS Glue. When developing AWS Glue job scripts locally, certain features are not available or supported, and the AWS Glue Parquet writer is one of them.

The AWS Glue Parquet writer is a feature that is only available within the AWS Glue job system on AWS, and it cannot be used in local development environments, including Docker containers. This limitation exists because some AWS Glue-specific optimizations and integrations are not possible to replicate in a local environment.

To work around this issue in your local development setup, you have a few options:

1. Use a standard Spark Parquet writer instead of the Glue-specific one. This should work in your local environment, although it may not have all the optimizations of the AWS Glue version.

2. For testing purposes, you can use a different file format that is supported in the local environment, such as CSV or JSON.

3. If you absolutely need to test with the Glue Parquet writer, you'll have to deploy and run your job in the actual AWS Glue environment.

Remember that local development for AWS Glue has some restrictions, and certain features like job bookmarks, the AWS Glue Parquet writer, and some transforms are only available within the AWS Glue job system. These limitations are in place because these features rely on AWS-specific integrations that can't be fully replicated locally.

For the most accurate testing of AWS Glue jobs, especially when using AWS Glue-specific features like the Parquet writer, it's recommended to test in the actual AWS environment. However, local development is still valuable for rapid iteration and basic functionality testing.
Sources
Developing and testing AWS Glue job scripts locally - AWS Glue