Configuring compute in Databricks

To configure compute for AutoML in Databricks, follow these steps:

1. Once your Databricks workspace is ready, select Compute from the navigation pane.

2. Choose Create compute to create a new cluster.

3. Configure your compute resources according to your needs. For AutoML workloads, you'll want to consider:

• Selecting an appropriate runtime version (such as Runtime 15.4 LTS which includes Apache Spark 3.5.0)
• Setting the access mode (e.g., "No isolation shared")
• Configuring the instance profile role if needed

4. You can set the cluster to terminate after a period of inactivity (e.g., 10 minutes) to avoid unnecessary costs.

5. For specific AutoML requirements, you may need to configure additional Spark parameters under the Advanced options section.

6. Start the compute and wait for the cluster to show a running state before using it for your AutoML workloads.

The configuration options you select should align with the size and complexity of your AutoML tasks. For larger datasets, you'll want to ensure you have sufficient compute resources allocated.
Sources
Start using Databricks Data Intelligence Platform with AWS Marketplace | AWS Marketplace
Access Apache Iceberg tables in Amazon S3 from Databricks using AWS Glue Iceberg REST Catalog in Amazon SageMaker Lakehouse | AWS Big Data Blog