SAGEMAKER Algorithm error

Question

Please help me to resolve below error-
AlgorithmError: ExecuteUserScriptError: ExitCode 1 ErrorMessage "ZeroDivisionError: float division by zero [2025-10-25 10:08:55,514] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 68) of binary: /opt/conda/bin/python3.10 Traceback (most recent call last) File "/opt/conda/bin/torchrun", line 8, in  sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent

**Parameters used-**
ErrorMessage "ZeroDivisionError: float division by zero [2025-10-25 10:08:55,514] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 68) of binary: /opt/conda/bin/python3.10 Traceback (most recent call last) File "/opt/conda/bin/torchrun", line 8, in  sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError ============================================================ llama_finetuning.py FAILED ------------------------------------------------------------ Failures  Root Cause (first observed failure) [0] time : 2025-10-25_10:08:55 host : algo-1 rank : 0 (local_rank: 0) exitcode : 1 (pid: 68) error_file:  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ERROR:root:Subprocess script failed with return code: 1 File "/opt/conda/lib/python3.10/site-packages/sagemaker_jumpstart_script_utilities/subprocess.py", line 9, in run_with_error_handling subprocess.run(command, shell=shell, check=True) File "/opt/conda/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '1', 'llama_finetuning.py', '--model_name', '/opt/ml/additonals3data', '--num_gpus', '1', '--pure_bf16', '--dist_checkpoint_root_folder', 'model_checkpoints', '--dist_checkpoint_folder', 'fine-tuned', '--batch_size_training', '32', '--micro_batch_size', '32', '--train_file', '/opt/ml/input/data/training', '--lr', '0.001', '--do_train', '--output_dir', 'saved_peft_model', '--num_epochs', '3', '--use_peft', '--peft_method', 'lora', '--max_train_samples', '-1', '--max_val_samples', '-1', '--seed', '10', '--per_device_eval_batch_size', '32', '--max_input_length', '-1', '--preprocessing_num_workers', '--None', '--validation_split_ratio', '0.2', '--train_data_split_seed', '0', '--num_workers_dataloader', '0', '--weight_decay', '0.1', '--lora_r', '8', '--lora_alpha', '16', '--lora_dropout', '0.05', '--target_modules', 'q_proj,v_proj', '--chat_template', 'Llama3.1', '--enable_fsdp', '--add_input_output_demarcation_key', '--instruction_tuned']' returned non-zero exit status 1. During handling of the above exception, another exception occurred File "/opt/ml/code/transfer_learning.py", line 175, in  subprocess.run_with_error_handling(command) File "/opt/conda/lib/python3.10/site-packages/sagemaker_jumpstart_script_utilities/subprocess.py", line 12, in run_with_error_handling raise RuntimeError(e) RuntimeError

re:Post Agent · Answer

Based on the error message, you're encountering a "ZeroDivisionError: float division by zero" in your SageMaker training job i believe.

In your specific case, this is happening within your training script (llama_finetuning.py) during distributed training with PyTorch.

To resolve this issue:

1. Review your CloudWatch logs to find the exact location in your code where the division by zero is occurring. The error stack trace should point to the specific line in your code.

2. If you're using distributed training features, ensure that your weight initialisation and data distribution across workers is properly configured.

We might need to check additional details in order to troubleshoot the issue and conduct a specific investigation into your concern, providing recommendations for the next steps. So, I request you to please reach out to AWS Support [3] (SageMaker) along with your issue by providing following details.

The cloudwatch logs details, How you are fine tuning the model and steps to replicate the scenario.

**Sources**  
[Troubleshoot algorithm errors in SageMaker AI processing job | AWS re:Post](https://repost.aws/knowledge-center/sagemaker-troubleshoot-algorithm-error)  
[SageMaker Python SDK Troubleshooting Guide - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-python-sdk-troubleshooting.html)
[Creating support cases and case management] - (https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case)

SAGEMAKER Algorithm error

Add your answer

Relevant content