JumpStart Llama2 Finetuning - Recurring Runtime Error

Question

Hi,

I'm using a small dataset to finetune a Llama2 model. Each time I try to train via JumpStart, I'm stalling out with the same error:

We encountered an error while training the model on your data. AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "raise RuntimeError(
 RuntimeError: Invalid function argument. Expected parameter `tensor` to be of type torch.Tensor.
 Traceback (most recent call last)
 File "/opt/ml/code/llama_finetuning.py", line 301, in 
 fire.Fire(main)
 File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
 component_trace = _Fire(component, args, parsed_flag_args, context, name)
 File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
 component, remaining_args = _CallAndUpdateTrace(
 File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
 component = fn(*varargs, **kwargs)
 File "/opt/ml/code/llama_finetuning.py", line 281, in main
 results = train(
 File "/opt/ml/code/llama-recipes/utils/train_utils.py", line 117, in train
 dist.all_reduce(total_loss, op=dist.ReduceOp.SUM)
 File "/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

Can anyone suggest what I may be doing wrong here?

Saad · Answer

This error indicates there is an issue with passing invalid data types to PyTorch distributed training functions when using multiple GPUs.

Specifically, it looks like you are passing a non-Tensor object to dist.all_reduce() which is expecting a PyTorch Tensor.

Some things to check:

- Make sure total_loss is a PyTorch Tensor before passing to dist.all_reduce()

- Verify the data types and shapes of all inputs to the model forward pass match what the model expects

- Print or log the types and shapes of inputs at various points to validate they are Tensors

- Try training on a single GPU first to isolate if it is a distributed training issue

- Ensure the latest compatible versions of PyTorch, TorchDistributed, and other libraries are installed

The key is tracing back to find where a non-Tensor input is being passed and handling that appropriately. Adding more logging or printing can help identify the source. And simplifying to single GPU training can determine if it is a distributed training bug.

Let me know if this helps point you in the right direction or if any other details would help troubleshoot further!

JumpStart Llama2 Finetuning - Recurring Runtime Error

回答を追加する

関連するコンテンツ