Fine-tune LLAMA2 with DPO (Direct Preference Optimization) in AWS

I'm exploring fine-tuning with DPO and successfully trained facebook/opt-model (HF model) with DPO (Ref: https://huggingface.co/blog/dpo-trl). As part of DPO training, I first performed SFT training, and using the final checkpoint I performed DPO training.

Now, I'm working on fine-tuning Llama2 with DPO in AWS. I have successfully fine-tuned Llama2 in AWS SageMaker Jumpstart, but stuck there figuring out how to perform DPO using the fine-tuned model artifact which is stored in S3 bucket.

It would be helpful if anyone could share some resources or insights on how to proceed DPO training in AWS. Thanks in advance!