Can we train Stable diffusion on sagemaker using dreambooth approach?

Yes! Check out https://aws.amazon.com/blogs/machine-learning/fine-tune-text-to-image-stable-diffusion-models-with-amazon-sagemaker-jumpstart/

@dayanand The Dreambooth approach is used in the training scripts, but the training scripts are customized so that they can be called using the Sagemaker SDK.

If you want to walk through it in a notebook, you can use https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart_text_to_image/Amazon_JumpStart_Text_To_Image.ipynb

However, in both cases the training scripts are not shown. They are retrieved from a repository associated with the model as part of the jump start process. The training scripts are where the Dreambooth process happens. You can see that in the notebook where it says:

    # Retrieve the training script. This contains all the necessary files including data processing, model training etc.
    train_source_uri = script_uris.retrieve(
        model_id=train_model_id, model_version=train_model_version, script_scope=train_scope)

What is it that you are looking for in the fine tuning process that you are not seeing?

@Dayanand

The way the Dreambooth finetuning process works is that you can provide multiple images of an object/person that you want to add to the model. The foundational model may be able to generate pictures of a celebrity that it "knows" about, but not pictures of me (Burtoft). In the examples above, you need a directory with some photos cropped on the thing to be trained (5 or so usually work), and a json file that describes the thing.

In the finetuning process, there are multiple photo examples of a single text description (instance) that is part of a class that already exists (e.g. humans or cats). If you are talking about a longer text description of a single photo, I haven't heard of anything that can do that. However, depending on the use case, you may be able to crop parts of your existing photos to create smaller training photos. If your description was "Burtoft on a model xyz chair in a type abc suit", you could train on Burtoft (human), model xyz(chair) and type abc (suit) if you can provide multiple photos of each.

I fine tuned Stable Diffusion to generate pictures of me by using a json that looked like this:

{"instance_prompt": "a photo of a burtoft human"
,"class_prompt": "a photo of a human"
}

(in my case, my name is pretty unique, so I could use that) I included that json in a directory that had (cropped) photos of me in it. You can also see an example in the notebook above in the S3 bucket jumpstart-cache-prod-{aws_region}/training-datasets/dogs_sd_finetuning/

If that doesn't answer your question, please provide some details around the photos and descriptions you are trying to fine tune.

@Dayanand In your example, the sofas would be the class and the different names would each be an instance. Fine tuning multiple instances of the same class can be challenging with the current sagemaker scripts. In this article look at the section on "Train on multiple datasets"

Unfortunately, JumpStart is currently limited to training on a single subject. 
You can’t fine-tune the model on multiple subjects at the same time. 
Furthermore, fine-tuning the model for different subjects sequentially 
results in the model forgetting the first subject if the subjects are similar.

If you look in that article on the section on Cats and Dogs, fine tuning for different types of dogs "overwrites" the previous dog training, but training for a cat and dog can work. In your example, you may be able to train for a sofa, a table, and a lamp of specific types, but if you want to change the type of lamp you would need to train another model.

For a "universal" model that would let you generate photos of different combinations of lamps and tables, you would need to bring your own code to do a different type of training that isn't available in sagemaker jumpstart.

@Dayanand The Sagemaker fine tuning process has already converted those scripts.

If you take a look at the Hyperparameter section in the jumpstart blog you will see a lot of the same parameters in the python script you shared.

You will have to check the parameters available in the Sagemaker training scripts. If you don't see what you want in the article, you can check the metadata from the model. Go through the notebook and print these hyperparameters to see what else can be passed in:

hyperparameters = hyperparameters.retrieve_default(
model_id=train_model_id, model_version=train_model_version
)

(additional hyperparameters may be available since the article was written)

You should walk through this notebook.