Stable diffusion finetune dataset prepration

0

I am working on finetuning the stable diffusion model. I have 5 images and I want to pass on all the images information how to create dataset_info.json

I have tried this

{ "instance_prompt": { "instance_image_1.png": "Instance prompt for image 1", "instance_image_2.png": "Instance prompt for image 2", "instance_image_3.png": "Instance prompt for image 3", "instance_image_4.png": "Instance prompt for image 4", "instance_image_5.png": "Instance prompt for image 5" } }

but it's not working. Please give me the solution

1개 답변
1

Hi there,

instance_prompt is the prompt we would type to generate the image we are attempting to fine tune, while class_prompt denotes a prompt without the unique identifier/instance. This prompt is used for generating "class images" for prior preservation.

The dataset_info.json file is used in training to pass these arguments, and must be of format:

{
	'instance_prompt':<<instance_prompt>>,
	'class_prompt':<<class_prompt>>
}

So for example, if you're fine tuning the model with the face of a specific person, the dataset_info.json file could look like this:

{
	"instance_prompt": "a photo of a zwx person",
	"class_prompt": "a photo of a person"
}

Where 'zwx' would be the unique identifier that is used to train the model.

See here for more details.

Hope this helps

profile pictureAWS
jnavrro
답변함 일 년 전
  • can you tell me how to write for multiple images

    { "instance_prompt": "a photo of a zwx person", "class_prompt": "a photo of a person" }

  • The group of images is used to train the model on the same instance prompt, not one prompt per image.

  • what about if we want to train the model on different variety of images and provide different prompt for different categories/ type of images

    { "instance_prompt": { "coco_set.jpg": "wooden shelf with modern style", "coco_sets_with_two_mirror.jpg": "wooden shelf with modern style", "maserati_black.jpg": "The Maserati chair with the black color is perfect for the study room", "maserati_gray.jpg": "The Maserati chair with the gray color is perfect for the living room", "maserati_white.jpg": "The Maserati chair with the white color is perfect for the living room", "panda_left_facing.jpg": "x chair with left facing with white color.", "panda_orange_chair.jpg": "hallowing chair with orange color is perfect for modern style.", "panda_white_chair.jpg": "wooden chair with white color is perfect for modern style.", "toronto_bedstead.jpg": "grey-bedstead with minimalistic style.", "toronto_bedstead1.jpg": "bedstead with minimalistic style." } } is this way correct?

    how to train multiple category or types of images and with different prompts

  • No, it's not correct. In the dataset_info.json file you must pass a single instance_prompt and class_prompt for all the images in the input directory, as you need more than one image to fine-tune the model on one subject. Moreover, you should specify a unique identifier in the instance prompt for the model to be able to distinguish it. Your dataset_info.json file should look similar to the one included in the answer as example.

    Please refer to this blogpost (https://aws.amazon.com/blogs/machine-learning/fine-tune-text-to-image-stable-diffusion-models-with-amazon-sagemaker-jumpstart/) and DreamBooth paper (https://dreambooth.github.io/) for a more in-depth explanation.

    As explained in the blogpost, if you are using JumpStart, it is currently limited to training on a single subject. You can’t fine-tune the model on multiple subjects at the same time. Furthermore, fine-tuning the model for different subjects sequentially results in the model forgetting the first subject if the subjects are similar. To achieve training on multiple subjects, you should:

    1. Fine-tune the model for Subject A.
    2. Fine-tune the resulting model from Step 1 for Subject B.
    3. Generate images of Subject A and Subject B using the output model from Step 2.
  • Hello jnavrro,

    I would like to express my gratitude for your assistance. Your guidance has been truly valuable.

    I'm currently seeking alternative methods to train a stable diffusion model across numerous images and categories, similar to the approach used in Dreambooth. I'm curious if this can be achieved through platforms like Jumpstart or SageMaker. Additionally, I'm interested in exploring the possibility of implementing the Dreambooth approach within the SageMaker framework for training on multiple images.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠