diff --git a/README.md b/README.md index 5c7b911c2ede..58dc5be8d0d5 100644 --- a/README.md +++ b/README.md @@ -353,7 +353,8 @@ Textual Inversion is a technique for capturing novel concepts from a small numbe ## Stable Diffusion Community Pipelines -The release of Stable Diffusion as an open source model has fostered a lot of interesting ideas and experimentation. Our [Community Examples folder](https://github.com/huggingface/diffusers/tree/main/examples/community) contains many ideas worth exploring, like interpolating to create animated videos, using CLIP Guidance for additional prompt fidelity, term weighting, and much more! [Take a look](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview) and [contribute your own](https://huggingface.co/docs/diffusers/using-diffusers/contribute_pipeline). +The release of Stable Diffusion as an open source model has fostered a lot of interesting ideas and experimentation. +Our [Community Examples folder](https://github.com/huggingface/diffusers/tree/main/examples/community) contains many ideas worth exploring, like interpolating to create animated videos, using CLIP Guidance for additional prompt fidelity, term weighting, and much more! [Take a look](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview) and [contribute your own](https://huggingface.co/docs/diffusers/using-diffusers/contribute_pipeline). ## Other Examples @@ -402,10 +403,14 @@ image.save("ddpm_generated_image.png") - [Unconditional Latent Diffusion](https://huggingface.co/CompVis/ldm-celebahq-256) - [Unconditional Diffusion with continuous scheduler](https://huggingface.co/google/ncsnpp-ffhq-1024) -**Other Notebooks**: +**Other Image Notebooks**: * [image-to-image generation with Stable Diffusion](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), * [tweak images via repeated Stable Diffusion seeds](https://colab.research.google.com/github/pcuenca/diffusers-examples/blob/main/notebooks/stable-diffusion-seeds.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +**Diffusers for Other Modalities**: +* [Molecule conformation generation](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), + ### Web Demos If you just want to play around with some web demos, you can try out the following 🚀 Spaces: | Model | Hugging Face Spaces | diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index d8efb5eee3d3..efbcbf04822d 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -29,6 +29,14 @@ - local: using-diffusers/contribute_pipeline title: "How to contribute a Pipeline" title: "Pipelines for Inference" + - sections: + - local: using-diffusers/rl + title: "Reinforcement Learning" + - local: using-diffusers/audio + title: "Audio" + - local: using-diffusers/other-modalities + title: "Other Modalities" + title: "Taking Diffusers Beyond Images" title: "Using Diffusers" - sections: - local: optimization/fp16 @@ -103,4 +111,8 @@ - local: api/pipelines/repaint title: "RePaint" title: "Pipelines" + - sections: + - local: api/experimental/rl + title: "RL Planning" + title: "Experimental Features" title: "API" diff --git a/docs/source/api/experimental/rl.mdx b/docs/source/api/experimental/rl.mdx new file mode 100644 index 000000000000..65abb06e7523 --- /dev/null +++ b/docs/source/api/experimental/rl.mdx @@ -0,0 +1,15 @@ + + +# TODO + +Coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/audio.mdx b/docs/source/using-diffusers/audio.mdx new file mode 100644 index 000000000000..5a5c2241ca75 --- /dev/null +++ b/docs/source/using-diffusers/audio.mdx @@ -0,0 +1,16 @@ + + +# Using Diffusers for audio + +The [`DanceDiffusionPipeline`] can be used to generate audio rapidly! +More coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/other-modalities.mdx b/docs/source/using-diffusers/other-modalities.mdx new file mode 100644 index 000000000000..1dc0877adb24 --- /dev/null +++ b/docs/source/using-diffusers/other-modalities.mdx @@ -0,0 +1,20 @@ + + +# Using Diffusers with other modalities + +Diffusers is in the process of expanding to modalities other than images. + +Currently, one example is for [molecule conformation](https://www.nature.com/subjects/molecular-conformation#:~:text=Definition,to%20changes%20in%20their%20environment.) generation. +* Generate conformations in Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) + +More coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/rl.mdx b/docs/source/using-diffusers/rl.mdx new file mode 100644 index 000000000000..6e18e07001b6 --- /dev/null +++ b/docs/source/using-diffusers/rl.mdx @@ -0,0 +1,18 @@ + + +# Using Diffusers for reinforcement learning + +Support for one RL model and related pipelines is included in the `experimental` source of diffusers. + +To try some of this in colab, please look at the following example: +* Model-based reinforcement learning on Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)