The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR25)

Abstract

CLICK for the full abstract

The evolution of Text-to-video (T2V) generative models, trained on large-scale datasets, has been marked by significant progress. However, the sensitivity of T2V generative models to input prompts highlights the critical role of prompt design in influencing generative outcomes. Prior research has predominantly relied on Large Language Models (LLMs) to align user-provided prompts with the distribution of training prompts, albeit without tailored guidance encompassing prompt vocabulary and sentence structure nuances. To this end, we introduce RAPO, a novel Retrieval- Augmented Prompt Optimization framework. In order to address potential inaccuracies and ambiguous details generated by LLM-generated prompts. RAPO refines the naive prompts through dual optimization branches, selecting thesuperior prompt for T2V generation. The first branch augments user prompts with diverse modifiers extracted from a learned relational graph, refining them to align with the format of training prompts via a fine-tuned LLM. Conversely, the second branch rewrites the naive prompt using a pre-trained LLM following a well-defined instruction set. Extensive experiments demonstrate that RAPO can effectively enhance both the static and dynamic dimensions of generated videos, demonstrating the significance of prompt optimization for user-provided prompts.

📥 Installation

1. Clone the Repository

git clone https://github.com/Vchitect/RAPO.git
cd RAPO

2. Set up Environment

conda create -n RAPO python=3.10
conda activate RAPO
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

🤗 Checkpoint

Download the required model weights RAPO, relation graph and pretrained LLM (e.g. , Mistral-7B-Instruct-v0.3 )and place them in the ckpt/ and relation_graph/ directory.

ckpt/
│── all-MiniLM-L6-v2/
│── llama3_1_instruct_lora_rewrite/
│── Mistral-7B-Instruct-v0.3/
relation_graph/
│── graph_data/

🖥️ Inference

1. Retrieve related modifiers from relation graph. You can adjust the hyperparameters in retrieve_modifiers.py to modify the number of retrieval modifiers.

sh retrieve_modifiers.sh

2. Word augmentation and sentence refactoring.

sh word_augment.sh
sh refactoring.sh

3. Rewrite via instruction.

sh rewrite_via_instruction.sh

✒️ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝

@article{gao2025devil,
  title={The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation},
  author={Gao, Bingjie and Gao, Xinyu and Wu, Xiaoxue and Zhou, Yujie and Qiao, Yu and Niu, Li and Chen, Xinyuan and Wang, Yaohui},
  journal={arXiv preprint arXiv:2504.11739},
  year={2025}
}