Prompt engineering: Difference between revisions

===Basic prompt structure===

To enhance the quality of the output images, it is common to include specific keywords before and after the image description following the formula prompt = [keyword1, . . . , keywordm−1] [description] [keywordm, . . . , keywordn]. For example, a user wanting to generate an image of a cat using a text-to-image model may use a specific prompt template that includes a description of a painting of a calico cat and keywords such as highly detailed, cinematic lighting, dramatic atmosphere, and others. This approach helps to provide additional information to the model and improve the generated image's quality. <ref name="”8”">Pavlichenko, N, Zhdanov and Ustalov, D (2022) Best Prompts for Text-to-Image Models and How to Find Them. arXiv:2209.11711v2</ref>

*'''Prompt engineering in Human-Computer Interaction (HCI):''' a research area that is gaining interest due to the increasing use of deep generative models by people without technical expertise. Social aspects of prompt engineering are important since text-to-image systems were trained on images and text scraped from the web. Prompt engineers need to predict how others described and reacted to the images posted on the web, making describing an image in detail often not enough. There are also dedicated communities that have recently emerged, adding another social aspect to prompt engineering.

*'''Bias:''' an interesting area for future work is bias encoded in text-to-image generation systems.

*'''Computational aesthetics and Human-AI alignment:''' Making computers evaluate and understand aesthetics is an old goal that has recently received renewed attention. Computational aesthetics and Human-AI alignment are areas of research that are being explored through neural image assessment and computational aesthetics. <ref name="”1”"></ref>

*'''Capacity and Role:''' "Act as an expert on software development on the topic of machine learning frameworks, and an expert blog writer."

*'''Statement:''' "Provide a comprehensive overview of the most popular machine learning frameworks, including their strengths and weaknesses. Include real-life examples and case studies to illustrate how these frameworks have been successfully used in various industries."

*'''Personality:''' "When responding, use a mix of the writing styles of Andrej Karpathy, Francois Chollet, Jeremy Howard, and Yann LeCun."

The process of prompt refinement is a method to improve the quality of written content by transforming it into a compelling, imaginative, and relatable piece, fixing "soulless writing". The aim is to make the content engaging and impactful by focusing on storytelling, using persuasive language, emphasizing emotion and sensory details, making the content concise and highlighting key points. To create a sense of urgency and make the content relatable, the language can be personalized to the reader and potential objections can be addressed. <ref name="”11”"></ref>

===Prompt Engineering for Code Generation Models===

Genearte [[code]] using [[models]] like the [[OpenAI Codex]].

However, sometimes the generated code may not be optimal, in which case you can provide more specific instructions such as importing libraries before using them. By combining a high-level task description with detailed user instructions, you can create a more effective prompt for coding model to generate code.

Gives the coding model examples. Imagine you prefer a unique style of writing Python code that differs from what model produces. Take, for instance, when adding two numbers, you prefer to label the arguments differently. The key to working with models like Codex is to clearly communicate what you want it to do. One effective way to do this is to provide examples for Codex to learn from and strive to match its output to your preferred style. If you give the model a longer prompt that includes the example mentioned, it will then name the arguments in the same manner as in the example.

*Edit and revise. Don't be afraid of revising and editing the generated text.

*You can ask the chatbot for assistance. The chatbot will explain why it selected a specific detail or phrase in a reply. The chatbot can also help you create a better prompt. You can point out individual phrases and ask the chatbot for alternatives or suggestions.

<blockquote>

Describe ''YOUR SCENE''. Use sensory language and detail to describe the ''OBJECTS IN THE SCENE vividly''. Describe ''SPECIFIC DETAILS'' and any other sensory details that come to mind. Vary the sentence structure and use figurative language as appropriate. Avoid telling the reader how to feel or think about the scene.

</blockquote>

Text prompts can be used to generate images using a text-to-image model, where words are used to describe an image and the model creates it accordingly. Emojis or single lines of text can also be used as prompts to get optimal results. However, the subject term is important to control the generation of digital images. <ref name="”1”"></ref><ref name="”12”">Zerkova, A (2022). How to Create Effective Prompts for AI Image Generation. Re-thought. https://re-thought.com/how-to-create-effective-prompts-for-ai-image-generation/</ref> In the online community for AI-generated art, templates for writing input prompts have emerged, such as the "Traveler's Guide to the Latent Space," which recommends specific prompt templates such as [Medium][Subject][Artist(s)][Details][Image repository support]. <ref name="”2”"></ref>

The prompt should contain a noun, adjective, and verb to create an interesting subject. A prompt with more than three words should be written to give the AI a clear context. Multiple adjectives should be used to infuse multiple feelings into the artwork. It is also recommended to include the name of the artist, which will mimic the style of that artist. Additionally, banned words by the AI generator should be avoided to prevent being banned. <ref name="”12”"></ref> The use of abstract words leads to more diverse results, while concrete words lead to all pictures showing the same concrete thing. For tokenization (the separation of a text into smaller units—tokens), commas, pipes, or double colons can be used as hard separators, but the direct impact of tokenization is not always clear. <ref name="”13”"></ref>

*'''Style:''' instead of using the style of artists, the prompt can include keywords related to certain styles like “surrealism,” “fantasy,” “contemporary,” “pixel art”, etc. <ref name="”6”"></ref>

*'''Computer graphics:''' keywords like “octane render,” “Unreal Engine,” or “Ray Tracing” can enhance the effectiveness and meaning of the artwork. <ref name="”6”"></ref>

*'''Quality:''' quality of the generated image (e.g. high, 4K, 8K). <ref name="”6”"></ref>

*'''Art medium:''' there is a multitude of art mediums that can be chosen to modify the AI-generated image like “pencil art,” “chalk art,” “ink art,” “watercolor,” “wood,” and others. <ref name="”6”"></ref>

In-depth lists with modifier prompts can be found [https://decentralizedcreator.com/write-good-prompts-for-ai-art-generators/ here] and [https://aesthetics.fandom.com/wiki/List%20of%20Aesthetics here].

===Midjourney===

*'''Style setting:''' adding the command –s <number> after the prompt will increase or decrease the stylize option (e.g. /imagine firefighters --s 6000).

*Resolution: the resolution can be inserted in the prompt or using the standard commands --hd and --quality or --q <number>.

*'''Aspect ratio:''' the default aspect ratio is 1:1. This can be modified with the comman --ar <number: number> (e.g. /imagine jasmine in the wild flower --ar 4:3). A custom size image can also be specified using the command --w <number> --h <number> after the prompt.

*'''Images as prompts:''' Midjourney allows the user to use images to get outputs similar to the one used. This can be done by inserting a URL of the image in the prompt (e.g. /imagine http://www.imgur.com/Im3424.jpg box full of chocolates). Multiple images can be used.

*'''Filter:''' to discard unwanted elements from appearing in the output use the --no <keyword> command (e.g./imagine KFC fried chicken --no sauce). <ref name="”14”"></ref>

Other user experiments can be accessed [https://strikingloo.github.io/DALL-E-2-prompt-guide here]. <ref name="”15”"></ref>

Overall, prompt engineering in [[Stable Diffusion]] doesn’t differ from other AI image-generating models. However, it should be noted that it also allows prompt weighting and negative prompting. <ref name="”16”">DreamStudio. Prompt guide. DreamStudio. https://beta.dreamstudio.ai/prompt-guide</ref>

*'''Prompt weighting:''' varies between 1 and -1. Decimals can be used to reduce a prompt’s influence.

===Jasper Art===

==Research on Prompt engineering==

===Automatic prompt engineering===

Hao et al. (2022) mention that the implementation of manual prompt engineering towards specific text-to-image models can be laborious and sometimes infeasible. The process of manually engineering prompts is often not transferrable between various model versions. Therefore, a systematic way to automatically align user intentions and various model-preferred prompts is necessary. To address this, a prompt adaptation framework for automatic prompt engineering via reinforcement learning was proposed. The method uses supervised fine-tuning on a small collection of manually engineered prompts to initialize the prompt policy network for reinforcement learning. The model is trained by exploring optimized prompts of user inputs, where the training objective is to maximize the reward, which is defined as a combination of relevance scores and aesthetic scores of generated images. The goal of the framework is to automatically perform prompt engineering that generates model-preferred prompts to obtain better output images while preserving the original intentions of the user. <ref name="”4”"></ref>

Jian et al. (2020) proposed two automatic methods to improve the quality and scope of prompts used for querying language models about the existence of a relation. The methods are inspired by previous relation extraction techniques and use either mining-based or paraphrasing-based approaches to generate diverse prompts that are semantically similar to a seed prompt. The authors also investigated lightweight ensemble methods that can combine the answers from different prompts to improve retrieval accuracy for different subject-object pairs. <ref name="”18”">Jiang, Z, Xu, FF, Araki, J and Neubig, G (2020). How Can We Know What Language Models Know? https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know</ref>

Their paper examined the importance of prompts for retrieving factual knowledge from language models and proposed the use of automated techniques to generate diverse and semantically similar prompts. By combining the different prompts, the research shows that factual knowledge retrieval accuracy can be improved by up to 8% compared to manually designed prompts. The proposed methods outperform the traditional manual prompt design approach and the use of the ensemble approach allows for greater flexibility and improved accuracy for different subject-object pairs. <ref name="”18”"></ref>

===Prompt variables===

Prompt engineering for text-to-image generative models is an emerging area of research. Previous studies have used text-to-image models to generate visual blends of concepts. BERT, a large language model, was utilized to help users generate prompts, and generations were evaluated using crowd-source workers on Mechanical Turk. Similar crowd-sourced approaches have been used in the past to evaluate machine-generated images for quality and coherence. <ref name="”3”"></ref>

*Focusing on keywords during prompt engineering rather than rephrasings, as rephrasing does not have a significant impact on the quality of the generation.

*To speed up the iteration process, the user should choose lower lengths of optimization, as the number of iterations and length of optimization do not significantly impact user satisfaction with the generation.

*Users can experiment with a variety of artistic styles to manipulate the aesthetic of their generations, but should avoid style keywords with multiple meanings.

*Considering the interaction between levels of abstraction for the subject and style, as they can lead to incompatible representations. <ref name="”3”"></ref>

===Prompt keyword combinations===

Pavlichenko et al. (2022) aimed to improve the aesthetic appeal of computer-generated images by developing a human-in-the-loop approach that involves human feedback to determine the most effective combination of prompt keywords. In combination with this, they used a genetic algorithm that learned the optimal prompt formulation and keyword combination for generating aesthetically pleasing images. <ref name="”8”"></ref>

===Prompt Modifiers===

Creating a prompt to generate an image can be challenging. The authors propose starting with a clear noun-based statement that contains the main subject of the image. Then, to record what seeds are effective, look for artists and key styles to emulate and add that to the prompt, and experiment with descriptors such as adding lighting effects phrases and repeating words. <ref name="”7”"></ref>

Finally, Oppenlaender (2022) noted that text-to-image art practitioners uses six different types of prompt modifiers to create images of specific subjects in different styles and qualities. These six types of prompt modifiers are subject terms, image prompts, style modifiers, quality boosters, repetitions, and magic terms. <ref name="”2”"></ref>

==Overview of Tones==

*'''[[Authoritative]]''' - confident, knowledgeable,

*'''[[Casual]]''' - relaxed, friendly, playful

*'''[[Conversational]]''' - conversational, engaging,

*'''[[Empathetic]]''' - understanding, caring

*'''[[Expert]]''' - authoritative, respected

*'''[[Funny]]''' - humorous, entertaining

*'''[[Humorous]]''' - entertaining, playful,

*'''[[Informal]]''' and '''[[Humorous]]''' - social media posts, blog content, internal communication

*'''[[Informative]]''' and '''[[Authoritative]]''' - thought leadership articles, industry reports

*'''[[Professional]]''' and '''[[Authoritative]]''' - executive communication, industry presentation, boarding meeting

*'''[[Professional]]''' and '''[[Friendly]]''' - sales emails, customer service, marketing copy

*'''[[Trustworthy]]''' and '''[[Professional]]''' - business proposals, executive summaries, investor pitches

===Common Parameters===

====Introduction====

These are user-created parameters. They serve to convey the intent of the users in a more concise way. These are not part of the [[model]] API but patterns the [[LLM]] has picked up through its [[training]]. These parameters are just a compact way to deliver what is usually expressed in [[natural language]].

*[[Professionalism]] -

*[[Randomness]] -

*[[Sesquipedalianism]] -

*[[Sarcasm]] -