Prompt engineering

See also: Prompts, Prompt engineering for image generation and Prompt engineering for text generation

Introduction

Prompt engineering, also known as In-context prompting, is an emerging research area within Human-Computer Interaction (HCI) that involves the formal search for prompts to produce desired outcomes from AI models. Prompt engineering involves techniques that guide the behavior of Large language models (LLMs) towards specific goals without modifying the model's weights. As an experimental discipline, the impact of these prompting strategies can differ significantly across various models, requiring extensive trial and error along with heuristic approaches. This process involves selecting and composing sentences to achieve a certain result, such as a specific visual style in text-to-image models or a different tone in the response of a text-to-text one. Unlike the hard sciences of STEM fields, this is an evolving technique based on trial and error to produce effective AI outcomes. Prompt engineers serve as translators between "human language" and "AI language," transforming an idea into words that the AI model can comprehend.

The process of prompt engineering is similar to a conversation with the generative system, with practitioners adapting and refining prompts to improve outcomes. It has emerged as a new form of interaction with models that have learned complex abstractions from consuming large amounts of data from the internet. These models have metalearning capabilities and can adapt their abstractions on the fly to fit new tasks, making it necessary to prompt them with specific knowledge and abstractions to perform well on new tasks. The term "prompt engineering" was coined by Gwern (writer and technologist), who evaluated GPT3's capabilities on creative fiction and suggested that a new course of interaction would be to figure out how to prompt the model to elicit specific knowledge and abstractions.

In order to get the best results from these large and powerful generative models, prompt engineering is a critical skill that users must possess. Adding certain keywords and phrases to the textual input prompts known as "prompt modifiers" can improve the aesthetic qualities and subjective attractiveness of the generated images, for example. The process of prompt engineering is iterative and experimental in nature, where practitioners formulate prompts as probes into the generative models' latent space. There are various resources and guides available to novices to help them write effective input prompts for text-to-image generation systems, however, prompt engineering is still an emerging practice that requires extensive experimentation and trial and error.

Manual prompt engineering is laborious, it may be infeasible in some situations, and the prompt results may vary between various model versions. However, there have been developments in automated prompt generation which rephrases the input, making it more model-friendly.

Therefore, this field is important for the generation of high-quality AI-generated outputs. Text-to-image models, in particular, face limitations in their text encoders, making prompt design even more crucial to produce aesthetically pleasing images with current models. These models work based on caption matching techniques and are pre-trained using millions of text-image datasets. While a result will be generated for any prompt, the quality of the artwork is directly proportional to the quality of the prompt.

Image Generation

Prompt engineering for image generation

Text Generation

Prompt engineering for text generation

Prompt Template

See also: Prompt templates

Prompt template allows the prompt to use of variables. It allows the prompt to stay largely the same while being used with different input values.

Products

• LangChain - library for combining language models with other components to build applications.