Tokens

Tokens

Tokens are fragments of words, which may include trailing spaces or sub-words. They are used by natural language processing (NLP) systems, such as the OpenAI API, to process text input. The way words are broken down into tokens is language-dependent, which can affect the implementation cost of the API for languages other than English.

Understanding Token Lengths

To grasp the concept of tokens, consider the following generalizations about token lengths:

1 token ≈ 4 characters in English 1 token ≈ ¾ words 100 tokens ≈ 75 words In terms of sentences and paragraphs:

1-2 sentences ≈ 30 tokens 1 paragraph ≈ 100 tokens 1,500 words ≈ 2048 tokens For additional context, consider the following examples:

Wayne Gretzky’s quote "You miss 100% of the shots you don't take" contains 11 tokens. OpenAI’s charter contains 476 tokens. The transcript of the US Declaration of Independence contains 1,695 tokens.

Tokenization Tools

To further explore tokenization, the following tools and libraries can be used:

OpenAI's interactive Tokenizer tool Tiktoken, a fast BPE tokenizer specifically for OpenAI models Transformers package for Python gpt-3-encoder package for node.js

Token Limits

The token limit for requests depends on the model used, with a maximum of 4097 tokens shared between prompt and completion. If a prompt has 4000 tokens, the completion can only have up to 97 tokens. This limit is a technical constraint, but there are ways to work within it, such as condensing prompts or breaking text into smaller parts.

Token Pricing

API token pricing varies depending on the model type, with different capabilities and speeds offered at different price points. Davinci is the most capable model, while Ada is the fastest. Detailed token pricing information can be found on the OpenAI API's pricing page.

Token Context

GPT-3 processes tokens based on their context in the corpus data. Identical words may have different tokens depending on their structure within the text. For instance, the token value generated for the word "red" varies based on its context.

Lowercase in the middle of a sentence: " red" (token: "2266") Uppercase in the middle of a sentence: " Red" (token: "2297") Uppercase at the beginning of a sentence: "Red" (token: "7738") The more probable or frequent a token is, the lower the token number assigned to it. For example, the token for the period ("13") remains the same in all three sentences because its usage is similar across the corpus data.

Prompt Design and Token Knowledge

Understanding tokens can help improve prompt design in several ways.

Prompts Ending with a Space

Since tokens can include trailing space characters, prompts ending with a space may produce lower-quality output. The API's token dictionary already accounts for trailing spaces.

Using the logit_bias Parameter

The logit_bias parameter allows setting biases for specific tokens, modifying the likelihood of those tokens appearing in the completion. For example, if creating an AI Baking Assistant that considers users' egg allergies, the logit_bias parameter can be used to discourage the model from generating responses that include any variation of the word "egg".

First, use a tokenizer tool to identify the tokens for which biases should be set:

Singular with trailing space: " egg" (token: "5935") Plural with trailing space: " eggs" (token: "9653")

Subword token generated for "Egg" or "Eggs": "gg" (token: "1130") The logit_bias parameter accepts bias values ranging from -100 to +100. Extreme values result in either a ban (-100) or exclusive selection (100) of the relevant token.

By adding the logit biases to the prompt, the likelihood of the word "egg" (and its variations) appearing in the response for a banana bread recipe is reduced. As a result, the AI Baking Assistant generates a response that does not include eggs, fulfilling its requirement of considering the user's egg allergies.