370
edits
Tokens: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 45: | Line 45: | ||
*Uppercase at the beginning of a sentence: "Red" (token: "7738") | *Uppercase at the beginning of a sentence: "Red" (token: "7738") | ||
*The more likely or common a token is, the lower the token number assigned to it. For example, the token for the period ("13") remains consistent in all three sentences because its usage is similar throughout the corpus data. | *The more likely or common a token is, the lower the token number assigned to it. For example, the token for the period ("13") remains consistent in all three sentences because its usage is similar throughout the corpus data. | ||
<gallery mode=packed> | |||
File:tokens3.png | |||
File:tokens2.png | |||
File:tokens1.png | |||
</gallery> | |||
==Prompt Design and Token Knowledge== | ==Prompt Design and Token Knowledge== | ||