Choose what the answer must preserve.
Our fictional Harbor project needs a three-bullet handoff checklist. Its owner is Morgan, the due date is October 14, 2026, and export stays blocked until human review. The instruction explicitly forbids inventing approval.
The full practice file repeats background about meetings and dashboard colors twelve times. The brief keeps the task, three facts, and constraint. Both were written for this exercise; neither is a model output or a real workplace record.
Before trimming your own context, write down the facts, exceptions, source references, and constraints the answer must retain. A shorter input can still be a worse input if it removes evidence needed for the job.
Words, characters, and tokens are different units.
tiktoken converts text into token IDs using a selected encoding. Its versioned README introduces the tokenizer, and the encoding API defines the text operations used here.
We explicitly use o200k_base for a reproducible practice count. This does not establish the right tokenizer for every model or provider. Check the target’s documented tokenizer. tiktoken’s model mapping is versioned code, and unrecognized names can raise KeyError; do not silently treat an arbitrary fallback as an exact match.
Start in an empty practice folder.
Install Python, open PowerShell in a new empty folder, and confirm the version. tiktoken 0.14.0 requires Python 3.9 or newer. This exercise ran on Windows with Python 3.13.15.
python --version
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install 'tiktoken==0.14.0'The virtual environment keeps packages in this folder. It does not sandbox code or reduce operating-system permissions. Calling its interpreter directly avoids changing PowerShell activation policy. The top-level package is pinned; transitive dependencies may change.
Installation downloads packages. The first encoding load may download tokenizer data; tiktoken’s loader implements caching, and the encoding definition names the data. Our script contains no model request and sends no text to an inference API. That is a code-scope statement, not a broad audit of your computer or network.
Download the files, then read the counter.
Download the complete counting exercise contains the complete Python counter, two fictional text files, observed counts, and a README. Download it using the command below, extract it in your practice folder, and review the code before running it.
Invoke-WebRequest -Uri 'https://jabsystems.io/ai-anonymous/Social/prompt-budget-exercise.zip' -OutFile .\prompt-budget-exercise.zip
Expand-Archive -LiteralPath .\prompt-budget-exercise.zip -DestinationPath .\exercise
Get-Content -LiteralPath .\exercise\count-context.pyThe counter reads UTF-8 text (a BOM is accepted), normalizes Windows CRLF line endings to LF, and uses encode_ordinary. Token-looking strings are treated as ordinary text. It compares each file separately with budget - reserve; it does not add the alternatives together or cut text automatically.
"""Count UTF-8 plain text locally. No model calls, pricing, or truncation."""
import argparse
import json
from importlib.metadata import version
from pathlib import Path
import tiktoken
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("files", nargs="+", type=Path)
parser.add_argument("--encoding", choices=["o200k_base", "cl100k_base"], default="o200k_base")
parser.add_argument("--budget", type=int, default=300)
parser.add_argument("--reserve", type=int, default=100)
args = parser.parse_args()
if args.budget <= 0 or args.reserve < 0 or args.reserve >= args.budget:
parser.error("budget must be positive; reserve must be between 0 and budget - 1")
enc = tiktoken.get_encoding(args.encoding)
allowance = args.budget - args.reserve
results = []
for path in args.files:
try:
# utf-8-sig also accepts a UTF-8 BOM. Python normalizes CRLF to LF.
text = path.read_text(encoding="utf-8-sig")
except (OSError, UnicodeError) as error:
parser.error(f"Cannot read {path}: {error}")
# Token-looking strings are counted as ordinary text, never special tokens.
tokens = len(enc.encode_ordinary(text))
results.append({"file": path.name, "text_tokens": tokens,
"allowance": allowance, "within_text_budget": tokens <= allowance})
print(json.dumps({"tiktoken": version("tiktoken"), "encoding": args.encoding,
"budget": args.budget, "reserve": args.reserve,
"scope": "normalized plain text only; not a complete API request",
"results": results}, indent=2))
return 0 if all(item["within_text_budget"] for item in results) else 2
if __name__ == "__main__":
raise SystemExit(main())Make the budget check visible.
Here we choose an illustrative total of 300 tokens and reserve 100, leaving a 200-token allowance for the text file. These are practice values, not a model’s context limit. The reserve is a planning placeholder; it does not measure output or account for every request field.
.\.venv\Scripts\python.exe .\exercise\count-context.py .\exercise\context-full.txt .\exercise\context-brief.txt --budget 300 --reserve 100
$LASTEXITCODEThe combined command intentionally exits 2 because at least one file exceeds the allowance. It still prints the report. Run the brief alone to see a passing check:
.\.venv\Scripts\python.exe .\exercise\count-context.py .\exercise\context-brief.txt --budget 300 --reserve 100
$LASTEXITCODEThat command exits 0. Invalid arguments or unreadable files also produce an error exit; read the accompanying error rather than interpreting every 2 as an over-budget result.
What we actually observed
With the exact downloads, tiktoken 0.14.0, and o200k_base, the full file counted 396 text tokens and the brief counted 44. You can inspect the observed JSON report. Editing the text, line endings after normalization, or encoding may change the count.
{
"tiktoken": "0.14.0",
"encoding": "o200k_base",
"budget": 300,
"reserve": 100,
"scope": "normalized plain text only; not a complete API request",
"results": [
{
"file": "context-full.txt",
"text_tokens": 396,
"allowance": 200,
"within_text_budget": false
},
{
"file": "context-brief.txt",
"text_tokens": 44,
"allowance": 200,
"within_text_budget": true
}
]
}This shows that the brief fits our chosen text allowance. It does not demonstrate better answers, lower billing, a faster model, or accuracy on other data. No inference was performed.
Spend context on evidence that earns its place.
- Keep decision-critical material. Preserve source references, exceptions, deadlines, owners, and the exact output contract.
- Remove repetition deliberately. Replace repeated background with a reviewed statement. Recount the result and compare its retained facts with your checklist.
- Use the right encoding. This counter also accepts
--encoding cl100k_basefor comparison; a different number alone does not show which one your target uses. - Check the actual request. Include the surrounding instructions, conversation, tools, schemas, and attachments when validating the production payload.
- Test the answer separately. If you later submit the two alternatives to a model, grade whether each preserves the facts and human-review condition. That experiment is proposed here and has not been run.
Use synthetic or approved non-sensitive text for practice. This exercise establishes a small local counting workflow, not permission to process confidential records in another tool.
A plain-text count is one piece of the request.
OpenAI’s current token-counting guide distinguishes local text counts from the request structure, images, files, tools, and schemas. It documents a Responses API input-count endpoint for the actual payload. That endpoint sends a request to OpenAI and was not tested in this exercise.
Our arbitrary reserve does not certify that a real request fits a context window. It also does not establish a price: billing categories and output usage are outside this script. For another provider, use that provider’s documented tokenizer and request-count method.
Verified: installation and execution on the stated Windows/Python environment; the two downloaded text counts; expected passing and failing budget checks; input validation and ordinary-text treatment of Unicode and token-looking strings. Not tested: model inference, complete API totals, the remote counting endpoint, answer quality, savings, latency, or other operating systems.
Keep the exercise and its evidence together.
Primary sources checked October 10, 2026: the linked tiktoken 0.14.0 README, encoding API, model mapping, loader, encoding definition, PyPI package metadata, and official OpenAI token-counting guide. The tutorial is a practical exercise, not a claim that this package was released today.
Download the complete counting exercise · Caption · Alt text · Posting ZIP
Manual social delivery: the original graphic, teaser caption, alt text, and full article URL are ready for the user to post. Nothing is scheduled or published to social media; no Instagram profile-link change is claimed.