ExplainLLM LogoExplainLLM
LLM FundamentalsAI AgentsContext EngineeringPrompting TechniquesPlaygroundMy PathPricing
My Path
Rankings
LLM Evolution
Neural NetworksTraining DynamicsTransfer LearningTokenizationEmbeddingsAttention MechanismTransformer ArchitectureInference & KV-CacheDecoding StrategiesLLM SettingsPrompt ElementsPrompt Engineering BasicsBest PracticesQuantizationFine-tuning vs PromptingReasoning ModelsOpen-Source ModelsRLHFMixture of Experts (MoE)Knowledge DistillationRLVR and GRPODiffusion LLMsTest-Time Compute Scaling
ReAct PatternFunction CallingTool ImpactAgent MemoryMulti-Agent SystemsPlanning & DecompositionAgent ArchitecturesBuilding AI AgentsLangChain & LlamaIndexAgent Evaluation & TestingA2A Protocol (Agent-to-Agent)Computer Use & Browser AgentsAgent Protocols: A2A, ACP & AP2
Context EngineeringSystem PromptsContext WindowsPrompt StructureRAG — Retrieval Augmented GenerationPrompt SecurityContext Engineering Strategies: Write, Select, Compress, Isolate
Zero-Shot PromptingChain-of-Thought — Step-by-Step AI ReasoningFew-Shot LearningSelf-Consistency — Improving Accuracy via VotingTree of ThoughtsMeta-PromptingReflexion — Self-Correcting LLM OutputsLeast-to-Most — Bottom-Up Problem DecompositionProgram of ThoughtChain of VerificationRAG — Retrieval-Augmented GenerationPrompt ChainingGenerate KnowledgeStructured OutputAPE — Automatic Prompt EngineeringDSPy — Programming Language ModelsART — Automatic Reasoning and Tool-useMultimodal CoTPrompt FrameworksPractical PatternsCombining TechniquesSelf-Refine
Code GenerationText ClassificationSummarizationInformation ExtractionQuestion AnsweringData GenerationChatbots & Conversational AIText TransformationSemantic SearchContent GenerationText-to-SQL — Natural Language to SQL QueriesStructured OutputSentiment & Opinion MiningImage Generation PromptingPer-Model Prompting GuidesAgentic Coding
Model Selection GuideLLM BenchmarksVector DatabasesLLM ObservabilityCost OptimizationAPI Integration PatternsLLM DeploymentProduction GuardrailsRAG vs Fine-tuningSmall Language ModelsPrompt CachingLLMOpsModel RoutingHarness Engineering
Vision LLMsImage AnalysisPrompt Engineering for VisionDocument UnderstandingVision HallucinationsMultimodal RAGVoice AgentsReal-Time MultimodalVideo & AudioMultimodal CostsDiffusion Models
Prompt InjectionJailbreaking — LLM Safety Bypass TechniquesFactuality & HallucinationsBiases in LLMsAI Safety & AlignmentData Privacy & PII LeakageRed Teaming for LLMsContext Laundering
Introduction to Claude CodeInstallation & SetupYour First SessionThe Agent Loop18 Built-in ToolsFile OperationsSearch & NavigationSub-agents OverviewBuilt-in AgentsCreating Custom AgentsAgent PatternsMCP FundamentalsPopular MCP ServersMCP ConfigurationAdvanced MCPMemory & CLAUDE.mdModular RulesContext ManagementHooks SystemSkills & CommandsCreating PluginsAgent SDK
Prompting Techniques ComparisonAI Prompt Engineering ChallengesLLM Playground — Test AI PromptsBPE Tokenizer — See How AI Reads TextSemantic Search Demo — AI EmbeddingsToken Probabilities — AI Model PredictionsContext Window VisualizerPrompt Battle — Technique Comparison
Prompt Constructor
Glossary
What's New
AI Recipes
AI Digest
  1. Home
  2. Playground
  3. Tokenizer

Resources

GlossaryLearning PathPlaygroundWhat's New

Legal

Terms of ServicePrivacy PolicyContact UsAbout the Author

© 2024-2026 ExplainLLM. All rights reserved.

Telegram channel

BPE Tokenizer — See How AI Reads Text

Interactive BPE tokenizer visualization: see how GPT-4, Claude, and Llama split text into tokens in real time

Tokenization Mode

Local BPE simulation — demonstrates how the algorithm works

Tokenizer Explorer
65Tokens
76Characters
1.17Chars/Token
BPE Merge Process
Show BPE merges
Merge step 0/11
Tokens (76)
T
h
e
␣
q
u
i
c
k
␣
b
r
o
w
n
␣
f
o
x
␣
j
u
m
p
s
␣
o
v
e
r
␣
t
h
e
␣
l
a
z
y
␣
d
o
g
.
␣
L
L
M
s
␣
p
r
o
c
e
s
s
␣
t
e
x
t
␣
u
s
i
n
g
␣
t
o
k
e
n
s
.