deepseek-recipe/docs/tokenizer.md

Use with tokenizer

English | 中文

With a tokenizer attached, a PromptEncoding encodes conversations into model token IDs, and a StreamProcessor accepts token IDs. Neither loads a tokenizer on its own: pass one with with_tokenizer, then call the corresponding method. Without an attached tokenizer, encode returns EncodingError::MissingTokenizer, and a Tokens chunk fails the stream with StreamError::MissingTokenizer.

Use the tokenizer that matches your model. This repository bundles copies under static/tokenizers/ for the V4 and V4.1 prompt encodings; load tokenizer.json from the matching directory. You can also load a tokenizer from the Hugging Face Hub. Its added vocabulary must contain the spelling of every special token that rendering writes; otherwise encode returns EncodingError::Encode. IMAGE_SPECIAL_TOKEN_ID is the token ID of the image token in the bundled V4.1 tokenizer; a caller that uses this constant needs a tokenizer that assigns the image token the same token ID.

Rust

Add the dependencies to your application:

cargo add deepseek-recipe@0.2 deepseek-recipe-encoding@0.2 serde_json@1 tokenizers@0.23

Save this as src/main.rs. The tokenizer path is relative to the working directory, so run the application from this repository's root with cargo run --manifest-path path/to/your-app/Cargo.toml; elsewhere, use an absolute path or your own tokenizer.json:

use deepseek_recipe::openai::ChatCompletionRequest;
use deepseek_recipe::request::{ConversionOptions, ProtocolRequest};
use deepseek_recipe_encoding::{PromptEncoding, PromptTokenizer};
use deepseek_recipe_encoding::v4::dsv41::DeepseekV41Encoding;
use serde_json::json;
use tokenizers::Tokenizer;

fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let request: ChatCompletionRequest = serde_json::from_value(json!({
        "model": "DeepSeek-V4.1-Flash",
        "messages": [{"role": "user", "content": "Hello"}],
    }))?;
    let converted = request.convert(ConversionOptions::default())?;

    // Attach the tokenizer, then encode the conversation into token IDs.
    let tokenizer = Tokenizer::from_file("static/tokenizers/v41/tokenizer.json")?;
    let encoding = DeepseekV41Encoding::new().with_tokenizer(PromptTokenizer::new(&tokenizer));
    let token_ids = encoding.encode(&converted.conversation)?;

    println!("{token_ids:?}");
    Ok(())
}

Pass with_tokenizer a PromptTokenizer, which prepares a Tokenizer by keeping a copy without the added vocabulary, or attach any other TokenizerEncoder implementation. encode renders the conversation and then tokenizes the prompt.

render_conversation performs only the rendering step: it returns the prompt text, the byte ranges of its explicit special tokens (special_ranges), and its image sources. Rendering inserts the template's control tokens, the placeholders of a message's image sources, and the grounding tokens in assistant content and reasoning content as explicit special tokens. During tokenization, each special range encodes as the token ID of its spelling, and the text between special ranges is plain text, encoded without the added vocabulary. PromptTokenizer encodes the plain regions as one batch, which the tokenizers crate can encode in parallel, depending on its TOKENIZERS_PARALLELISM setting.

A special-token spelling typed into a user message is plain text, so it cannot act as a control token. A caller that tokenizes RenderedPrompt::prompt itself must apply special_ranges the same way.

After encoding, the caller resolves the max_tokens to send with ConversationRequest::resolve_max_tokens(prompt_tokens, context_limit). prompt_tokens is the number of encoded token IDs plus the image token counts beyond one per placeholder (MultiModalData::image_token_adjustment), and context_limit is the number of tokens the model's context holds. A requested max_tokens is kept. An unspecified one becomes 8192 outside thinking mode, 131072 in thinking mode with max reasoning effort, and 65536 in thinking mode otherwise, lowered to the context left after the prompt. The method returns a ConversionError when the prompt and the requested max_tokens together exceed context_limit.

Decoding

A backend can produce token IDs instead of text. Attach the same tokenizer to the stream processor to decode those chunks. The example also needs tokio-stream, plus tokio with the macros and rt-multi-thread features, as listed in response streaming. The tokenizer path has the same requirement as in the encoding example:

use deepseek_recipe::openai::ChatCompletionRequest;
use deepseek_recipe::request::{ConversionOptions, ProtocolRequest};
use deepseek_recipe::stream::{InferenceChunk, InferenceFinishReason, StreamProcessor};
use serde_json::json;
use tokenizers::Tokenizer;
use tokio_stream::{StreamExt, iter};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let request: ChatCompletionRequest = serde_json::from_value(json!({
        "model": "DeepSeek-V4.1-Flash",
        "messages": [{"role": "user", "content": "Hello"}],
        "thinking": {"type": "disabled"},
    }))?;
    let converted = request.convert(ConversionOptions::default())?;
    let generator = ChatCompletionRequest::chunk_generator(
        &converted,
        "id-1".into(),
        "DeepSeek-V4.1-Flash".into(),
    );

    // The backend answers with token IDs; the attached tokenizer decodes them.
    let tokenizer = Tokenizer::from_file("static/tokenizers/v41/tokenizer.json")?;
    let token_ids = tokenizer.encode("Hello there!", false)?.get_ids().to_vec();
    let processor =
        StreamProcessor::new(generator, converted.parsing_options).with_tokenizer(tokenizer);

    let inference = iter([
        InferenceChunk::Tokens { token_ids, logprobs: None },
        InferenceChunk::Finish { finish_reason: InferenceFinishReason::Stop },
    ]);

    let mut chunks = std::pin::pin!(processor.into_protocol_stream(inference));
    while let Some(chunk) = chunks.next().await {
        println!("{}", serde_json::to_string(&chunk?)?);
    }
    Ok(())
}

The processor decodes token IDs incrementally and buffers them while a multi-token character is incomplete; token IDs that produced text count as completion tokens. Each special token is decoded as its own piece and marked, so the output parser recognizes </think> and the DSML tags only as tokens; the same text spelled with plain tokens is content. Text chunks carry no token boundaries, so their markers match by spelling. A Tokens chunk without an attached tokenizer fails the stream with StreamError::MissingTokenizer. Tokenizer implements the StreamTokenizer trait used by with_tokenizer.

InferenceChunk::Tokens carries the token IDs of one backend output, which the processor decodes together. When the converted request sets InferenceOptions::logprobs, the caller supplies one InferenceTokenLogprob per token ID: the decoded text then carries the log probabilities of its tokens, and Chat Completions and Responses report them. A Tokens chunk whose log probability count differs from its token ID count fails the stream with StreamError::LogprobCountMismatch. In thinking mode, the processor also uses the attached tokenizer to count reasoning_tokens for usage by encoding the emitted reasoning content, for text chunks as well as token chunks. Without a tokenizer, Chat Completions usage omits completion_tokens_details, and Responses usage reports reasoning_tokens as 0.

Python

Install the package with python3 -m pip install deepseek-recipe. The tokenizer path has the same requirement as in the Rust example:

from deepseek_recipe import (
    ChatCompletionRequest,
    ConversionOptions,
    DeepseekV41Encoding,
    Tokenizer,
)

request = ChatCompletionRequest({
    "model": "DeepSeek-V4.1-Flash",
    "messages": [{"role": "user", "content": "Hello"}],
})
converted = request.convert(ConversionOptions())

tokenizer = Tokenizer.from_file("static/tokenizers/v41/tokenizer.json")
encoding = DeepseekV41Encoding().with_tokenizer(tokenizer)
print(encoding.encode(converted.conversation))

Tokenizer.from_pretrained downloads and caches a tokenizer.json from the Hugging Face Hub, and Tokenizer.from_str loads the file contents directly. render_conversation and encode return the same results as in Rust; encode raises RuntimeError when no tokenizer is attached or encoding fails. After encoding, ConversationRequest.resolve_max_tokens resolves max_tokens as in Rust and raises ConversionError where Rust returns one.

Decoding

Pass the tokenizer as the processor's third argument. A tokens chunk is decoded with the same behavior as in Rust; without a tokenizer, such a chunk raises RuntimeError and closes the processor.

from deepseek_recipe import (
    ChatCompletionRequest,
    ConversionOptions,
    InferenceChunk,
    InferenceFinishReason,
    StreamProcessor,
    Tokenizer,
)

converted = ChatCompletionRequest({
    "model": "DeepSeek-V4.1-Flash",
    "messages": [{"role": "user", "content": "Hello"}],
    "thinking": {"type": "disabled"},
}).convert(ConversionOptions())
generator = ChatCompletionRequest.chunk_generator(converted, "id-1", "DeepSeek-V4.1-Flash")

# The backend answers with token IDs; the tokenizer passed to the processor
# decodes them.
tokenizer = Tokenizer.from_file("static/tokenizers/v41/tokenizer.json")
token_ids = tokenizer.encode("Hello there!")
processor = StreamProcessor(generator, converted.parsing_options, tokenizer)

chunks = []
for chunk in (
    InferenceChunk.tokens(token_ids),
    InferenceChunk.finish(InferenceFinishReason.Stop),
):
    chunks.extend(processor.push(chunk))
chunks.extend(processor.finish())

for chunk in chunks:
    print(chunk.to_json())
processor.close()

See the project README for supported scope and the response streaming guide for the full chunk sequence.