Skip to content

Respect tokenizer.ggml.add_bos_token value when tokenizing#4040

Merged
KerfuffleV2 merged 4 commits into
ggml-org:masterfrom
KerfuffleV2:fix-respect-use-bos-token
Nov 17, 2023
Merged

Respect tokenizer.ggml.add_bos_token value when tokenizing#4040
KerfuffleV2 merged 4 commits into
ggml-org:masterfrom
KerfuffleV2:fix-respect-use-bos-token

gguf-py: Try to fix SpecialVocab giving up too easily for the Nth time

a169862
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs