gm.testing.DummyTokenizer#

class gemma.gm.testing.DummyTokenizer(

*,

path: str | os.PathLike = '/tmp/dummy_tokenizer.model',

custom_tokens: dict[int,

str] = <factory>,

)[source]

Bases: gemma.gm.text._tokenizer.Tokenizer

Dummy tokenizer.

path: epath.PathLike = '/tmp/dummy_tokenizer.model'

special_tokens[source]

alias of gemma.gm.text._tokenizer._Gemma3SpecialTokens