InterwebAlchemy
/

PGNTokenizer

Model card Files Files and versions

PGNTokenizer / tokenizer.json

Commit History

Revert back to NFD tokenizer

3341d84
verified

InterwebAlchemy commited on Feb 11, 2025

Remove normalizer

767e594
verified

InterwebAlchemy commited on Feb 11, 2025

Switch from NFD -> NFC

0dcea10
verified

InterwebAlchemy commited on Feb 11, 2025

Upload tokenizer

57dcf89
verified

InterwebAlchemy commited on Jan 28, 2025