Update tokenizer_config.json
#7
by ybelkada - opened
No description provided.
Muennighoff changed pull request status to merged
It's annoying that we're missing indent here ... Whoever is writing this file should probably have indent option set.
Just to be sure, I think tokenizer_config.json comes from transformers, and it's supposed to be fixed for quite a while: https://github.com/huggingface/transformers/blame/862888a35834527fed61beaf42373423ffdbd216/src/transformers/tokenization_utils_base.py#L2155 @ybelkada did you perhaps change this manually instead of going through transformers?