SuperBPE tokenizer

SuperBPE tokenizer trained on mix of code + edu data

"Big cat sat on a mat" -> ['Big', 'Ġcat', 'Ġsat', 'ĠonĠa', 'Ġmat']

"this function does summation on a last axis and then returns a scalar." -> ['this', 'Ġfunction', 'Ġdoes', 'Ġsummation', 'ĠonĠa', 'Ġlast', 'Ġaxis', 'ĠandĠthen', 'ĠreturnsĠa', 'Ġscalar', '.']

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support