Kudos on Atom 2.7M
I just wanted to say I'm a big fan of what you've developed with Atom 2.7M. I always saw Arithmark-2 as one of those benchmarks that could only be beaten with scale and curated synthetic data, and you went and showed me how wrong I was with an inspired architecture. I'm really looking forward to seeing what else you come up with.
Thank you, that means a lot. I’m really excited about where this is headed and can’t wait to share more future projects soon.
Thank you, that means a lot. I’m really excited about where this is headed and can’t wait to share more future projects soon.
please do atom 500k
I did try an Atom model at around 500k parameters, but it struggled to handle language modeling and arithmetic simultaneously. An arithmetic-only Transformer at roughly the same parameter scale was successful, though, and I’ll link the article below:
https://huggingface.co/blog/ucr-max/atom2-7m-arithmetic-representation
My focus has now shifted toward more commonsense-oriented pretraining objectives, especially alternative tokenization approaches and datasets with higher educational density.
I did try an Atom model at around 500k parameters, but it struggled to handle language modeling and arithmetic simultaneously. An arithmetic-only Transformer at roughly the same parameter scale was successful, though, and I’ll link the article below:
https://huggingface.co/blog/ucr-max/atom2-7m-arithmetic-representationMy focus has now shifted toward more commonsense-oriented pretraining objectives, especially alternative tokenization approaches and datasets with higher educational density.
o cool, have you tried training a model on bad and good model weights to generate models from scratch instantly no training😮