Exp-4

Small language model (3.0M parameters) trained from scratch.

Architecture

Property	Value
Layers	5
Hidden size	192
Intermediate size	512
Attention heads	6 (GQA kv=6)
Max sequence length	1024
Vocab size	4096
Tied embeddings	True
Total parameters	3.000M

Training

• Tokens seen: 15,605,382,144
• Val loss: 2.3172
• Val PPL: 10.15

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Exp-4")
model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Exp-4")
inputs = tokenizer("Hello", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))