Fair points. What really matters IMO is total training compute and FLOPs/token. On that front:
Nenya vs Qwen3.5 2B: Nenya has 71% fewer FLOPs/token and at least 99.8% less training compute (conservative assumption of only 10T tokens for Qwen, but the real number is likely 30T+).
Nenya vs OLMo-2 1B: 57% fewer FLOPs/token (2.3×) and 99.4% less training compute (155×).
Another way to look at it: at iso-compute (2e20 FLOPs), OLMo-1B (which is actually around 1.5B) scores 45.7 vs. Nenya at 58.3.
it's 2.59B total with 553M active, so "0.5B vs 2B" more like 2.5B vs 2B? the card's own table has OLMo-2 1B (61.1) beating both.
Fair points. What really matters IMO is total training compute and FLOPs/token. On that front:
Nenya vs Qwen3.5 2B: Nenya has 71% fewer FLOPs/token and at least 99.8% less training compute (conservative assumption of only 10T tokens for Qwen, but the real number is likely 30T+).
Nenya vs OLMo-2 1B: 57% fewer FLOPs/token (2.3×) and 99.4% less training compute (155×).
Another way to look at it: at iso-compute (2e20 FLOPs), OLMo-1B (which is actually around 1.5B) scores 45.7 vs. Nenya at 58.3.
Technical reports for both of these analyses: https://www.arsalanzafar.ai/nenya-technical-report