Grokking: A 777-Parameter Transformer Learned Real Math

(themenonlab.blog)

4 points | by peter_d_sherman 11 hours ago ago

3 comments

  • andsoitis 9 hours ago ago

    > ...addition with 99.69% accuracy.

    > Neural networks can learn algorithms, not just patterns. The 777-parameter model didn’t memorize. It couldn’t. It discovered a general procedure for addition that works on numbers it had never seen.

    If that's true, I would have expected accuracy to be 100%. Why isn't it 100% accurate?

    • Arnt 2 hours ago ago

      Are neural networks ever 100% accurate?

  • paaloeye 10 hours ago ago

    I'm glad it's not 737-Max-Parameter