Apple Shocks Again： Introducing OpenELM Open Source AI Model That Changes Everything!

https://www.youtube.com/watch?v=huH0fKmw0H0

Rmsnorm 사용, swiGLU 사용, Grouped query attention
LR 공개
optimizer 공개
weight decay 공개
rmsnorm 사용 - 정확하지만 느림

https://arxiv.org/abs/2404.14619

OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

https://machinelearning.apple.com/research/openelm

OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

https://github.com/apple/corenet

GitHub - apple/corenet: CoreNet: A library for training deep neural networks

공개해서 좋은데, 애플 홍보 같은 느낌

그래도 대규모 모델에 이렇게 다 공개하는 건 좋은 일