Running 400B model on iPhone!
0.6 t/s
Credit @danveloper @alexintosh @danpacary
@anemll
kept by eddie
A 400B parameter mixture-of-experts AI model was demonstrated running on an iPhone at 0.6 tokens per second, highlighting on-device inference progress.