Unix: add a history of llamas
This commit is contained in:
parent
e509097d20
commit
9128bf5799
GPG key ID:
8DE0764E1809E9FC
167
src/content/unix/a-history-of-llamas.md
Normal file
167
src/content/unix/a-history-of-llamas.md
Normal file
|
|
@ -0,0 +1,167 @@
|
||||||
|
---
|
||||||
|
title: 'A Brief History of Local Llamas'
|
||||||
|
description: 'A Brief History of Local Llamas'
|
||||||
|
updateDate: 'Dec 31 2023'
|
||||||
|
heroImage: '/images/tiny-llama-logo.avif'
|
||||||
|
---
|
||||||
|
|
||||||
|
<p style="font-size: max(2vh, 10px); margin-top: 0; text-align: right">
|
||||||
|
"TinyLlama logo" by <a href="https://github.com/jzhang38/TinyLlama">The
|
||||||
|
TinyLlama project</a>. Licensed under Apache 2.0
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## My Background
|
||||||
|
|
||||||
|
I've been taking machine learning courses throughout the "modern history" of
|
||||||
|
llamas. When ChatGPT was first released, we bought in a guest lecturer on NLP
|
||||||
|
methods of the time. Since then, I've also taken an NLP course, though not one
|
||||||
|
focused on deep learning.
|
||||||
|
|
||||||
|
Most my knowledge of this field comes from a few guest lectures, and the
|
||||||
|
indispensable [r/locallama](https://www.reddit.com/r/LocalLLaMA/) community,
|
||||||
|
which always has the latest news about local llamas. I've become a fan of the
|
||||||
|
local llama movement in December 2023, so the "important points" covered here
|
||||||
|
are coming from a retrospective.
|
||||||
|
|
||||||
|
I use the terms "large language model" and "llama" interchangeably, throughout
|
||||||
|
this piece. I write "open source and locally hosted llama" as "local llama".
|
||||||
|
Whenever you see numbers 7B, that means the llama has 7 billion parameters. More
|
||||||
|
parameters means the model is smarter but bigger.
|
||||||
|
|
||||||
|
## Modern History
|
||||||
|
|
||||||
|
The modern history of llamas begins in 2022. 90% of sources llama papers cite
|
||||||
|
these days are from 2022 onwards.
|
||||||
|
|
||||||
|
Here's a brief timeline:
|
||||||
|
|
||||||
|
1. **March 2022**: InstructGPT paper is pre-print.
|
||||||
|
2. **November 2022**: ChatGPT is released.
|
||||||
|
3. **March 2023**: LLaMa (open source) and GPT4 is released.
|
||||||
|
4. **July 2023**: LLaMa 2 is released, alongside GGUF quantization.
|
||||||
|
5. **August 2023**: AWQ quantization paper.
|
||||||
|
6. **September 2023**: Mistral 7B is released.
|
||||||
|
7. **December 2023**: Mixtral 8x7B becomes the first MoE local llama.
|
||||||
|
|
||||||
|
#### Early 2022
|
||||||
|
|
||||||
|
In March 2022, OpenAI released a paper improving the conversational ability of
|
||||||
|
their then uncontested GPT3. Interest in
|
||||||
|
[InstructGPT](https://arxiv.org/pdf/2203.02155.pdf) was largely limited to the
|
||||||
|
academic community. As of writing, InstructGPT remains the last major paper
|
||||||
|
OpenAI released on this topic.
|
||||||
|
|
||||||
|
The remainder of 2022 was mostly focused on text-to-image models. OpenAI's
|
||||||
|
[DALL-E 2](https://openai.com/dall-e-2) lead the way, but the open source
|
||||||
|
community kept pace with [Stable
|
||||||
|
Diffusion](https://github.com/Stability-AI/generative-models).
|
||||||
|
[Midjourney](https://www.midjourney.com/) eventually ends up on top by mid-2022,
|
||||||
|
and causes a huge amount of controversy with artists by [winning the Colorado
|
||||||
|
State
|
||||||
|
Fair's](https://en.wikipedia.org/wiki/Th%C3%A9%C3%A2tre_D%27op%C3%A9ra_Spatial)
|
||||||
|
fine-art contest.
|
||||||
|
|
||||||
|
#### Late 2022
|
||||||
|
|
||||||
|
In late November 2022, OpenAI [released
|
||||||
|
ChatGPT](https://openai.com/blog/chatgpt), which is generally speculated to be a
|
||||||
|
larger version of InstructGPT. This model is single-handedly responsible for the
|
||||||
|
NLP Boom. ChatGPT revolutionized the corporate perception on chatbots and AI in
|
||||||
|
general. It was considered a form of disruptive innovation in the search engine
|
||||||
|
market, leading Google to hold record layoffs in [January
|
||||||
|
2023](https://blog.google/inside-google/message-ceo/january-update/).
|
||||||
|
|
||||||
|
At this point, the local llama movement practically didn't exist. Front-ends,
|
||||||
|
especially for chatbot role-play as later exemplified by
|
||||||
|
[SillyTavern](https://github.com/SillyTavern/SillyTavern), began development,
|
||||||
|
but they were all still running through the OpenAI's ChatGPT API.
|
||||||
|
|
||||||
|
#### Early 2023
|
||||||
|
|
||||||
|
In March 2023, Meta kick-started the local llama movement, by releasing
|
||||||
|
[LlaMa](https://ai.meta.com/blog/large-language-model-llama-meta-ai/) a 65B
|
||||||
|
parameter llama that was open source! Benchmarks aside, it was not very good.
|
||||||
|
ChatGPT continued to be viewed as considerably better. However, it provided a
|
||||||
|
strong base for further iteration, and gave the local llama community a much
|
||||||
|
stronger model than any other local llama at the time.
|
||||||
|
|
||||||
|
Around this time, OpenAI released [GPT4](https://openai.com/research/gpt-4), a
|
||||||
|
model that undeniably broke through all records. In fact, GPT4 remains the best
|
||||||
|
llama, as of December 2023. The original ChatGPT is now referred to as GPT3.5,
|
||||||
|
to disambiguate it from GPT4. This caused much of the local llama community to
|
||||||
|
continue focusing on building frontends, while using GPT4's API for inference.
|
||||||
|
Nothing open source was even remotely close to GPT4 at this point.
|
||||||
|
|
||||||
|
#### Mid 2023
|
||||||
|
|
||||||
|
We finally see the local llama movement really take off around August 2023. Meta
|
||||||
|
released [LLaMa2](https://ai.meta.com/blog/llama-2/), which has decent
|
||||||
|
performance even on its 7B version. One key contribution of LLaMa2 was the GGUF
|
||||||
|
quantization format. This format allows a model to be run on a mix of RAM and
|
||||||
|
VRAM, which meant many home computers could now run 4-bit quantized 7B models!
|
||||||
|
Previously, most enthusiasts would have to rent cloud GPUs to run their "local"
|
||||||
|
llamas. Quantizing into GGUF is a very expensive process, so
|
||||||
|
[TheBloke](https://huggingface.co/TheBloke) on Huggingface emerges the defacto
|
||||||
|
source for pre-quantized llamas.
|
||||||
|
|
||||||
|
Based on LLaMa, the open source
|
||||||
|
[llama.cpp](https://github.com/ggerganov/llama.cpp) becomes the leader of local
|
||||||
|
llama inference backends. Its support extends far beyond only running LLaMa2,
|
||||||
|
it's the first major backend to support running GGUF quantizations!
|
||||||
|
|
||||||
|
In addition, the [Activation-aware Weight
|
||||||
|
Quantization](https://arxiv.org/abs/2306.00978) (AWQ) paper is released at this
|
||||||
|
time. It uses mixed-quantization to increase both the speed and performance of
|
||||||
|
quantized models. This is especially true for very heavily quantized models like
|
||||||
|
4-bit quantization, that has become the standard for the local llama
|
||||||
|
community at this point. AWQ lacks support anywhere at this time.
|
||||||
|
|
||||||
|
In late September 2023, out of nowhere came a French startup with a 7B model
|
||||||
|
that made leaps on top of LLaMa2 7B.
|
||||||
|
[Mistral](https://mistral.ai/news/announcing-mistral-7b/) remains the best local
|
||||||
|
llama until mid-December 2023. Huge work in improving model tuning, particularly
|
||||||
|
character creation and code-assistant models, is done on top of Mistral 7B.
|
||||||
|
|
||||||
|
#### Late 2023
|
||||||
|
|
||||||
|
It's worth noting that at this point, there is *still* no local competitor to
|
||||||
|
GPT3.5. But that was all about to change on December 11th 2023, when Mistral
|
||||||
|
released [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/). Mixtral is
|
||||||
|
the first major local llama to use the same technique as GPT4; a mixture of
|
||||||
|
experts (MoE). While about 1/40th the speculated size of GPT4 and 1/3rd of
|
||||||
|
GPT3.5, Mixtral is able to go toe-to-toe with GPT3.5 both in benchmarks and user
|
||||||
|
reviews. This is hailed as a landmark achievement by the local llama community,
|
||||||
|
demonstrating that open source models are able to compete with commercially
|
||||||
|
developed ones. The achievement is amplified by comparing Mixtral against Google
|
||||||
|
freshly-unveiled [Gemini](https://blog.google/technology/ai/google-gemini-ai/)
|
||||||
|
models in the same week. R/localllama reviews generally suggest Mixtral pulls
|
||||||
|
ahead of Gemini in conversational tasks.
|
||||||
|
|
||||||
|
Apple unexpectedly joins the local llama movement, by open-sourcing their
|
||||||
|
[Ferret model](https://github.com/apple/ml-ferret) in mid-December. This model
|
||||||
|
builds upon LLaVA, previously the leading multi-modal llama for images in the
|
||||||
|
local llama community.
|
||||||
|
|
||||||
|
In very late December, llama.cpp [merges
|
||||||
|
AWQ](https://github.com/ggerganov/llama.cpp/pull/4593) support. In the coming
|
||||||
|
year, I expect AWQ to largely replace GPTQ in the local llama community, though
|
||||||
|
GGUF will remain more popular in general.
|
||||||
|
|
||||||
|
Just 3 days from the end of the year,
|
||||||
|
[TinyLlama](https://github.com/jzhang38/TinyLlama) releases their 3 trillion
|
||||||
|
token checkpoint, on their 1.1B model. This miniscule model sets a new
|
||||||
|
lower-bound for the number of neurons required to make a capable llama, enabling
|
||||||
|
more users to locally host their llama. In practice, TinyLlama easily goes
|
||||||
|
toe-to-toe with Microsoft's closed-source [Phi-2
|
||||||
|
2.7B](https://huggingface.co/microsoft/phi-2) released just a few weeks earlier.
|
||||||
|
This is a huge win for the open source community, demonstrating how open source
|
||||||
|
models can get ahead of commercial ones.
|
||||||
|
|
||||||
|
#### Going Forward
|
||||||
|
|
||||||
|
With the release of Mixtral, the local llama community is hoping 2024 will be
|
||||||
|
the turning point where open source models finally break ahead of commercial
|
||||||
|
models. However as of writing, it's very unclear how the community will break
|
||||||
|
through GPT4, the llama that remains uncontested in practice.
|
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in a new issue