REFERENCES:
1. Build an LLM from Scratch book: https://amzn.to/4fqvn0D
2. Build an LLM from Scratch repo: https://github.com/rasbt/LLMs-from-scratch
3. Slides: https://sebastianraschka.com/pdf/slides/2024-build-llms.pdf
4. LitGPT: https://github.com/Lightning-AI/litgpt
5. TinyLlama pretraining: https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b
DESCRIPTION:
This video provides an overview of the three stages of developing an LLM: Building, Training, and Finetuning. The focus is on explaining how LLMs work by describing how each step works.
—
To support this channel, please consider purchasing a copy of my books: https://sebastianraschka.com/books/
—
https://linkedin.com/in/sebastianraschka/
https://magazine.sebastianraschka.com
—
OUTLINE:
00:00 โ Using LLMs
02:50 โ The stages of developing an LLM
05:26 โ The dataset
10:15 โ Generating multi-word outputs
12:30 โ Tokenization
15:35 โ Pretraining datasets
21:53 โ LLM architecture
27:20 โ Pretraining
35:21 โ Classification finetuning
39:48 โ Instruction finetuning
43:06 โ Preference finetuning
46:04 โ Evaluating LLMs
53:59 โ Pretraining & finetuning rules of thumb
source
