Build an LLM from Scratch 2: Working with text data

Links to the book:
– https://amzn.to/4fqvn0D (Amazon)
– https://mng.bz/M96o (Manning)

Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch

This is a supplementary video going over text data preparations steps (tokenization, byte pair encoding, data loaders, etc.) for LLM training.

00:00 2.2 Tokenizing text
14:02 2.3 Converting tokens into token IDs
23:56 2.4 Adding special context tokens
30:26 2.5 Byte pair encoding
44:00 2.6 Data sampling with a sliding window
1:07:10 2.7 Creating token embeddings
1:15:45 2.8 Encoding word positions

You can find additional bonus materials on GitHub:

Byte Pair Encoding (BPE) Tokenizer From Scratch, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

Comparing Various Byte Pair Encoding (BPE) Implementations, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/02_bonus_bytepair-encoder/compare-bpe-tiktoken.ipynb

Understanding the Difference Between Embedding Layers and Linear Layers, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb

Data sampling with a sliding window with number data, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

A video on the effect of random seeds: https://www.youtube.com/watch?v=ii89_SqKB08&feature=youtu.be

source