llm.cpp

Making LLMs from scratch in CPP to build a more fundamental understanding of AI & machines a la Karpathy. Things I plan to focus on:

Produce Code & Write

• Focus on producing lots of code.
• Write interleaved with code in markdown.
• Produce markdown box & arrow diagrams to explain concepts frequently.
• Produce DOT diagrams where more detail is needed.
• Write concise tests that keep results stable.
• Adapt a popular clang formatting style that you like right away.

Iterations on CUDA GPU Kernels

• Detailed descriptions & comparisons between kernels.
• Not skipping to optimal, but exercising creative problem solving skills through all the intermediate steps.
• Look at the PTX being generated & get comfortable with it, even write it.
• Add AMD, Metal, & other accelerators down the road.

Memory Efficiency & Management

• What are some of the ways different projects do this & why?

System Design

• Rewrite the code numerous times exploring the different design decisions.
• Static vs Dynamic computation graphs, is this something we need to think about?
• Can we write declaritive code in something like a DSL (ONNX) & then compile it into something fast?

Study Framework & Projects

• Try out lot of different things.
• Write about the design decisions people have made.