Roadmap: Large Language Models (LLMs)
This roadmap covers the fundamentals of Large Language models, including topics such as retrieval-augmented generation and LLM fine-tuning.
The topics in this roadmap include materials that are collected together in Jupyter notebooks that can be run to reproduce the results contained in each topic page. Access to the notebooks is described in the Lab page at the end of each topic. If you would like to run the code in the notebook as you work through the materials in this topic, consult the Lab pages for information on how to proceed.
These roadmap materials are made available through Chishiki-AI, a transformative project at the forefront of integrating Artificial Intelligence (AI) with Civil and Environmental Engineering (CEE). Chishiki.AI, meaning 'knowledge through AI,' is a pioneering initiative funded by the National Science Foundation and led by a team of experts at the University of Texas at Austin in collaboration with Cornell University.
Chishiki-AI is funded by the National Science Foundation, award #2321040.
Objectives
The objectives for the topics in LLM are:
- Explain what an LLM is and how it generates text
- Briefly understand the architecture of the Transformer
- Know the best practices for using an LLM
- Build a simple RAG pipeline and prepare the data source through embedding
- Run inference with the RAG pipeline
- Distinguish fine-tuning from RAG
- Fine-tune the
gpt-ossmodel and run inference on it
Prerequisites
The prerequisites for the topics in LLM are:
- Experience with programming in Python.
Requirements
To perform the labs outlined in this module you will need an allocation on the Vista cluster from TACC and access to the TACC Analysis Portal.
If you don't have an allocation: Register with the Chishiki-AI project to be added to the TACC account.