Course Title: Advanced Applications of Large Language Models (LLMs)
Course Number: COSC 650
Semester: Fall 2024
Credit Hours: 3
Instructors:
This course immerses students in the practical and theoretical aspects of Generative AI and Large Language Models (LLMs). Individual projects will focus on prompt engineering, retrieval-augmented generation (RAG), function calling, fine-tuning techniques, use of multi-modal inputs, and creating AI assistants. The curriculum includes hands-on interaction with LLM APIs, integration into applications using Streamlit, JavaScript/Node.js, LangChain, LangGraph, and ethical AI practices.
- Develop a deep understanding of LLMs and their applications.
- Learn and apply techniques such as prompt engineering, retrieval-augmented generation, function calling, and fine-tuning.
- Design and implement LLM-based solutions for real-world tasks, including integrating AI assistants into interactive applications using frameworks like Streamlit or JavaScript and Node.js.
- Build autonomous LLM-powered AI assistants capable of executing multi-step workflows and integrating them with user interfaces for enhanced interactivity.
- Apply ethical considerations and responsible AI practices in LLM deployment.
- Topics: History of Language Models.
- Taking someone from no LM understanding up to today's state of the art. How did we get here?
- Introduction to Generative AI, Large Language Models (LLMs), key terminology (tokens, embeddings, prompts), real-world applications through a survey of LLMs.
- Discuss the families of LLMs and how each evolved
- OpenAI GPTs
- Meta Llama
- Google Palm
- Anthropic
- Hands-on: Setup, tokenization and embedding exercises, basic prompt creation using LLM APIs.
- Overview: Introduction to upcoming projects and individual objectives.
- Topics: APIs for LLM interaction (OpenAI, Azure OpenAI), calling LLM APIs with various parameters, exploring API-driven generation tasks and then evaluating the LLMs performance.
- Hands-on: Calling LLM APIs for text generation and handling API responses.
- Topics: Advanced prompt engineering techniques, designing effective prompts, common pitfalls, and improvement strategies.
- Hands-on: Experimenting with prompts to achieve desired responses.
- Topics: RAG architecture and concepts, vector search concepts, similarity and distance metrics, indexing strategies, using vectors in document retrieval and LLMs.
- Hands-on: Building a basic RAG pipeline with pre-trained models, implementing a vector search mechanism with a document corpus.
- Topics: Understanding function calling in LLMs, when and how to use function calling in LLM-based applications.
- Hands-on: Setting up LLM function calls for external operations.
- Topics: Overview of LangChain and LlamaIndex, building workflows for LLM-driven applications using LangChain, document management with LlamaIndex.
- Hands-on: Set up a LangChain-based project that utilizes LlamaIndex for managing and retrieving documents.
- Topics: Introduction to AI assistants powered by LLMs, use cases for LLMs as AI assistants, building multi-step workflows.
- Hands-on: Implementing a basic LLM-powered AI assistant using LangChain to solve a task autonomously.
- Topics: Integrating LLMs into interactive applications using frameworks like Streamlit and JavaScript/Node.js, creating user interfaces for AI assistants.
- Hands-on: Build a simple interactive application using Streamlit or JavaScript/Node.js that interfaces with an AI assistant to showcase real-time interactions.
- Topics: Overview of Assistant API, basic use cases and features, integrating it with LLM-driven applications.
- Hands-on: Simple Assistant API integration using OpenAI/Azure OpenAI for generating conversational responses within a web interface.
- Topics: Evaluating chatbot performance, key metrics (accuracy, relevance, user satisfaction), iterating on chatbot design based on feedback and evaluation.
- Hands-on: Evaluate your chatbot using test cases and performance metrics, incorporate improvements based on evaluation results.
- Topics: When and why to fine-tune an LLM, understanding fine-tuning models for specific domains, tools for fine-tuning (Hugging Face, Azure OpenAI).
- Hands-on: Overview and demonstration of fine-tuning on a domain-specific task, integrate the fine-tuned model into an application.
- Topics: Introduction to graph databases (e.g., Neo4j), integrating graph databases to enhance retrieval, graph embeddings, chunking documents, leveraging graph data for enhanced decision-making in LLMs, and using LangGraph for integrating graph structures with language models.
- Hands-on: Creating and querying a simple graph database to augment LLM responses, integrating graph data into your AI assistant using LangGraph, building applications that combine LangChain and LangGraph.
- Topics: Refining and integrating project components into a cohesive application.
- Hands-on: Complete all projects, focusing on integrating LLM functionalities into the interactive application, including graph integration, preparing for final presentations.
- Topics: Ethical considerations in AI development, Microsoft’s Responsible AI principles, implementing fairness, transparency, and accountability in AI systems.
- Hands-on: Analyzing your AI assistant or LLM-based project through the lens of Responsible AI, ensuring ethical AI deployment.
- Project Presentations: Begin final project presentations, incorporating discussions on how Responsible AI practices were implemented.
- Topics: Continuation of final project presentations, peer review, and feedback.
- Hands-on: Final project presentations and Q&A, evaluation based on technical merit, innovation, application of learned material, and integration of Responsible AI practices.
- Course Review: Reflective discussions on key concepts learned, exploration of emerging trends and future directions in LLMs and AI assistants.
- Projects 1-4: Each project 15% (Total 60%)
- Participation and Weekly Assignments: 10%
- Peer Reviews and Interaction: 10%
- Final Presentations: 20%
- Primary Textbook: Building LLMs for Production, Enhancing LLM Abilities and Reliability with Prompting, Fine-Tuning, and RAG by Louis-François Bouchard.
- Possible other Textbook: Large Language Models: A Deep Dive: Bridging Theory and Practice
- Supplementary book materials: https://towardsai.net/book
- Supplementary materials including research articles, online courses, and tutorials.
- Programming languages: Python
- AI frameworks: TensorFlow, PyTorch, OpenAI
- Application frameworks: Streamlit, JavaScript/Node.js
- LLM frameworks: LangChain, LlamaIndex, LangGraph
- Graph databases: Neo4j
- Cloud services and AI platforms: Azure, Open Source
Notes on Changes:
- Added LangGraph to Week 12: Incorporated LangGraph into Week 12: Graph Data Structures and Applications with LangGraph to enhance the focus on integrating graph structures with language models.
- Updated Week 12 Topics and Hands-on Activities: Included topics on using LangGraph for integrating graph data with LLMs and updated hands-on activities to reflect the use of LangGraph.
- Added LangGraph to Software and Tools: Listed LangGraph under LLM frameworks in the Software and Tools section to highlight its importance in the course.
This updated syllabus now includes LangGraph in the graph session, providing students with the opportunity to learn and apply this tool in their projects.