John Berryman moved from aerospace engineering to search, then to ML and LLMs. His path: Eventbrite search → GitHub code search → data science → GitHub Copilot. He was drawn to more math and ML throughout his career.
**RAG Explained**
"RAG is not a thing. RAG is two things." It breaks into:
1. Search - finding relevant information
2. Prompt engineering - presenting that information to the model
These should be treated as separate problems to optimize.
**The Little Red Riding Hood Principle**
When prompting LLMs, stay on the path of what models have seen in training. Use formats, structures, and patterns they recognize from their training data:
- For code, use docstrings and proper formatting
- For financial data, use SEC report structures
- Use Markdown for better formatting
Models respond better to familiar structures.
**Testing Prompts**
Testing strategies:
- Start with "vibe testing" - human evaluation of outputs
- Develop systematic tests based on observed failure patterns
- Use token probabilities to measure model confidence
- For few-shot prompts, watch for diminishing returns as examples increase
**Managing Token Limits**
When designing prompts, divide content into:
- Static elements (boilerplate, instructions)
- Dynamic elements (user inputs, context)
Prioritize content by:
1. Must-have information
2. Nice-to-have information
3. Optional if space allows
Even with larger context windows, efficiency remains important for cost and latency.
**Completion vs. Chat Models**
Chat models are winning despite initial concerns about their constraints:
- Completion models allow more flexibility in document format
- Chat models are more reliable and aligned with common use cases
- Most applications now use chat models, even for completion-like tasks
**Applications: Workflows vs. Assistants**
Two main LLM application patterns:
- **Assistants**: Human-in-the-loop interactions where users guide and correct
- **Workflows**: Decomposed tasks where LLMs handle well-defined steps with safeguards
**Breaking Down Complex Problems**
Two approaches:
- **Horizontal**: Split into sequential steps with clear inputs/outputs
- **Vertical**: Divide by case type, with specialized handling for each scenario
Example: For SOX compliance, break horizontally (understand control, find evidence, extract data, compile report) and vertically (different audit types).
**On Agents**
Agents exist on a spectrum from assistants to workflows, characterized by:
- Having some autonomy to make decisions
- Using tools to interact with the environment
- Usually requiring human oversight
**Best Practices**
For building with LLMs:
1. Start simple: API key + Jupyter notebook
2. Build prototypes and iterate quickly
3. Add evaluation as you scale
4. Keep users in the loop until models prove reliability
**John Berryman:**
- LinkedIn https://www.linkedin.com/in/john-berryman-864b1713/
- X (Twitter) https://x.com/JnBrymn
- Arcturus Labs https://arcturus-labs.com/
- Prompt Engineering for LLMs (Book) https://www.amazon.de/-/en/John-Berryman-ebook/dp/B0DM3VLNSK/ref=sr_1_1?crid=1DYT2Q98IMLVS&dib=eyJ2IjoiMSJ9.v0f-Y12RTIShwCHImj9g8pzG3cDhXhgln1v6dKvX-ZnUGP67Ned--Ry0_d1Surk48QFpHgGbkhOgGw3jrj09bbXDDK4YCNVMGonsbB09NlE8AhdjmCI4RN-M24AaGSwRgjiThMQym3iqJIkPWp_fTiREzW0NMBHshO2QddeWw_KuEVFFfmq3Q203LK0DjFQdlUvxFhCxDTLMK490SPit-X7MQGRrt-wwmofEGiLUSAE.MjUWoivUthTYo6wqPxfYMqDcgB3l1BBb_raeQngH5S8&dib_tag=se&keywords=john.+berryman+prompt+engineering&qid=1740663682&s=digital-text&sprefix=john.+berryman+prompt+engineering%2Cdigital-text%2C77&sr=1-1
**Nicolay Gerold:**
- LinkedIn. https://www.linkedin.com/in/nicolay-gerold/
- X (Twitter) https://twitter.com/nicolaygerold
00:00 Introduction to RAG: Retrieval and Generation
00:19 Optimizing Retrieval Systems
01:11 Introducing John Berryman
02:31 John's Journey from Search to Prompt Engineering
04:05 Understanding RAG: Search and Prompt Engineering
05:39 The Little Red Riding Hood Principle in Prompt Engineering
14:14 Balancing Static and Dynamic Elements in Prompts
25:52 Assistants vs. Workflows: Choosing the Right Approach
30:15 Defining Agency in AI
30:35 Spectrum of Assistance and Workflows
34:35 Breaking Down Problems Horizontally and Vertically
37:57 SOX Compliance Case Study
40:56 Integrating LLMs into Existing Applications
44:37 Favorite Tools and Missing Features
46:37 Exploring Niche Technologies in AI
52:52 Key Takeaways and Future Directions