Tag Archives: Llm

Build an LLM Text Processing Pipeline: Tokenization & Vocabulary [Day -2]

Why every developer should understand the fundamentals of language model processing

TL;DR

Text processing is the foundation of all language model applications, yet most developers use pre-built libraries without understanding the underlying mechanics. In this Day 2 tutorial of our learning journey, I’ll walk you through building a complete text processing pipeline from scratch using Python. You’ll implement tokenization strategies, vocabulary building, word embeddings, and a simple language model with interactive visualizations. The focus is on understanding how each component works rather than using black-box solutions. By the end, you’ll have created a modular, well-structured text processing system for language models that runs locally, giving you deeper insights into how tools like ChatGPT process language at their core. Get ready for a hands-on, question-driven journey into the fundamentals of LLM text processing!

Introduction: Why Text Processing Matters for LLMs

Have you ever wondered what happens to your text before it reaches a language model like ChatGPT? Before any AI can generate a response, raw text must go through a sophisticated pipeline that transforms it into a format the model can understand. This processing pipeline is the foundation of all language model applications, yet it’s often treated as a black box.

In this Day 2 project of our learning journey, we’ll demystify the text processing pipeline by building each component from scratch. Instead of relying on pre-built libraries that hide the inner workings, we’ll implement our own tokenization, vocabulary building, word embeddings, and a simple language model. This hands-on approach will give you a deeper understanding of the fundamentals that power modern NLP applications.

What sets our approach apart is a focus on question-driven development — we’ll learn by doing. At each step, we’ll pose real development questions and challenges (e.g., “How do different tokenization strategies affect vocabulary size?”) and solve them hands-on. This way, you’ll build a genuine understanding of text processing rather than just following instructions.

Learning Note: Text processing transforms raw text into numerical representations that language models can work with. Understanding this process gives you valuable insights into why models behave the way they do and how to optimize them for your specific needs.

Project Overview: A Complete Text Processing Pipeline

The Concept

We’re building a modular text processing pipeline that transforms raw text into a format suitable for language models and includes visualization tools to understand what’s happening at each step. The pipeline includes text cleaning, multiple tokenization strategies, vocabulary building with special tokens, word embeddings with dimensionality reduction visualizations, and a simple language model for text generation. We’ll implement this with a clean Streamlit interface for interactive experimentation.

Key Learning Objectives

Tokenization Strategies: Implement and compare different approaches to breaking text into tokens
Vocabulary Management: Build frequency-based vocabularies with special token handling
Word Embeddings: Create and visualize vector representations that capture semantic meaning
Simple Language Model: Implement a basic LSTM model for text generation
Visualization Techniques: Use interactive visualizations to understand abstract NLP concepts
Project Structure: Design a clean, maintainable code architecture

Learning Note: What is tokenization? Tokenization is the process of breaking text into smaller units (tokens) that a language model can process. These can be words, subwords, or characters. Different tokenization strategies dramatically affect a model’s abilities, especially with rare words or multilingual text.

Project Structure

I’ve organized the project with the following structure to ensure clarity and easy maintenance:

GitHub Repository: day-02-text-processing-pipeline

day-02-text-processing-pipeline/
│
├── data/                       # Data directory
│   ├── raw/                    # Raw input data
│   │   └── sample_headlines.txt  # Sample text data
│   └── processed/              # Processed data outputs
│
├── src/                        # Source code
│   ├── preprocessing/          # Text preprocessing modules
│   │   ├── cleaner.py          # Text cleaning utilities
│   │   └── tokenization.py     # Tokenization implementations
│   │
│   ├── vocabulary/             # Vocabulary building
│   │   └── vocab_builder.py    # Vocabulary construction
│   │
│   ├── models/                 # Model implementations
│   │   ├── embeddings.py       # Word embedding utilities
│   │   └── language_model.py   # Simple language model
│   │
│   └── visualization/          # Visualization utilities
│       └── visualize.py        # Plotting functions
│
├── notebooks/                  # Jupyter notebooks
│   ├── 01_tokenization_exploration.ipynb
│   └── 02_language_model_exploration.ipynb
│
├── tests/                      # Unit tests
│   ├── test_preprocessing.py
│   ├── test_vocabulary.py
│   ├── test_embeddings.py
│   └── test_language_model.py
│
├── app.py                      # Streamlit interactive application
├── requirements.txt            # Project dependencies
└── README.md                   # Project documentation

The Architecture: How It All Fits Together

Our pipeline follows a clean, modular architecture where data flows through a series of transformations:

Let’s explore each component of this architecture:

1. Text Preprocessing Layer

The preprocessing layer handles the initial transformation of raw text:

Text Cleaning (src/preprocessing/cleaner.py): Normalizes text by converting to lowercase, removing extra whitespace, and handling special characters.
Tokenization (src/preprocessing/tokenization.py): Implements multiple strategies for breaking text into tokens:
Basic word tokenization (splits on whitespace with punctuation handling)
Advanced tokenization (more sophisticated handling of special characters)
Character tokenization (treats each character as a separate token)

Learning Note: Different tokenization strategies have significant tradeoffs. Word-level tokenization creates larger vocabularies but handles each word as a unit. Character-level has tiny vocabularies but requires longer sequences. Subword methods like BPE offer a middle ground, which is why they’re used in most modern LLMs.

2. Vocabulary Building Layer

The vocabulary layer creates mappings between tokens and numerical IDs:

Vocabulary Construction (src/vocabulary/vocab_builder.py): Builds dictionaries mapping tokens to unique IDs based on frequency.
Special Tokens: Adds utility tokens like <|unk|> (unknown), <|endoftext|>, [BOS] (beginning of sequence), and [EOS] (end of sequence).
Token ID Conversion: Transforms text to sequences of token IDs that models can process.

3. Embedding Layer

The embedding layer creates vector representations of tokens:

Embedding Creation (src/models/embeddings.py): Initializes vector representations for each token.
Embedding Visualization: Projects high-dimensional embeddings to 2D using PCA or t-SNE for visualization.
Semantic Analysis: Provides tools to explore relationships between words in the embedding space

4. Language Model Layer

The model layer implements a simple text generation system:

Model Architecture (src/models/language_model.py): Defines an LSTM-based neural network for sequence prediction.
Text Generation: Using the model to produce new text based on a prompt.
Temperature Control: Adjusting the randomness of generated text.

5. Interactive Interface Layer

The user interface provides interactive exploration of the pipeline:

Streamlit App (app.py): Creates a web interface for experimenting with all pipeline components.
Visualization Tools: Interactive charts and visualizations that help understand abstract concepts.
Parameter Controls: Sliders and inputs for adjusting model parameters and seeing results in real-time.

By separating these components, the architecture allows you to experiment with different approaches at each layer. For example, you could swap the tokenization strategy without affecting other parts of the pipeline, or try different embedding techniques while keeping the rest constant.

Data Flow: From Raw Text to Language Model Input

To understand how our pipeline processes text, let’s follow the journey of a sample sentence from raw input to model-ready format:

In this diagram, you can see how raw text transforms through each step:

Raw Text: “The quick brown fox jumps over the lazy dog.”
Text Cleaning: Conversion to lowercase, whitespace normalization
Tokenization: Breaking into tokens like [“the”, “quick”, “brown”, …]
Vocabulary Mapping: Converting tokens to IDs (e.g., “the” → 0, “quick” → 1, …)
Embedding: Transforming IDs to vector representations
Language Model: Processing embedded sequences for prediction or generation

This end-to-end flow demonstrates how text gradually transforms from human-readable format to the numerical representations that language models require.

Key Implementation Insights

GitHub Repository: day-02-text-processing-pipeline

Multiple Tokenization Strategies

One of the most important aspects of our implementation is the support for different tokenization approaches. In src/preprocessing/tokenization.py, we implement three distinct strategies:

Basic Word Tokenization: A straightforward approach that splits text on whitespace and handles punctuation separately. This is similar to how traditional NLP systems process text.

Advanced Tokenization: A more sophisticated approach that provides better handling of special characters and punctuation. This approach is useful for cleaning noisy text from sources like social media.

Character Tokenization: The simplest approach that treats each character as an individual token. While this creates shorter vocabularies, it requires much longer sequences to represent the same text.

By implementing multiple strategies, we can compare their effects on vocabulary size, sequence length, and downstream model performance. This helps us understand why modern LLMs use more complex methods like Byte Pair Encoding (BPE).

Vocabulary Building with Special Tokens

Our vocabulary implementation in src/vocabulary/vocab_builder.py demonstrates several important concepts:

Frequency-Based Ranking: Tokens are sorted by frequency, ensuring that common words get lower IDs. This is a standard practice in vocabulary design.

Special Token Handling: We explicitly add tokens like <|unk|> for unknown words and [BOS]/[EOS] for marking sequence boundaries. These special tokens are crucial for model training and inference.

Vocabulary Size Management: The implementation includes options to limit vocabulary size, which is essential for practical language models where memory constraints are important.

Word Embeddings Visualization

Perhaps the most visually engaging part of our implementation is the embedding of the visualization in src/models/embeddings.py:

Vector Representation: Each token is a high-dimensional vector, capturing semantic relationships between words.

Dimensionality Reduction: We use techniques like PCA and t-SNE to project these high-dimensional vectors into a 2D space for visualization.

Semantic Clustering: The visualizations reveal how semantically similar words cluster together in the embedding space, demonstrating how embeddings capture meaning.

Simple Language Model Implementation

The language model in src/models/language_model.py demonstrates the core architecture of sequence prediction models:

LSTM Architecture: We use a Long Short-Term Memory network to capture sequential dependencies in text.

Embedding Layer Integration: The model begins by converting token IDs to their embedding representations.

Text Generation: We implement a sampling-based generation approach that can produce new text based on a prompt.

Interactive Exploration with Streamlit

The Streamlit application in app.py ties everything together:

Interactive Input: Users can enter their own text to see how it’s processed through each stage of the pipeline.

Real-Time Visualization: The app displays tokenization results, vocabulary statistics, embedding visualizations, and generated text.

Parameter Tuning: Sliders and controls allow users to adjust model parameters like temperature or embedding dimension and see the effects instantly.

Challenges & Learnings

Challenge 1: Creating Intuitive Visualizations for Abstract Concepts

The Problem: Many NLP concepts like word embeddings are inherently high-dimensional and abstract, making them difficult to visualize and understand.

The Solution: We implemented dimensionality reduction techniques (PCA and t-SNE) to project high-dimensional embeddings into 2D space, allowing users to visualize relationships between words.

What You’ll Learn: Abstract concepts become more accessible when visualized appropriately. Even if the visualizations aren’t perfect representations of the underlying mathematics, they provide intuitive anchors that help develop mental models of complex concepts.

Challenge 2: Ensuring Coherent Component Integration

The Problem: Each component in the pipeline has different input/output requirements. Ensuring these components work together seamlessly is challenging, especially when different tokenization strategies are used.

The Solution: We created a clear data flow architecture with well-defined interfaces between components. Each component accepts standardized inputs and returns standardized outputs, making it easy to swap implementations.

What You’ll Learn: Well-defined interfaces between components are as important as the components themselves. Clear documentation and consistent data structures make it possible to experiment with different implementations while maintaining a functional pipeline.

Results & Impact

By working through this project, you’ll develop several key skills and insights:

Understanding of Tokenization Tradeoffs

You’ll learn how different tokenization strategies affect vocabulary size, sequence length, and the model’s ability to handle out-of-vocabulary words. This understanding is crucial for working with custom datasets or domain-specific language.

Vocabulary Management Principles

You’ll discover how vocabulary design impacts both model quality and computational efficiency. The practices you learn (frequency-based ordering, special tokens, size limitations) are directly applicable to production language model systems.

Embedding Space Intuition

The visualizations help build intuition about how semantic information is encoded in vector spaces. You’ll see firsthand how words with similar meanings cluster together, revealing how models “understand” language.

Model Architecture Insights

Building a simple language model provides the foundation for understanding more complex architectures like Transformers. The core concepts of embedding lookup, sequential processing, and generation through sampling are universal.

Practical Applications

These skills apply directly to real-world NLP tasks:

Custom Domain Adaptation: Apply specialized tokenization for fields like medicine, law, or finance
Resource-Constrained Deployments: Optimize vocabulary size and model architecture for edge devices
Debugging Complex Models: Identify issues in larger systems by understanding fundamental components
Data Preparation Pipelines: Build efficient preprocessing for large-scale NLP applications

Final Thoughts & Future Possibilities

Building a text processing pipeline from scratch gives you invaluable insights into the foundations of language models. You’ll understand that:

Tokenization choices significantly impact vocabulary size and model performance
Vocabulary management involves important tradeoffs between coverage and efficiency
Word embeddings capture semantic relationships in a mathematically useful way
Simple language models can demonstrate core principles before moving to transformers

As you continue your learning journey, this project provides a solid foundation that can be extended in multiple directions:

Implement Byte Pair Encoding (BPE): Add a more sophisticated tokenization approach used by models like GPT
Build a Transformer Architecture: Replace the LSTM with a simple Transformer encoder-decoder
Add Attention Mechanisms: Implement basic attention to improve model performance
Create Cross-Lingual Embeddings: Extend the system to handle multiple languages
Implement Model Fine-Tuning: Add capabilities to adapt pre-trained embeddings to specific domains

What component of the text processing pipeline are you most interested in exploring further? The foundations you’ve built in this project will serve you well as you continue to explore the fascinating world of language models.

This is part of an ongoing series on building practical understanding of LLM fundamentals through hands-on mini-projects. Check out Day 1: Building a Local Q&A Assistant if you missed it, and stay tuned for more installments!

Enterprise LLM Scaling: Architect’s 2025 Blueprint

From Reference Models to Production-Ready Systems

TL;DR

Imagine deploying a cutting-edge Large Language Model (LLM), only to watch it struggle — its responses lagging, its insights outdated — not because of the model itself, but because the data pipeline feeding it can’t keep up. In enterprise AI, even the most advanced LLM is only as powerful as the infrastructure that sustains it. Without a scalable, high-throughput pipeline delivering fresh, diverse, and real-time data, an LLM quickly loses relevance, turning from a strategic asset into an expensive liability.

That’s why enterprise architects must prioritize designing scalable data pipelines — systems that evolve alongside their LLM initiatives, ensuring continuous data ingestion, transformation, and validation at scale. A well-architected pipeline fuels an LLM with the latest information, enabling high accuracy, contextual relevance, and adaptability. Conversely, without a robust data foundation, even the most sophisticated model risks being starved of timely insights, and forced to rely on outdated knowledge — a scenario that stifles innovation and limits business impact.

Ultimately, a scalable data pipeline isn’t just a supporting component — it’s the backbone of any successful enterprise LLM strategy, ensuring these powerful models deliver real, sustained value.

**Enterprise View: LLM Pipeline Within Organizational Architecture**

The Scale Challenge: Beyond Traditional Enterprise Data

LLM data pipelines operate on a scale that surpasses traditional enterprise systems. Consider this comparison with familiar enterprise architectures:

While your data warehouse may manage terabytes of structured data, LLMs necessitate petabytes of diverse content. GPT-4 is reportedly trained on approximately 13 trillion tokens, with estimates suggesting the training data size could be around 1 petabyte. This vast dataset necessitates distributed processing across thousands of specialized computing units. Even a modest LLM project within an enterprise will likely handle data volumes 10–100 times larger than your largest data warehouse.

The Quality Imperative: Architectural Implications

For enterprise architects, data quality in LLM pipelines presents unique architectural challenges that go beyond traditional data governance frameworks.

A Fortune 500 manufacturer discovered this when their customer-facing LLM began generating regulatory advice containing subtle inaccuracies. The root cause wasn’t a code issue but an architectural one: their traditional data quality frameworks, designed for transactional consistency, failed to address semantic inconsistencies in training data. The resulting compliance review and remediation cost $4.3 million and required a complete architectural redesign of their quality assurance layer.

The Enterprise Integration Challenge

LLM pipelines must seamlessly integrate with your existing enterprise architecture while introducing new patterns and capabilities.

Traditional enterprise data integration focuses on structured data with well-defined semantics, primarily flowing between systems with stable interfaces. Most enterprise architects design for predictable data volumes with predetermined schema and clear lineage.

LLM data architecture, however, must handle everything from structured databases to unstructured documents, streaming media, and real-time content. The processing complexity extends beyond traditional ETL operations to include complex transformations like tokenization, embedding generation, and bias detection. The quality assurance requirements incorporate ethical dimensions not typically found in traditional data governance frameworks.

The Governance and Compliance Imperative

For enterprise architects, LLM data governance extends beyond standard regulatory compliance.

The EU’s AI Act and similar emerging regulations explicitly mandate documentation of training data sources and processing steps. Non-compliance can result in significant penalties, including fines of up to €35 million or 7% of the company’s total worldwide annual turnover for the preceding financial year, whichever is higher. This has significant architectural implications for traceability, lineage, and audit capabilities that must be designed into the system from the outset.

The Architectural Cost of Getting It Wrong

Beyond regulatory concerns, architectural missteps in LLM data pipelines create enterprise-wide impacts:

For instance, a company might face substantial financial losses if data contamination goes undetected in its pipeline, leading to the need to discard and redo expensive training runs.
A healthcare AI startup delayed its market entry by 14 months due to pipeline scalability issues that couldn’t handle its specialized medical corpus
A financial services company found their data preprocessing costs exceeding their model training costs by 5:1 due to inefficient architectural patterns

As LLM initiatives become central to digital transformation, the architectural decisions you make today will determine whether your organization can effectively harness these technologies at scale.

The Architectural Solution Framework

Enterprise architects need a reference architecture for LLM data pipelines that addresses the unique challenges of scale, quality, and integration within an organizational context.

Reference Architecture: Six Architectural Layers

The reference architecture for LLM data pipelines consists of six distinct architectural layers, each addressing specific aspects of the data lifecycle:

Data Source Layer: Interfaces with diverse data origins including databases, APIs, file systems, streaming sources, and web content
Data Ingestion Layer: Provides adaptable connectors, buffer systems, and initial normalization services
Data Processing Layer: Handles cleaning, tokenization, deduplication, PII redaction, and feature extraction
Quality Assurance Layer: Implements validation rules, bias detection, and drift monitoring
Data Storage Layer: Manages the persistence of data at various stages of processing
Orchestration Layer: Coordinates workflows, handles errors, and manages the overall pipeline lifecycle

Unlike traditional enterprise data architectures that often merge these concerns, the strict separation enables independent scaling, governance, and evolution of each layer — a critical requirement for LLM systems.

**Architectural Decision Framework for LLM Data Pipelines**

Architectural Principles for LLM Data Pipelines

Enterprise architects should apply these foundational principles when designing LLM data pipelines:

Key Architectural Patterns

The cornerstone of effective LLM data pipeline architecture is modularity — breaking the pipeline into independent, self-contained components that can be developed, deployed, and scaled independently.

When designing LLM data pipelines, several architectural patterns have proven particularly effective:

Event-Driven Architecture: Using message queues and pub/sub mechanisms to decouple pipeline components, enhancing resilience and enabling independent scaling.
Lambda Architecture: Combining batch processing for historical data with stream processing for real-time data — particularly valuable when LLMs need to incorporate both archived content and fresh data.
Tiered Processing Architecture: Implementing multiple processing paths optimized for different data characteristics and quality requirements. This allows fast-path processing for time-sensitive data alongside deep processing for complex content.
Quality Gate Pattern: Implementing progressive validation that increases in sophistication as data moves through the pipeline, with clear enforcement policies at each gate.
Polyglot Persistence Pattern: Using specialized storage technologies for different data types and access patterns, recognizing that no single storage technology meets all LLM data requirements.

Selecting the right pattern mix depends on your specific organizational context, data characteristics, and strategic objectives.

Architectural Components in Depth

Let’s explore the architectural considerations for each component of the LLM data pipeline reference architecture.

Data Source Layer Design

The data source layer must incorporate diverse inputs while standardizing their integration with the pipeline — a design challenge unique to LLM architectures.

Key Architectural Considerations:

Source Classification Framework: Design a system that classifies data sources based on:

Data velocity (batch vs. streaming)
Structural characteristics (structured, semi-structured, unstructured)
Reliability profile (guaranteed delivery vs. best effort)
Security requirements (public vs. sensitive)

Connector Architecture: Implement a modular connector framework with:

Standardized interfaces for all source types
Version-aware adapters that handle schema evolution
Monitoring hooks for data quality and availability metrics
Circuit breakers for source system failures

Access Pattern Optimization: Design source access patterns based on:

Pull-based retrieval for stable, batch-oriented sources
Push-based for real-time, event-driven sources
Change Data Capture (CDC) for database sources
Streaming integration for high-volume continuous sources

Enterprise Integration Considerations:

When integrating with existing enterprise systems, carefully evaluate:

Impacts on source systems (load, performance, availability)
Authentication and authorization requirements across security domains
Data ownership and stewardship boundaries
Existing enterprise integration patterns and standards

Quality Assurance Layer Design

The quality assurance layer represents one of the most architecturally significant components of LLM data pipelines, requiring capabilities beyond traditional data quality frameworks.

Key Architectural Considerations:

Multidimensional Quality Framework: Design a quality system that addresses multiple dimensions:

Accuracy: Correctness of factual content
Completeness: Presence of all necessary information
Consistency: Internal coherence and logical flow
Relevance: Alignment with intended use cases
Diversity: Balanced representation of viewpoints and sources
Fairness: Freedom from harmful biases
Toxicity: Absence of harmful content

Progressive Validation Architecture: Implement staged validation:

Early-stage validation for basic format and completeness
Mid-stage validation for content quality and relevance
Late-stage validation for context-aware quality and bias detection

Quality Enforcement Strategy: Design contextual quality gates based on:

Blocking gates for critical quality dimensions
Filtering approaches for moderate concerns
Weighting mechanisms for nuanced quality assessment
Transformation paths for fixable quality issues

Enterprise Governance Considerations:

When integrating with enterprise governance frameworks:

Align quality metrics with existing data governance standards
Extend standard data quality frameworks with LLM-specific dimensions
Implement automated reporting aligned with governance requirements
Create clear paths for quality issue escalation and resolution

Security and Compliance Considerations

Architecting LLM data pipelines requires comprehensive security and compliance controls that extend throughout the entire stack.

Key Architectural Considerations:

Identity and Access Management: Design comprehensive IAM controls that:

Implement fine-grained access control at each pipeline stage
Integrate with enterprise authentication systems
Apply principle of least privilege throughout
Provide separation of duties for sensitive operations
Incorporate role-based access aligned with organizational structure

Data Protection: Implement protection mechanisms including:

Encryption in transit between all components
Encryption at rest for all stored data
Tokenization for sensitive identifiers
Data masking for protected information
Key management integrated with enterprise systems

Compliance Frameworks: Design for specific regulatory requirements:

GDPR and privacy regulations requiring data minimization and right-to-be-forgotten
Industry-specific regulations (HIPAA, FINRA, etc.) with specialized requirements
AI-specific regulations like the EU AI Act requiring documentation and risk assessment
Internal compliance requirements and corporate policies

Enterprise Security Integration:

When integrating with enterprise security frameworks:

Align with existing security architecture principles and patterns
Leverage enterprise security monitoring and SIEM systems
Incorporate pipeline-specific security events into enterprise monitoring
Participate in organization-wide security assessment and audit processes

Architectural Challenges & Solutions

When implementing LLM data pipelines, enterprise architects face several recurring challenges that require thoughtful architectural responses.

Challenge #1: Managing the Scale-Performance Tradeoff

The Problem: LLM data pipelines must balance massive scale with acceptable performance. Traditional architectures force an unacceptable choice between throughput and latency.

Architectural Solution:

We implemented a hybrid processing architecture with multiple processing paths to effectively balance scale and performance:

Hybrid Processing Architecture for Scale-Performance Balance

Intelligent Workload Classification: We designed an intelligent routing layer that classifies incoming data based on:

Complexity of required processing
Quality sensitivity of the content
Time sensitivity of the data
Business value to downstream LLM applications

Multi-Path Processing Architecture: We implemented three distinct processing paths:

Fast Path: Optimized for speed with simplified processing, handling time-sensitive or structurally simple data (~10% of volume)
Standard Path: Balanced approach processing the majority of data with full but optimized processing (~60% of volume)
Deep Processing Path: Comprehensive processing for complex, high-value data requiring extensive quality checks and enrichment (~30% of volume)

Resource Isolation and Optimization: Each path’s infrastructure is specially tailored:

Fast Path: In-memory processing with high-performance computing resources
Standard Path: Balanced memory/disk approach with cost-effective compute
Deep Path: Storage-optimized systems with specialized processing capabilities

Architectural Insight: The classification system is implemented as an event-driven service that acts as a smart router, examining incoming data characteristics and routing to the appropriate processing path based on configurable rules. This approach increases overall throughput while maintaining appropriate quality controls based on data characteristics and business requirements.

Challenge #2: Ensuring Data Quality at Architectural Scale

The Problem: Traditional quality control approaches that rely on manual review or simple rule-based validation cannot scale to handle LLM data volumes. Yet quality issues in training data severely compromise model performance.

One major financial services firm discovered that 22% of their LLM’s hallucinations could be traced directly to quality issues in their training data that escaped detection in their pipeline.

Architectural Solution:

We implemented a multi-layered quality architecture with progressive validation:

The diagram will provide visual reinforcement of how data flows through the four validation layers (structural, statistical, ML-based semantic, and targeted human validation), showing the increasingly sophisticated quality checks at each stage.

Layered Quality Framework: We designed a validation pipeline with increasing sophistication:

Layer 1 — Structural Validation: Fast, rule-based checks for format integrity
Layer 2 — Statistical Quality Control: Distribution-based checks to detect anomalies
Layer 3 — ML-Based Semantic Validation: Smaller models validating content for larger LLMs
Layer 4 — Targeted Human Validation: Intelligent sampling for human review of critical cases

Quality Scoring System: We developed a composite quality scoring framework that:

Assigns weights to different quality dimensions based on business impact
Creates normalized scores across disparate checks
Implements domain-specific quality scoring for specialized content
Tracks quality metrics through the pipeline for trend analysis

Feedback Loop Integration: We established connections between model performance and data quality:

Tracing model errors back to training data characteristics
Automatically adjusting quality thresholds based on downstream impact
Creating continuous improvement mechanisms for quality checks
Implementing quality-aware sampling for model evaluation

Architectural Insight: The quality framework design pattern separates quality definition from enforcement mechanisms. This allows business stakeholders to define quality criteria while architects design the optimal enforcement approach for each criterion. For critical dimensions (e.g., regulatory compliance), we implement blocking gates, while for others (e.g., style consistency), we use weighting mechanisms that influence but don’t block processing.

Challenge #3: Governance and Compliance at Scale

The Problem: Traditional governance frameworks aren’t designed for the volume, velocity, and complexity of LLM data pipelines. Manual governance processes become bottlenecks, yet regulatory requirements for AI systems are becoming more stringent.

Architectural Solution:

The diagram visually represents how policies flow from definition through implementation to enforcement, with feedback loops between the layers. It illustrates the relationship between regulatory requirements, corporate policies, and their technical implementation through specialized services.

We implemented an automated governance framework with three architectural layers:

Policy Definition Layer: We created a machine-readable policy framework that:

Translates regulatory requirements into specific validation rules
Codifies corporate policies into enforceable constraints
Encodes ethical guidelines into measurable criteria
Defines data standards as executable quality checks

Policy Implementation Layer: We built specialized services to enforce policies:

Data Protection: Automated PII detection, data masking, and consent verification
Bias Detection: Algorithmic fairness analysis across demographic dimensions
Content Filtering: Toxicity detection, harmful content identification
Attribution: Source tracking, usage rights verification, license compliance checks

Enforcement & Monitoring Layer: We created a unified system to:

Enforce policies in real-time at multiple pipeline control points
Generate automated compliance reports for regulatory purposes
Provide dashboards for governance stakeholders
Manage policy exceptions with appropriate approvals

Architectural Insight: The key architectural innovation is the complete separation of policy definition (the “what”) from policy implementation (the “how”). Policies are defined in a declarative, machine-readable format that stakeholders can review and approve, while technical implementation details are encapsulated in the enforcement services. This enables non-technical governance stakeholders to understand and validate policies while allowing engineers to optimize implementation.

Results & Impact

Implementing a properly architected data pipeline for LLMs delivers transformative results across multiple dimensions:

Performance Improvements

Processing Throughput: Increased from 500GB–1TB/day to 10–25TB/day, representing a 10–25 times improvement.
End-to-End Pipeline Latency: Reduced from 7–14 days to 8–24 hours (85–95% reduction)
Data Freshness: Improved from 30+ days to 1–2 days (93–97% reduction) from source to training
Processing Success Rate: Improved from 85–90% to 99.5%+ (~10% improvement)
Resource Utilization: Increased from 30–40% to 70–85% (~2x improvement)
Scaling Response Time: Decreased from 4–8 hours to 5–15 minutes (95–98% reduction)

These performance gains translate directly into business value: faster model iterations, more current knowledge in deployed models, and greater agility in responding to changing requirements.

Quality Enhancements

The architecture significantly improved data quality across multiple dimensions:

Factual Accuracy: Improved from 75–85% to 92–97% accuracy in training data, resulting in 30–50% reduction in factual hallucinations
Duplication Rate: Reduced from 8–15% to <1% (>90% reduction)
PII Detection Accuracy: Improved from 80–90% to 99.5%+ (~15% improvement)
Bias Detection Coverage: Expanded from limited manual review to comprehensive automated detection
Format Consistency: Improved from widely varying to >98% standardized (~30% improvement)
Content Filtering Precision: Increased from 70–80% to 90–95% (~20% improvement)

Architectural Evolution and Future Directions

As enterprise architects design LLM data pipelines, it’s critical to consider how the architecture will evolve over time. Our experience suggests a four-stage evolution path:

This stage represents the architectural north star — a pipeline that can largely self-manage, continuously adapt, and require minimal human intervention for routine operations.

Emerging Architectural Trends

Looking ahead, several emerging architectural patterns will shape the future of LLM data pipelines:

AI-Powered Data Pipelines: Self-optimizing pipelines using AI to adjust processing strategies, detect quality issues, and allocate resources will become standard. This meta-learning approach — using ML to improve ML infrastructure — will dramatically reduce operational overhead.
Federated Data Processing: As privacy regulations tighten and data sovereignty concerns grow, processing data at or near its source without centralization will become increasingly important. This architectural approach addresses privacy and regulatory concerns while enabling secure collaboration across organizational boundaries.
Semantic-Aware Processing: Future pipeline architectures will incorporate deeper semantic understanding of content, enabling more intelligent filtering, enrichment, and quality control through content-aware components that understand meaning rather than just structure.
Zero-ETL Architecture: Emerging approaches aim to reduce reliance on traditional extract-transform-load patterns by enabling more direct integration between data sources and consumption layers, thereby minimizing intermediate transformations while preserving governance controls.

Key Takeaways for Enterprise Architects

As enterprise architects designing LLM data pipelines, we recommend focusing on these critical architectural principles:

Embrace Modularity as Non-Negotiable: Design pipeline components with clear boundaries and interfaces to enable independent scaling and evolution. This modularity isn’t an architectural nicety but an essential requirement for managing the complexity of LLM data pipelines.
Prioritize Quality by Design: Implement multi-dimensional quality frameworks that move beyond simple validation to comprehensive quality assurance. The quality of your LLM is directly bounded by the quality of your training data, making this an architectural priority.
Design for Cost Efficiency: Treat cost as a first-class architectural concern by implementing tiered processing, intelligent resource allocation, and data-aware optimizations from the beginning. Cost optimization retrofitted later is exponentially more difficult.
Build Observability as a Foundation: Implement comprehensive monitoring covering performance, quality, cost, and business impact metrics. LLM data pipelines are too complex to operate without deep visibility into all aspects of their operation.
Establish Governance Foundations Early: Integrate compliance, security, and ethical considerations into the architecture from day one. These aspects are significantly harder to retrofit and can become project-killing constraints if discovered late.

As LLMs continue to transform organizations, the competitive advantage will increasingly shift from model architecture to data pipeline capabilities. The organizations that master the art and science of scalable data pipelines will be best positioned to harness the full potential of Large Language Models.

Thank you for being a part of the community

Before you go:

Be sure to clap and follow the writer ️👏️️
Follow us: X | LinkedIn | YouTube | Newsletter | Podcast | Differ | Twitch
Check out CoFeed, the smart way to stay up-to-date with the latest in tech 🧪
Start your own free AI-powered blog on Differ 🚀
Join our content creators community on Discord 🧑🏻‍💻
For more content, visit plainenglish.io + stackademic.com

How We Built LLM Infrastructure That Actually Works — And What We Learned

A Data Engineer’s Complete Roadmap: From Napkin Diagrams to Production-Ready Architecture

TL;DR

This article provides data engineers with a comprehensive breakdown of the specialized infrastructure needed to effectively implement and manage Large Language Models. We examine the unique challenges LLMs present for traditional data infrastructure, from compute requirements to vector databases. Offering both conceptual explanations and hands-on implementation steps, this guide bridges the gap between theory and practice with real-world examples and solutions. Our approach uniquely combines architectural patterns like RAG with practical deployment strategies to help you build performant, cost-efficient LLM systems.

The Problem (Why Does This Matter?)

Large Language Models have revolutionized how organizations process and leverage unstructured text data. From powering intelligent chatbots to automating content generation and enabling advanced data analysis, LLMs are rapidly becoming essential components of modern data stacks. For data engineers, this represents both an opportunity and a significant challenge.

The infrastructure traditionally used for data management and processing simply wasn’t designed for LLM workloads. Here’s why that matters:

Scale and computational demands are unprecedented. LLMs require massive computational resources that dwarf traditional data applications. While a typical data pipeline might process gigabytes of structured data, LLMs work with billions of parameters and are trained on terabytes of text, requiring specialized hardware like GPUs and TPUs.

Unstructured data dominates the landscape. Traditional data engineering focuses on structured data in data warehouses with well-defined schemas. LLMs primarily consume unstructured text data that doesn’t fit neatly into conventional ETL paradigms or relational databases.

Real-time performance expectations have increased. Users expect LLM applications to respond with human-like speed, creating demands for low-latency infrastructure that can be difficult to achieve with standard setups.

Data quality has different dimensions. While data quality has always been important, LLMs introduce new dimensions of concern, including training data biases, token optimization, and semantic drift over time.

These challenges are becoming increasingly urgent as organizations race to integrate LLMs into their operations. According to a recent survey, 78% of enterprise organizations are planning to implement LLM-powered applications by the end of 2025, yet 65% report significant infrastructure limitations as their primary obstacle.

Without specialized infrastructure designed explicitly for LLMs, data engineers face:

Prohibitive costs from inefficient resource utilization
Performance bottlenecks that impact user experience
Scalability limitations that prevent enterprise-wide adoption
Integration difficulties with existing data ecosystems

“The gap between traditional data infrastructure and what’s needed for effective LLM implementation is creating a new digital divide between organizations that can harness this technology and those that cannot.”

The Solution (Conceptual Overview)

Building effective LLM infrastructure requires a fundamentally different approach to data engineering architecture. Let’s examine the key components and how they fit together.

Core Infrastructure Components

A robust LLM infrastructure rests on four foundational pillars:

Compute Resources: Specialized hardware optimized for the parallel processing demands of LLMs, including:

GPUs (Graphics Processing Units) for training and inference
TPUs (Tensor Processing Units) for TensorFlow-based implementations
CPU clusters for certain preprocessing and orchestration tasks

2. Storage Solutions: Multi-tiered storage systems that balance performance and cost:

Object storage (S3, GCS, Azure Blob) for large training datasets
Vector databases for embedding storage and semantic search
Caching layers for frequently accessed data

3. Networking: High-bandwidth, low-latency connections between components:

Inter-node communication for distributed training
API gateways for service endpoints
Content delivery networks for global deployment

4. Data Management: Specialized tools and practices for handling LLM data:

Data ingestion pipelines for unstructured text
Vector embedding generation and management
Data versioning and lineage tracking

The following comparison highlights the key differences between traditional data infrastructure and LLM-optimized infrastructure:

Key Architectural Patterns

Two architectural patterns have emerged as particularly effective for LLM infrastructure:

1. Retrieval-Augmented Generation (RAG)

RAG enhances LLMs by enabling them to access external knowledge beyond their training data. This pattern combines:

Text embedding models that convert documents into vector representations
Vector databases that store these embeddings for efficient similarity search
Prompt augmentation that incorporates retrieved-context into LLM queries

RAG solves the critical “hallucination” problem where LLMs generate plausible but incorrect information by grounding responses in factual source material.

2. Hybrid Deployment Models

Rather than choosing between cloud and on-premises deployment, a hybrid approach offers optimal flexibility:

Sensitive workloads and proprietary data remain on-premises
Burst capacity and specialized services leverage cloud resources
Orchestration layers manage workload placement based on cost, performance, and compliance needs

This pattern allows organizations to balance control, cost, and capability while avoiding vendor lock-in.

Why This Approach Is Superior

This infrastructure approach offers several advantages over attempting to force-fit LLMs into traditional data environments:

Cost Efficiency: By matching specialized resources to specific workload requirements, organizations can achieve 30–40% lower total cost of ownership compared to general-purpose infrastructure.
Scalability: The distributed nature of this architecture allows for linear scaling as demands increase, avoiding the exponential cost increases typical of monolithic approaches.
Flexibility: Components can be upgraded or replaced independently as technology evolves, protecting investments against the rapid pace of LLM advancement.
Performance: Purpose-built components deliver optimized performance, with inference latency improvements of 5–10x compared to generic infrastructure.

Implementation

Let’s walk through the practical steps to implement a robust LLM infrastructure, focusing on the essential components and configuration.

Step 1: Configure Compute Resources

Set up appropriate compute resources based on your workload requirements:

For Training: High-performance GPU clusters (e.g., NVIDIA A100s) with NVLink for inter-GPU communication
For Inference: Smaller GPU instances or specialized inference accelerators with model quantization
For Data Processing: CPU clusters for preprocessing and orchestration tasks

Consider using auto-scaling groups to dynamically adjust resources based on workload demands.

Step 2: Set Up Distributed Storage

Implement a multi-tiered storage solution:

Object Storage: Set up cloud object storage (S3, GCS) for large datasets and model artifacts
Vector Database: Deploy a vector database (Pinecone, Weaviate, Chroma) for embedding storage and retrieval
Caching Layer: Implement Redis or similar for caching frequent queries and responses

Configure appropriate lifecycle policies to manage storage costs by automatically transitioning older data to cheaper storage tiers.

Step 3: Implement Data Processing Pipelines

Create robust pipelines for processing unstructured text data:

Data Collection: Implement connectors for various data sources (databases, APIs, file systems)
Preprocessing: Build text cleaning, normalization, and tokenization workflows
Embedding Generation: Set up services to convert text into vector embeddings
Vector Indexing: Create processes to efficiently index and update vector databases

Use workflow orchestration tools like Apache Airflow to manage dependencies and scheduling.

Step 4: Configure Model Management

Set up infrastructure for model versioning, deployment, and monitoring:

Model Registry: Establish a central repository for model versions and artifacts
Deployment Pipeline: Create CI/CD workflows for model deployment
Monitoring System: Implement tracking for model performance, drift, and resource utilization
A/B Testing Framework: Build infrastructure for comparing model versions in production

Step 5: Implement RAG Architecture

Set up a Retrieval-Augmented Generation system:

Document Processing: Create pipelines for chunking and embedding documents
Vector Search: Implement efficient similarity search capabilities
Context Assembly: Build services that format retrieved context into prompts
Response Generation: Set up LLM inference endpoints that incorporate retrieved context

Step 6: Deploy a Serving Layer

Create a robust serving infrastructure:

API Gateway: Set up unified entry points with authentication and rate limiting
Load Balancer: Implement traffic distribution across inference nodes
Caching: Add result caching for common queries
Fallback Mechanisms: Create graceful degradation paths for system failures

Challenges & Learnings

Building and managing LLM infrastructure presents several significant challenges. Here are the key obstacles we’ve encountered and how to overcome them:

Challenge 1: Data Drift and Model Performance Degradation

LLM performance often deteriorates over time as the statistical properties of real-world data change from what the model was trained on. This “drift” occurs due to evolving terminology, current events, or shifting user behaviour patterns.

The Problem: In one implementation, we observed a 23% decline in customer satisfaction scores over six months as an LLM-powered support chatbot gradually provided increasingly outdated and irrelevant responses.

The Solution: Implement continuous monitoring and feedback loops:

Regular evaluation: Establish a benchmark test set that’s periodically updated with current data.
User feedback collection: Implement explicit (thumbs up/down) and implicit (conversation abandonment) feedback mechanisms.
Continuous fine-tuning: Schedule regular model updates with new data while preserving performance on historical tasks.

Key Learning: Data drift is inevitable in LLM applications. Build infrastructure with the assumption that models will need ongoing maintenance, not just one-time deployment.

Challenge 2: Scaling Costs vs. Performance

The computational demands of LLMs create a difficult balancing act between performance and cost management.

The Problem: A financial services client initially deployed their document analysis system using full-precision models, resulting in monthly cloud costs exceeding $75,000 with average inference times of 2.3 seconds per query.

The Solution: Implement a tiered serving approach:

Model quantization: Convert models from 32-bit to 8-bit or 4-bit precision, reducing memory footprint by 75%.
Query routing: Direct simple queries to smaller models and complex queries to larger models.
Result caching: Cache common query results to avoid redundant processing.
Batch processing: Aggregate non-time-sensitive requests for more efficient processing.

Key Learning: There’s rarely a one-size-fits-all approach to LLM deployment. A thoughtful multi-tiered architecture that matches computational resources to query complexity can reduce costs by 60–70% while maintaining or even improving performance for most use cases.

Challenge 3: Integration with Existing Data Ecosystems

LLMs don’t exist in isolation; they need to connect with existing data sources, applications, and workflows.

The Problem: A manufacturing client struggled to integrate their LLM-powered equipment maintenance advisor with their existing ERP system, operational databases, and IoT sensor feeds.

The Solution: Develop a comprehensive integration strategy:

API standardization: Create consistent REST and GraphQL interfaces for LLM services.
Data connector framework: Build modular connectors for common data sources (SQL databases, document stores, streaming platforms).
Authentication middleware: Implement centralized auth to maintain security across systems.
Event-driven architecture: Use message queues and event streams to decouple systems while maintaining data flow.

Key Learning: Integration complexity often exceeds model deployment complexity. Allocate at least 30–40% of your infrastructure planning to integration concerns from the beginning, rather than treating them as an afterthought.

Results & Impact

Properly implemented LLM infrastructure delivers quantifiable improvements across multiple dimensions:

Performance Metrics

Organizations that have adopted the architectural patterns described in this guide have achieved remarkable improvements:

Before-and-After Scenarios

Building effective LLM infrastructure represents a significant evolution in data engineering practice. Rather than simply extending existing data pipelines, organizations need to embrace new architectural patterns, hardware configurations, and deployment strategies specifically optimized for language models.

The key takeaways from this guide include:

Specialized hardware matters: The right combination of GPUs, storage, and networking makes an enormous difference in both performance and cost.
Architectural patterns are evolving rapidly: Techniques like RAG and hybrid deployment are becoming standard practice for production LLM systems.
Integration is as important as implementation: LLMs deliver maximum value when seamlessly connected to existing data ecosystems.
Monitoring and maintenance are essential: LLM infrastructure requires continuous attention to combat data drift and optimize performance.

Looking ahead, several emerging trends will likely shape the future of LLM infrastructure:

Hardware specialization: New chip designs specifically optimized for inference workloads will enable more cost-efficient deployments.
Federated fine-tuning: The ability to update models on distributed data without centralization will address privacy concerns.
Multimodal infrastructure: Systems designed to handle text, images, audio, and video simultaneously will become increasingly important.
Automated infrastructure optimization: AI-powered tools that dynamically tune infrastructure parameters based on workload characteristics.

To start your journey of building effective LLM infrastructure, consider these next steps:

Audit your existing data infrastructure to identify gaps that would impact LLM performance
Experiment with small-scale RAG implementations to understand the integration requirements
Evaluate cloud vs. on-premises vs. hybrid approaches based on your organization’s needs
Develop a cost model that captures both direct infrastructure expenses and potential efficiency gains

What challenges are you facing with your current LLM infrastructure, and which architectural pattern do you think would best address your specific use case?

Thank you for being a part of the community

Before you go:

Be sure to clap and follow the writer ️👏️️
Follow us: X | LinkedIn | YouTube | Newsletter | Podcast | Differ
Check out CoFeed, the smart way to stay up-to-date with the latest in tech 🧪
Start your own free AI-powered blog on Differ 🚀
Join our content creators community on Discord 🧑🏻‍💻
For more content, visit plainenglish.io + stackademic.com

Generative AI 101: Building LLM-Powered Application

Application Overview

The system uses machine learning to find answers. It converts documents into numerical vectors and stores them in a Vector Database for easy retrieval. When a user asks a question, the system combines it with the documents using a Prompt Template and sends it to a Language Learning Model (LLM). The LLM generates an answer, which the system displays to the user. This AI-powered system efficiently processes language and finds information.

Step 1: Install Necessary Libraries

First, you’ll need to install the necessary Python libraries. These include tools for handling rich text, connecting to the OpenAI API, and various utilities from the langchain library.

!pip install -Uqqq rich openai tiktoken langchain unstructured tabulate pdf2image chromadb

Step 2: Import Libraries and Set Up API Key

Import the required libraries and ensure that your OpenAI API key is set up. The script will prompt you to enter your API key if it’s not found in your environment variables.

import os
from getpass import getpass

# Check for OpenAI API key and prompt if not found
if os.getenv("OPENAI_API_KEY") is None:
    os.environ["OPENAI_API_KEY"] = getpass("Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n")
assert os.getenv("OPENAI_API_KEY", "").startswith("sk-"), "This doesn't look like a valid OpenAI API key"

Step 3: Set the Model Name

Define the model you’ll be using from OpenAI. In this case, we’re using text-davinci-003, but you can replace it with gpt-4 or any other model you prefer.

MODEL_NAME = "text-davinci-003"

Note: text-davinci-003 is a version of OpenAI’s GPT-3 series, representing the most advanced in the “Davinci” line, known for its sophisticated natural language understanding and generation. As a versatile model, it’s used for a wide range of tasks requiring nuanced context, high-quality outputs, and creative content generation. While powerful, it’s also accessibly costly and requires careful consideration for ethical and responsible use due to its potential to influence and generate a wide array of content.

Step 4: Download Sample Data

Clone a repository containing sample markdown files to use as your data source.

!git clone https://github.com/mundimark/awesome-markdown.git

Note: Ensure the destination path doesn’t already contain a folder with the same name to avoid errors.

Step 5: Load and Prepare Documents

Load the markdown files and prepare them for processing. This involves finding all markdown files in the specified directory and loading them into a format suitable for the langchain library.

from langchain.document_loaders import DirectoryLoader

def find_md_files(directory):
    dl = DirectoryLoader(directory, "**/*.md")
    return dl.load()

documents = find_md_files('awesome-markdown/')

Step 6: Tokenize Documents

Count the number of tokens in each document. This is important to ensure that the inputs to the model don’t exceed the maximum token limit.

import tiktoken

tokenizer = tiktoken.encoding_for_model(MODEL_NAME)
def count_tokens(documents):
    return [len(tokenizer.encode(document.page_content)) for document in documents]
token_counts = count_tokens(documents)
print(token_counts)

Note: TPM stands for “Tokens Per Minute,” a metric indicating the number of tokens the API can process within a minute. For the “gpt-3.5-turbo” model, the limit is set to 40,000 TPM. This means you can send up to 40,000 tokens worth of data to the API for processing every minute. This limit is in place to manage the load on the API and ensure fair usage among consumers. It’s important to design your interactions with the API to stay within these limits to avoid service interruptions or additional fees.

Step 7: Split Documents into Sections

Use the MarkdownTextSplitter to split the documents into manageable sections that the language model can handle.

from langchain.text_splitter import MarkdownTextSplitter

md_text_splitter = MarkdownTextSplitter(chunk_size=1000)
document_sections = md_text_splitter.split_documents(documents)
print(len(document_sections), max(count_tokens(document_sections)))

Step 8: Create Embeddings and a Vector Database

Use OpenAIEmbeddings to convert the text into embeddings, and then store these embeddings in a vector database (Chroma) for fast retrieval.

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma

embeddings = OpenAIEmbeddings()
db = Chroma.from_documents(document_sections, embeddings)

Note: Embeddings in machine learning are numerical representations of text data, transforming words, sentences, or documents into vectors of real numbers so that computers can process natural language. They capture semantic meaning, allowing words with similar context or meaning to have similar representations, which enhances the ability of models to understand and perform tasks like translation, sentiment analysis, and more. Embeddings reduce the complexity of text data and enable advanced functionalities in language models, making them essential for understanding and generating human-like language in various applications.

Step 9: Create a Retriever

Create a retriever from the database to find the most relevant document sections based on your query.

retriever = db.as_retriever(search_kwargs=dict(k=3))

Step 10: Define and Retrieve Documents for Your Query

Define the query you’re interested in and use the retriever to find relevant document sections.

query = "Can you explain the origins and primary objectives of the Markdown language as described by its creators, John Gruber and Aaron Swartz, and how it aims to simplify web writing akin to composing plain text emails?"
docs = retriever.get_relevant_documents(query)

Step 11: Construct and Send the Prompt

Build the prompt by combining the context (retrieved document sections) with the query, and send it to the OpenAI model for answering.

context = "\n\n".join([doc.page_content for doc in docs])
prompt = f"""Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {query}
Helpful Answer:"""

from langchain.llms import OpenAI

llm = OpenAI()
response = llm.predict(prompt)

Step 12: Display the Result

Finally, print out the model’s response to your query.

print(response)

Stackademic

Thank you for reading until the end. Before you go:

Please consider clapping and following the writer! 👏
Follow us on Twitter(X), LinkedIn, and YouTube.
Visit Stackademic.com to find out more about how we are democratizing free programming education around the world.