Unstructured Data & Vector Embeddings in Modern Data Pipelines

For decades, data platforms focused almost exclusively on structured data — rows and columns in relational tables. But 80–90% of enterprise data is unstructured: emails, documents, call recordings, product images, sensor readings, social media posts, and PDFs.

The rise of Large Language Models (LLMs) and embedding models has given us a new way to work with this data — vector embeddings — and opened entirely new architectural patterns for AI-powered applications.

Types of Unstructured Data

Processing Unstructured Data: The Pipeline

Stage 1: Ingestion

Unstructured data lands in the Bronze zone of a data lake in its original format:

Documents (PDF/Word) → ADLS/S3 container (raw/)
Images (JPEG/PNG)   → ADLS/S3 container (raw/)
Audio (WAV/MP3)     → ADLS/S3 container (raw/)

Use object storage metadata tags for organisation:

• content_type: application/pdf
• source_system: contracts_crm
• ingestion_date: 2025-09-19

Stage 2: Extraction & Pre-processing

Convert unstructured files into machine-processable representations:

Stage 3: Chunking (for Text)

Long documents must be split into chunks before embedding — embedding models have token limits (typically 512–8192 tokens):

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,        # tokens per chunk
    chunk_overlap=50,      # overlap for context continuity
    separators=["\n\n", "\n", " ", ""]
)
chunks = splitter.split_text(document_text)

Chunking strategies:

• Fixed-size: simple, uniform chunk sizes
• Sentence-aware: split at sentence boundaries to preserve semantic units
• Paragraph-aware: split at natural document section breaks
• Semantic chunking: use an embedding model to detect when topic shifts (advanced)

Vector Embeddings

A vector embedding is a numerical representation of content — a list of floating-point numbers (typically 768 to 4096 dimensions) that encode the semantic meaning of the content.

"The Eiffel Tower is in Paris"  →  [0.23, -0.17, 0.89, 0.04, ...]  (768 numbers)
"La Tour Eiffel est à Paris"    →  [0.24, -0.16, 0.87, 0.05, ...]  (very similar!)
"Apple stock rose 5% today"     →  [0.91, 0.03, -0.44, 0.76, ...]  (very different)

Key property: semantically similar content has similar vectors (high cosine similarity). This enables similarity search — finding documents “about the same thing” without exact keyword matching.

Embedding Models

from openai import OpenAI

client = OpenAI()
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="What is the return policy for damaged goods?"
)
embedding = response.data[0].embedding  # list of 1536 floats

Vector Databases

A vector database is optimised for storing embeddings and performing fast approximate nearest neighbour (ANN) search — finding the most similar vectors to a query vector.

How Vector Search Works

Query: "What are the refund policies?"
   ↓
Embed query using same model  →  [0.34, -0.22, 0.78, ...]
   ↓
Find K nearest vectors in database (cosine/dot-product similarity)
   ↓
Return top-K most semantically similar chunks
   [1] "Returns accepted within 30 days with receipt..." (similarity: 0.94)
   [2] "For damaged items, refunds are processed in..."  (similarity: 0.91)
   [3] "Store credit issued for all returns over $50..." (similarity: 0.87)

ANN Algorithms

Vector Database Options

RAG: Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) is the dominant architecture for building LLM-powered applications that answer questions using your private data.

The Problem RAG Solves

LLMs are trained on public internet data up to a knowledge cutoff. They don’t know:

• Your company’s internal policies
• Your product documentation
• Your customer service history
• Your proprietary research

RAG solves this by retrieving relevant context from your private knowledge base and injecting it into the LLM prompt at query time.

RAG Architecture

                    ┌─────────────────────────────────────────────┐
INDEXING (offline)  │                                             │
                    │  Documents → Chunk → Embed → Vector Store   │
                    │                                             │
                    └─────────────────────────────────────────────┘

QUERYING (online)
User Question
     │
     ├──1. Embed question using same embedding model
     │
     ├──2. Query vector store for top-K similar chunks
     │
     ├──3. Build prompt:
     │       "Context: {retrieved chunks}\n\nQuestion: {user question}\nAnswer:"
     │
     ├──4. Send to LLM (GPT-4, Claude, Gemini, Llama)
     │
     └──5. Return grounded, cited answer to user

Advanced RAG Patterns

Hybrid Search: combine vector similarity search with traditional BM25 keyword search for better recall:

# Azure AI Search: hybrid query
results = search_client.search(
    search_text="refund policy",          # BM25 keyword
    vector_queries=[VectorizedQuery(       # vector similarity
        vector=query_embedding,
        k_nearest_neighbors=5,
        fields="content_vector"
    )],
    query_type="semantic"
)

Re-ranking: use a cross-encoder model to re-score the top-K retrieved chunks for more accurate relevance ordering before passing to the LLM.

Multi-hop Retrieval: for complex questions requiring multiple facts, retrieve → generate sub-queries → retrieve again — iterating until sufficient context is gathered.

Integrating Unstructured Data into Your Data Platform

In a Lakehouse (Databricks)

# Store raw files in Unity Catalog volumes (Bronze)
# spark reads PDFs, extracts text, chunks it, embeds it

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
# Use Databricks Vector Search with Unity Catalog integration
# Delta table → auto-sync to vector search index

index = w.vector_search_indexes.create(
    name="catalog.schema.policy_index",
    endpoint_name="my_endpoint",
    primary_key="id",
    embedding_source_column="text_chunk",
    embedding_model_endpoint_name="databricks-bge-large-en"
)

In Microsoft Fabric

• Store documents in OneLake (ADLS Gen2 via OneLake volumes)
• Use Azure AI Document Intelligence for extraction (connected to Fabric via Data Factory)
• Embed with Azure OpenAI (text-embedding-3-small)
• Store vectors in Azure AI Search (hybrid search over OneLake content)
• Serve via Fabric Eventhouse (KQL) for vector + metadata queries

When to Add Unstructured Data to Your Platform

✅ Good fit when:

• You have significant volumes of documents, emails, or records that contain business-critical information not captured in structured databases
• AI-powered search, Q&A, or summarisation is a key business use case
• Regulatory compliance requires analysis of communications, contracts, or filings
• You are building LLM-powered products that need to answer questions from private knowledge

❌ Wait when:

• Your structured analytics foundation is immature — get that right first
• You don’t have a clear use case driving unstructured data investment
• Your team lacks ML/AI engineering capability to operate embedding pipelines and vector indexes