Editing Information Retrieval and Neural Search (section)

== <span style="color: #FFFFFF;">Understanding</span> ==
The fundamental challenge of information retrieval: the user's information need and the documents that satisfy it may use entirely different words. "How do I fix a flat tire?" and "Procedures for replacing a deflated automotive tire" have zero word overlap but are semantically equivalent. Bridging this vocabulary mismatch is the central challenge.

**Lexical retrieval** (BM25): Count and weight term occurrences. Fast (inverted index lookup), handles exact matches perfectly, but fails on synonyms, paraphrases, and semantic equivalence. BM25 is the baseline that all modern systems must beat.

**Dense retrieval** (DPR, E5, BGE): Encode query and documents into dense vectors using a transformer model trained for retrieval. Retrieve by finding vectors close to the query vector (cosine similarity or dot product). Understands semantics but slower to index, requires approximate nearest neighbor search for scale, and may miss exact keyword matches.

**Hybrid retrieval**: BM25 handles exact keyword matches; dense retrieval handles semantic similarity. Combining both (e.g., Reciprocal Rank Fusion of BM25 and dense results) outperforms either alone.

**The re-ranking pipeline**: (1) **Retrieval** (top-1000): BM25 or dense retrieval. Fast, high recall. (2) **Re-ranking** (top-10): Cross-encoder scores all 1000 candidates. Slow but highly accurate. This two-stage pipeline balances speed and accuracy.

**Neural IR in RAG**: In retrieval-augmented generation, the IR component retrieves relevant document chunks that are passed to an LLM for response generation. The quality of retrieval directly determines the quality of the generated answer — bad retrieval = bad generation, even with a perfect LLM.
</div>

<div style="background-color: #8B0000; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">