text_extraction_methods_comprehensive_20260307_deep_research

方法论库 · 引用级 · none

本页是 <code>contexts/methodology/text_extraction_methods_comprehensive_20260307_deep_research.md</code> 的逐字投影（仅隐私清洗，零改写）。

name: text_extraction_methods_comprehensive_20260307_deep_research
description: 文本提取/摘要方法综述
domain: infra
consumption:
  surface: none
  trigger: ""
  consumer: orchestrator
status: library
promoted_to: null

Text Extraction and Summarization Methods

This document provides an overview of various methods for text extraction and summarization, covering embeddings, semantic representation, traditional NLP methods, deep learning approaches, and evaluation metrics.

1. Embeddings for Text Extraction

Vector Representations

Embeddings are vectors that represent data in a continuous space. In NLP, embeddings typically represent words, phrases, or sentences, allowing them to be processed mathematically. The key advantage is that semantically similar items are positioned close to each other in the embedding space.

Sentence Embeddings

Sentence embeddings capture the meaning of entire sentences as dense vectors. Popular approaches include:

• BERT-based embeddings: Extract representations from pretrained transformer models like BERT
• Representation-shaping techniques: Methods that significantly improve sentence embeddings from BERT-based models through techniques like unsupervised learning and prompt engineering
• Variational auto-encoder approaches: Sample latent space distributions to obtain isotropic sentence embeddings and utilize supervised contrastive learning to uniform distribution

Dense Retrievers

Dense retrievers use neural networks to encode queries and documents into dense vector representations:

• Dense Passage Retriever (DPR): Facebook AI's popular model for semantic similarity search
• EASE-DR: Enhanced Sentence Embeddings for Dense Retrieval that splits query/document into natural sentences, generates embeddings with BERT, and uses contrastive learning to improve representation quality

Key advantage: Unlike sparse methods (like TF-IDF), dense retrievers can capture semantic relationships beyond exact word matches.

2. Semantic Representation

Knowledge Graphs

Knowledge graphs represent structured information as entities and relationships:

• Information Extraction Pipeline: Converts text to knowledge graphs through entity extraction, relationship categorization, and integration
• KGGen: Uses language models to create high-quality graphs from plaintext, clustering related entities to reduce sparsity
• FREDFoot: State-of-the-art tool for producing framed knowledge graphs using complex NLP pipelines

Semantic Parsing

Semantic parsing converts natural language into structured representations:

• Frame-based parsing: Represents text in neo-Davidsonian logical forms
• Semantic role labeling: Identifies predicate-argument structures in text

Entity Extraction

• Rule-based relationship extraction: Uses grammatical dependencies to extract relationships
• Machine learning approaches: Train NLP models to extract relationships between entity pairs
• Pretrained models: Leverage strong language understanding capabilities for entity and relationship extraction

3. Traditional NLP Methods

TF-IDF (Term Frequency-Inverse Document Frequency)

A classic approach for text representation and retrieval:

• Term Frequency (TF): Measures how often a term appears in a document
• Inverse Document Frequency (IDF): Measures how rare a term is across documents
• Limitation: TF grows linearly with keyword occurrences, which doesn't account for saturation

BM25 (Best Matching 25)

An improved ranking function that addresses TF-IDF limitations:

• Three main components: Term Frequency (TF), Inverse Document Frequency (IDF), and Document Length Normalization
• Variants: BM25F accounts for document structure and anchor text
• Still widely used: As a baseline and in hybrid retrieval systems

Extractive Summarization

Selects important sentences/phrases from the original text:

• TextRank algorithm: Graph-based text summarization method
• Combined approaches: Integrating TextRank with BM25 for improved results
• Evaluation: Uses precision, recall, and F1 score

Keyword Extraction

Identifies the most important terms in a document:

• Statistical methods: Frequency-based approaches
• Graph-based methods: TextRank for keyword extraction
• Combined preprocessing: Using BM25 similarity before keyword extraction

4. Deep Learning Methods

Seq2Seq Models

Sequence-to-sequence models revolutionized NLP tasks:

• Introduced by Google (2014): Originally for machine translation
• Applications: Text summarization, speech recognition, image captioning, question-answering
• Architecture: Encoder-decoder framework that maps input sequences to output sequences

Attention Mechanisms

Attention allows models to focus on relevant parts of the input:

• Self-attention: Tokens look at each other to understand context
• Encoder-decoder attention: Decoder attends to encoder states
• Multiple layers: Typically 6+ layers for complex reasoning

Transformer Networks

Transformers rely on self-attention mechanisms rather than recurrence:

• Long-range dependencies: Capture relationships across entire sequences
• Parallel processing: More efficient than RNN-based approaches
• Pretrained language models: BERT (encoder), GPT (decoder), BART (encoder-decoder)

Modern Summarization Approaches

• BERT + GPT-2: BERT encodes text, GPT-2 generates summaries
• Fine-tuned transformers: Specialized models for abstractive summarization
• Hybrid approaches: Combining extractive and abstractive methods

5. Evaluation Methods

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• Measures n-gram overlap between generated and reference texts
• Variants: ROUGE-N (n-grams), ROUGE-L (longest common subsequence), ROUGE-S (skip-bigrams)
• Limitation: Relies on exact word presence, doesn't capture semantics

BLEU (Bilingual Evaluation Understudy)

• Originally designed for machine translation
• Measures precision of n-grams between candidate and reference translations
• Limitation: Penalizes valid rephrasings and rewards exact matches

BERTScore

• Leverages contextual embeddings from BERT
• Evaluates semantic similarity between predicted and reference texts
• Advantage: Captures meaning beyond surface-level word overlap
• Use case: Particularly valuable for abstractive summarization where wording differs

Human Evaluation

• Fluency: Is the text grammatically correct?
• Coherence: Does the text make sense overall?
• Informative: Does it capture key information?
• Relevance: Are important aspects covered?
• Challenges: Subjective, time-consuming, expensive

Metric Comparison

References

• Pinecone: Dense Vectors - https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/
• EASE-DR: Enhanced Sentence Embeddings - ACM SIGIR 2024
• Neo4j: Text to Knowledge Graph Pipeline
• KGGen: Extracting Knowledge Graphs from Plain Text (ArXiv 2025)
• GeeksforGeeks: BM25 Algorithm
• O'Reilly: Text Summarization with Seq2seq Attention and Transformer Networks
• Jay Alammar: Visualizing Neural Machine Translation
• OpenAI Cookbook: How to evaluate a summarization task
• ArXiv: Comparative Study of Quality Evaluation Methods for Text Summarization

Last updated: 2025-03-05

6. Latest Research Updates (March 2026)

6.1 MAMBA-Transformer Hybrids for Extractive Summarization

Key Paper: "Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios" (arXiv:2603.01288, March 2026)

This represents the first hybrid Transformer-State Space Model (SSM) architecture for extractive summarization:

• Architecture Components:
• Transformer encoder for sentence-level semantics
• Mamba state space model to capture inter-sentence dependencies efficiently
• Linear classifier for sentence relevance prediction

• Key Advantages:
• Leverages Mamba's linear O(L) complexity for processing full documents without truncation
• Preserves context while maintaining strong summarization quality
• 24-27% faster inference on CNN/DailyMail news summarization
• Achieves +0.23 ROUGE-1 improvement over BERTSUM and MATCHSUM on ArXiv dataset
• Strong performance with limited training data (low-resource scenarios)
• Best performance on longest documents

6.2 Hybrid Search: BM25 + Vector Retrieval

The consensus in 2025-2026 is that hybrid retrieval combining BM25 with semantic search outperforms either approach alone:

BM25 Strengths:

• Term frequency (TF) saturation prevents keyword-stuffed documents from dominating
• Document length normalization prevents long documents from outranking concise answers
• Inverse document frequency (IDF) gives rare terms more weight
• Excels at exact identifier queries (SKUs, error codes, API endpoints, legal clause numbers)

Hybrid Patterns:

1. Full-text as primary retriever → Works well for keyword-based queries with strong terminology
2. Full-text as filter for vector search → Narrow candidates with keywords, then rank by semantic similarity
3. Full-text as backstop → Run vector search, fall back to BM25 if results are poor

Implementation: RRF (Reciprocal Rank Fusion) merges BM25 and vector results. Typical weighting: alpha=0.75 for vector, alpha=0.25 for BM25.

6.3 Knowledge Graph Extraction with LLMs

Recent advances (2025-2026) have revolutionized KG extraction:

KGGen Approach (Feb 2025):

• Multi-stage LLM pipeline:
• Extract entities and relations from each source text
• Aggregate graphs across sources
• Iteratively cluster entities and relations

Production Tools:

• Neo4j LLM Knowledge Graph Builder: Transforms PDFs, documents, images, web pages into lexical graphs with embeddings
• GraphRAG: Uses knowledge graphs for retrieval-augmented generation
• Multi-agent architectures (KARMA framework): Schema-guided extraction with entity normalization

Key Trend: LLMs have reframed information extraction as a generative task, eliminating the need for training specialized NER and RE models.

6.4 RAG Evaluation Metrics: RAGAs and Faithfulness

RAGAs (RAG Assessment) has become the standard framework for RAG pipeline evaluation:

Key Metrics:

• Faithfulness: Measures how factually consistent a response is with retrieved context (0-1 scale)
• Answer Relevancy: How well the answer addresses the question
• Context Recall: Coverage of ground truth in retrieved context
• Context Precision: Relevance of retrieved chunks
• Context Utilization: How well the generator uses retrieved context

Implementation:

• Uses LLM (GPT-4 or open-source) to perform natural language judgment
• Higher accuracy and closer to human evaluation than lexical metrics

G-Eval: LLM-based evaluation framework using chain-of-thought prompting for summarization quality assessment.

6.5 MTEB and Sentence Transformer Ecosystem

Massive Text Embeddings Benchmark (MTEB) continues to drive embedding model advances:

• 15,000+ pre-trained models available on Hugging Face
• Key trends in 2025-2026:
• Multilingual models (1000+ languages)
• Domain-specific models (medicine, code)
• Multimodal models (text-image-audio)
• Task-automated alignment models
• Native low-power device execution

Popular Models:

• all-MiniLM-L6-v2: Fast, good quality, widely used
• Domain-specific models significantly improve retrieval quality for specialized content

Section 6 added: March 5, 2026

7. Advanced Embedding Architectures (March 2026)

7.1 Bi-encoders vs Cross-encoders vs Late Interaction

The 2026 retrieval landscape emphasizes multi-stage retrieval pipelines combining different encoder architectures:

Bi-encoders (Siamese Networks):

• Encode query and document independently into fixed vectors
• Fast computation ( cosine similarity only at query time)
• Trade-off: Cannot capture query-document interactions during encoding
• Examples: all-MiniLM-L6-v2, BGE-base, Qwen3-embed

Cross-encoders:

• Jointly encode query-document pairs for each candidate
• Higher accuracy but computationally expensive (no pre-computed document vectors)
• Used primarily for reranking top-K candidates from bi-encoder stage
• Examples: monoT5, Qwen-Rerank, Cross-Encoder MS MARCO

Late Interaction Models (ColBERTv2):

• Token-level embeddings with late interaction
• Query tokens attend to document tokens after separate encoding
• Better than bi-encoders, faster than cross-encoders
• Key advantage: Can be applied to pre-computed document embeddings

2026 Best Practice Pipeline:

1. BM25/sparse retrieval → candidate set
2. Bi-encoder (e.g., E5-large-v2, Contriever) → initial ranking
3. Late-interaction model (ColBERTv2) → refined ranking
4. Cross-encoder (monoT5) → final reranking
5. LLM (72B Qwen 2.5 Instruct) → listwise refinement

7.2 Late Chunking and Context-Aware Embeddings

Late Chunking (Jina AI, advanced in 2025-2026):

• Process entire document through transformer first, then chunk in latent space
• Each chunk inherits full document context before separation
• Captures both coarse (document-level) and fine-grained (chunk-level) semantic signals

Context-Aware Embeddings (Anthropic, 2025):

• Prepend document-level summary to each chunk before embedding
• Prevents meaning loss when chunks are retrieved in isolation
• Particularly effective for long documents with complex structures

Perplexity pplx-embed (February 2026):

• New family of open-source multilingual embedding models
• Built on Qwen architecture with late chunking support
• Context-aware design for improved retrieval quality

7.3 Matryoshka Representation Learning (MRL)

MRL enables flexible dimension trade-offs without retraining:

Key Concept:

• Train embeddings where earlier dimensions carry most semantic information
• Post-training, can truncate to 64/128/256/512/1024 dimensions as needed
• Storage and compute scale linearly with chosen dimension

2026 Advances:

• Amazon Nova Multimodal Embeddings: Uses MRL for storage cost reduction
• EmbeddingGemma: Google's bi-directionally trained Gemma variant with MRL
• Cross-lingual MRL (arXiv:2602.19991): Extends MRL to speech-text cross-modal retrieval
• MatGPTQ: Combines MRL with quantization for ultra-efficient deployment

Performance at 64 dimensions:

• Achieves ~90% of full-dimension retrieval quality
• 16x reduction in storage and compute costs
• Critical for large-scale production systems

7.4 Instruction-Aware Embeddings

Qwen3 Embeddings (2026):

• Available in 0.6B, 4B, 8B variants
• 32K context (small models) / 128K context (large/MoE models)
• Instruction-aware architecture: Same model handles embedding and reranking
• Apache 2.0 license for commercial use
• The 4B and 8B models outperform others for RAG and enterprise pipelines

Voyage AI:

• Consistently best performing for book data in benchmarks
• Higher latency but superior retrieval quality
• Often combined with local Qwen3 embedder (90% performance at fraction of cost)

8. Sparse + Dense Hybrid Retrieval (March 2026)

8.1 BGE-M3 and Multi-Vector Retrieval

BGE-M3 (BAAI):

• Supports dense, sparse, and multi-vector retrieval in single framework
• Sparse: Learned term weights (not just BM25)
• Multi-vector: Multiple embeddings per document for fine-grained matching
• Compact model with strong performance across tasks

Architecture Benefits:

• Single model handles multiple retrieval modalities
• Reduces infrastructure complexity
• Enables dense+sparse fusion without separate models

8.2 Learned Sparse Embeddings

Beyond traditional BM25:

• SPADE: Learns sparse term weights via transformer
• SPLADE: Expands queries with learned term expansions
• BGE-M3 sparse mode: Learned term importance vs. count-based IDF

9. Chunking Strategies (March 2026)

9.1 Token Overlap Strategies

2026 Consensus:

• Always include 50-100 token overlap between consecutive chunks
• Without overlap, answers spanning chunk boundaries are lost
• Critical for question-answering scenarios

9.2 Query-Dependent Chunking

AI21 Labs (2026):

• Chunk size should be query-dependent, not fixed
• Multi-scale approach: Retrieve chunks at multiple granularities
• Combine small chunks (precise) + large chunks (contextual)

9.3 Hierarchical/Graph-Based Chunking

RAPTOR (2025):

• Builds hierarchical summaries over fixed-size chunks
• Enables tree-based retrieval at multiple abstraction levels
• Good for very large documents

Graph-Based:

• Build similarity graphs between chunks
• Identify semantically connected clusters
• Retrieve at cluster level for better coverage

10. New Evaluation Methods Beyond ROUGE/BERTScore (March 2026)

10.1 LLM-as-a-Judge

Framework:

• Use large language models (GPT-4, Claude, Qwen) to evaluate generation quality
• Prompt engineering for rubric-based scoring
• Chain-of-thought prompting for explainable judgments

Advantages over lexical metrics:

• Captures semantic equivalence, not just n-gram overlap
• Evaluates coherence, fluency, relevance, factuality
• Closer to human evaluation than ROUGE/BERTScore

Best Practices (2026):

• Use reference-free evaluation where possible
• Include specific criteria in prompts (0-1 scale with definitions)
• Aggregate across multiple LLM judges for stability

10.2 G-Eval

G-Eval (2024-2025):

• LLM-based evaluation framework using chain-of-thought
• Specifically designed for summarization quality
• Outputs scores with reasoning for transparency

10.3 RAG Evaluation: RAGAs, ARES, TruLens

RAGAs (RAG Assessment):

• Faithfulness: Factual consistency with retrieved context
• Answer Relevancy: Addresses the question
• Context Recall: Ground truth coverage
• Context Precision: Chunk relevance

ARES (Automated RAG Evaluation System):

• Uses lightweight LLM annotators
• Factuality, relevance, and groundedness scoring
• Reduces human annotation cost

TruLens:

• Neural and LLM-based feedback functions
• RagEvaluator class for end-to-end pipeline evaluation
• Integrates with LangChain and LlamaIndex

10.4 Factual Consistency Metrics

QAFactEval:

• Question-answering based factual evaluation
• Generates questions from summary, checks answerability from source
• Strong correlation with human factual judgments

GloSA-sum (February 2026):

• Global structure-aware summarization
• Reports ROUGE-1/2/L, BERTScore, and QAFactEval
• Consistently outperforms LLM baselines across datasets

10.5 SummQ: Quiz-Based Evaluation

SummQ Framework (September 2025):

• Learn to summarize by learning to quiz
• Adversarial agentic collaboration for long document summarization
• Evaluates via ROUGE, BERTScore, LLM-as-Judge, and human evaluation

10.6 Self-BLEU and Diversity Metrics

For multi-candidate generation:

• Self-BLEU: Measures similarity between generated outputs
• Higher Self-BLEU = less diverse
• Important for summarization where multiple valid summaries exist

11. CPU-Optimized Embeddings (2026)

11.1 Cost Reduction Techniques

2026 Focus: Reducing embedding inference costs:

Key Approaches:

• Quantization (int8, binary embeddings)
• Knowledge distillation from larger models
• Optimized CPU kernels (MatGPTQ-style)
• Mobile-friendly architectures

Benchmark Results:

• Achieved nearly same processing time for <50% cost
• Critical for billion-token scale deployments

11.2 Local Embedding Options

Popular Local Models:

• Qwen3-0.6B: 90% of Voyage quality at fraction of latency/cost
• BGE-M3: Compact with multi-vector support
• all-MiniLM-L6-v2: Fast, good quality, widely adopted

12. Emerging Techniques (March 2026)

12.1 HippoRAG-v2

Memory-aware retrieval:

• Inspired by hippocampal memory systems
• Better for multi-hop question answering
• Outperforms previous memory-enhanced approaches on LoCoMo benchmark

12.2 GraphRAG Integration

Knowledge Graph + RAG:

• Extract entities and relationships from documents
• Build graph structure for retrieval
• Combine with vector search for hybrid retrieval

12.3 Multimodal Embeddings

Amazon Nova Multimodal Embeddings:

• Process video, audio, and text
• Bridge "dark data" gap
• Enable cross-modal retrieval

12.4 Generative Embeddings

Let LLMs Speak Embedding Languages (arXiv:2509.24291):

• Iterative contrastive refinement with LLMs
• Generative approach to producing text embeddings
• Addresses limitations of contrastive learning alone

References (Updated March 2026)

• Perplexity AI pplx-embed release: https://youtube.com/watch?v=y4TFjCPbs-s
• Reddit RAG 2026 discussion: https://reddit.com/r/Rag/comments/1rcba6y
• MarkTechPost: Matryoshka-Optimized Sentence Embedding Models (Feb 2026)
• arXiv:2603.01288 - MAMBA-Transformer Hybrids for Extractive Summarization
• arXiv:2602.19991 - Cross-lingual Matryoshka Representation Learning
• arXiv:2602.09548 - ReSIM: Re-ranking Binary Similarity Embeddings
• arXiv:2602.10321 - Multi-Stage Hybrid Re-Ranking
• arXiv:2602.16299 - MICE: Minimal Interaction Cross-Encoders
• Weights & Biases: LLM evaluation benchmarking beyond BLEU/ROUGE
• AI21 Labs: Query-dependent chunking approach
• Anthropic: Contextual retrieval documentation
• Zedtreeo: RAG Explained - The Complete 2026 Guide

13. Lightweight NER and Relation Extraction (2025-2026)

13.1 GLiNER - Generalist Lightweight NER

Paper: NAACL 2024, arXiv:2311.08526 | GitHub: https://github.com/urchade/GLiNER

GLiNER represents a breakthrough in lightweight named entity recognition:

• Zero-shot capabilities: Extract any entity types from text without task-specific training
• Bidirectional transformer encoder: Parallel entity extraction
• Performance: Outperforms both ChatGPT and fine-tuned LLMs in zero-shot evaluations
• Efficiency: Fast CPU inference with competitive accuracy
• Joint extraction: Supports both entity and relation extraction

2025 Update - GLiNER2 (arXiv:2507.18546):

• Multi-task information extraction system
• Schema-driven interface
• Builds on GLiNER's synthetic data fine-tuning paradigm

13.2 GLiREL - Zero-Shot Relation Extraction

GitHub: https://github.com/jackboyla/GLiREL | Paper: arXiv:2501.03172

• Generalist model for extracting any relationship types from text
• Works in conjunction with GLiNER for end-to-end information extraction
• Zero-shot capability for novel relation types

13.3 Relik - Entity Linking and Relationship Extraction

Reference: Neo4j Developer Blog (August 2025) - https://neo4j.com/blog/developer/entity-linking-relationship-extraction-relik-llamaindex/

• LLM-based approach replacing traditional specialized models (coreference resolution, NER, RE)
• Joint entity and relation extraction in single pipeline
• Integrated with LlamaIndex for RAG pipelines

13.4 Reamend - LLM Augmented Joint Learning

Paper: Information Sciences, October 2025 - https://www.sciencedirect.com/science/article/abs/pii/S1568494625014073

• Novel framework for entity-relation extraction
• Leverages LLM capabilities for simultaneous entity recognition and relation detection
• Addresses complex scientific text extraction

13.5 ZERONER - Advanced Zero-Shot NER

Paper: ACL Findings 2025 - https://aclanthology.org/2025.findings-acl.805/

• Fueling zero-shot NER with improved techniques
• Addresses limitations in previous zero-shot approaches

14. Transformer Summarization: Comparative Analysis (2025)

14.1 PEGASUS, BART, T5 Comparative Study

Paper: MDPI Future Internet, August 2025 - https://www.mdpi.com/1999-5903/17/9/389

Key Findings (2025):

• T5-Base reduces hallucination to 5%, repetition to 4%, factual errors to 6%
• T5-Base narrows performance gap with PEGASUS and BART
• BART shows superior recall in business news summarization
• PEGASUS excels at extracting key sentences (gap-sentence generation)

14.2 Fine-tuning Considerations

• All models benefit from domain-specific fine-tuning
• Controlled experimental settings with token length limits
• Lower temperature reduces hallucinations and incoherence
• Pre-trained summarization models (BART, PEGASUS) outperform generic models

15. Long-Context LLM Summarization (2025-2026)

15.1 Million-Token Context Models

Benchmark: InfiniteBench (Zhang et al., 2024) - tasks on 100K+ tokens

Leading Models (2026):

15.2 Context Rot Phenomenon

Reference: Chroma Research - https://research.trychroma.com/context-rot

• Performance degrades with increasingly long contexts
• Needle in a Haystack (NIAH) tests lexical retrieval only
• Real applications (summarization, agent tasks) require reasoning over ambiguous information
• Solution: Hierarchical summarization - replace context with summary when approaching limits

15.3 Summarization Strategies for Long Documents

1. Hierarchical approach: Summarize sections, then summarize summaries
2. Sliding window with memory: Keep running summary of processed content
3. Retrieval-based: Extract relevant passages before summarization
4. LLM-based agents: Multi-step reasoning over long documents

16. Advanced Embedding Models (2025-2026)

16.1 GritLM - Generation-Retrieval Interleaving

Reference: https://research.aimultiple.com/open-source-embedding-models/

• GritLM-7B: Highest Top-5 accuracy (90%) among 7B+ models
• Unique generation-retrieval interleaving capability
• Superior performance on retrieval tasks

16.2 E5 Embedding Family

Performance Comparison:

• E5-small: 14x faster than llama-embed-nemotron-8b (16ms vs 195ms)
• E5-Large-Instruct: 27.4% Recall@1 (monolingual), 20.7% (cross-lingual)
• Best efficiency-accuracy tradeoff in 2026 benchmarks

16.3 BGE-M3 Multilingual Leadership

Reference: https://dasroot.net/posts/2026/01/embedding-models-comparison-bge-e5-instructor/

• Leads in reverse retrieval: 32.1% Recall@1 (12 Indian languages)
• Excels in dense, multi-vector, and sparse retrieval
• Multi-granularity: from sentences to 8192-token documents

References (Additional)

• GLiNER: https://github.com/urchade/GLiNER (NAACL 2024)
• GLiREL: https://github.com/jackboyla/GLiREL (arXiv:2501.03172)
• MDPI PEGASUS/BART/T5 Study: https://www.mdpi.com/1999-5903/17/9/389
• Neo4j Relik: https://neo4j.com/blog/developer/entity-linking-relationship-extraction-relik-llamaindex/
• Chroma Context Rot: https://research.trychroma.com/context-rot
• InfiniteBench: https://arxiv.org/abs/2402.19468

17. METEOR Metric (Extended)

17.1 Overview

METEOR (Metric for Evaluation of Translation with Explicit ORdering) was developed by IBM to address limitations in BLEU, particularly for tasks requiring paraphrase awareness.

Design Philosophy:

• Addresses BLEU's limitations by incorporating multiple levels of linguistic analysis
• Considers exact matches, synonyms, stemming, and paraphrases
• Evaluates translations more holistically and contextually

17.2 How METEOR Works

Scoring Process:

1. Alignment: Align candidate and reference sentences using exact matches, stemmed matches, and synonym matches
2. Fragmentation Penalty: Penalize fragmented/misaligned translations
3. Score Calculation: Harmonic mean of precision and recall with fragmentation penalty

Key Features:

• Uses WordNet for synonym matching
• Applies stemming to match word roots
• Ranges from 0 to 1 (higher is better)

17.3 Comparison with Other Metrics

17.4 Applications and Limitations

Use Cases:

• Machine translation evaluation
• Text summarization (when paraphrase quality matters)
• Paraphrasing system evaluation
• Image captioning evaluation

Limitations:

• Depends on WordNet quality for synonym matching
• Slower to compute than BLEU/ROUGE
• Less widely adopted than ROUGE in summarization community

18. Topic Modeling: LDA and NMF

18.1 Latent Dirichlet Allocation (LDA)

LDA is a generative probabilistic model that discovers abstract topics in a collection of documents.

How LDA Works:

1. Each document is a mixture of topics
2. Each topic is a distribution over words
3. The model infers these distributions from the text

Mathematical Framework:

• Dirichlet prior: Assumes topics are distributed according to a Dirichlet distribution
• Gibbs sampling: Commonly used for parameter estimation
• Outputs: Document-topic distributions and topic-word distributions

Practical Applications in Extraction:

• Document clustering and organization
• Keyword extraction from topic distributions
• Content-based recommendation
• Trend detection in document collections

Strengths:

• Interpretable topics
• Scalable to large corpora
• Well-established with extensive tooling

Limitations:

• Assumes topics are independent (bag-of-words assumption)
• Requires predefined number of topics
• May produce incoherent topics on short texts

18.2 Non-Negative Matrix Factorization (NMF)

NMF is a non-probabilistic matrix decomposition technique for topic modeling.

How NMF Works:

1. Decompose document-term matrix into two non-negative matrices: W (document-topic) and H (topic-term)
2. Optimize reconstruction error: ||V - WH||²

Key Properties:

• Parts-based representation (more interpretable)
• No probabilistic assumptions
• Deterministic output

Comparison with LDA (from Nature Scientific Reports, 2024):

18.3 Modern Topic Modeling (2024-2025)

BERTopic:

• Uses BERT embeddings for document representation
• HDBSCAN for clustering
• c-TF-IDF for topic representation
• Superior for short texts (Twitter, reviews)

Top2Vec:

• Joint embedding of documents and topic vectors
• Distributed topic representations
• Automatic topic number detection

Selection Guide (2025):

• LDA: Traditional corpus, interpretability priority, when topic coherence is measurable
• NMF: Short texts, need parts-based interpretation, faster training
• BERTopic: Semantic similarity matters, short texts, modern pipelines
• Top2Vec: When topic vectors need semantic operations

19. Pyramid Method for Summarization Evaluation

19.1 Overview

The Pyramid method (Nenkova et al., 2007) addresses limitations of purely automatic metrics by incorporating human-created content units.

Core Idea:

• Human annotators create "content units" (CUs) - factual statements extracted from source documents
• Evaluate summary coverage by counting how many CUs are present
• Weights CUs by importance (multiple annotators create CUs; more overlap = higher weight)

19.2 Implementation

Step-by-Step Process:

1. CU Extraction: Multiple annotators read source and extract factual statements
2. Weighting: CUs appearing in multiple annotations get higher weights
3. Scoring: Count weighted CUs present in candidate summary
4. Normalization: Divide by total possible CUs

Pyramid Score:

Pyramid Score = (Sum of weights of covered CUs) / (Sum of weights of all CUs in reference)

19.3 Advantages and Limitations

Advantages:

• More human-aligned than ROUGE/BLEU
• Captures content coverage comprehensively
• Accounts for information importance
• Better correlation with human quality judgments

Limitations:

• Requires significant human effort to create CUs
• Not fully automatic
• Time-consuming for large-scale evaluation
• Subjectivity in CU extraction

19.4 Modern Usage

Current Status (2025-2026):

• Considered gold standard for summarization evaluation when available
• Used in research benchmarks (e.g., CNN/DailyMail, XSum)
• Hybrid approaches: Use LLM-generated pseudo-CUs to approximate Pyramid evaluation
• RAGAs framework draws parallels with Pyramid's content coverage focus

Practical Application:

• Use for critical evaluation tasks (research papers, system comparisons)
• Use automatic metrics (ROUGE, BERTScore) for rapid iteration
• Combine with LLM-based evaluation (G-Eval) for balanced assessment

20. Complete Evaluation Method Comparison

20.1 Summary Table

20.2 2026 Best Practice

Recommended Evaluation Pipeline:

1. Automatic metrics (fast iteration): ROUGE + BERTScore
2. Semantic metrics (paraphrase-aware): METEOR or BERTScore
3. RAG-specific (if applicable): RAGAs framework
4. Gold standard (critical tasks): Pyramid or human evaluation
5. LLM-based (modern approach): G-Eval for comprehensive quality

Sections 17-20 added: March 5, 2026

21. Pointer-Generator Networks and Coverage Mechanism

21.1 Overview

Pointer-Generator Networks (PGN), introduced by See et al. (2017) in "Get To The Point: Summarization with Pointer-Generator Networks", combine the benefits of extractive and abstractive summarization.

Key Paper: arXiv:1704.04368

21.2 Architecture

The PGN has three main components:

1. Sequence-to-Sequence Attention Model:
• Encoder (typically bi-LSTM) processes source text
• Decoder (LSTM) generates tokens one at a time
• Attention mechanism over encoder hidden states

1. Pointer Mechanism:
• Allows copying words from source text via pointing
• Addresses rare/out-of-vocabulary (OOV) words
• Switches between generating (abstractive) and pointing (extractive)

1. Coverage Mechanism:
• Tracks what has been attended to prevent repetition
• Coverage vector accumulates attention weights over time
• Penalty term in loss function discourages attending to covered tokens

21.3 How It Works

Generation vs. Pointing:

• At each decoder step, compute probability $p_{gen}$ (generate) vs $p_{pointer}$ (point)
• Final vocabulary distribution = $p_{gen} × P_{vocab} + (1 - p_{gen}) × A_{attended}$
• Enables handling of unseen words, proper nouns, numbers

Coverage Loss:

coverage_loss = Σ min(attention_i, coverage_i)

• Penalizes repeatedly attending to same positions
• Helps reduce repetition in generated summaries

21.4 Modern Usage (2024-2025)

Transformer + PGN Hybrids (2025):

• Research on Automatic Text Summarization Using Transformer and Pointer-Generator Networks (Yu et al., March 2025)
• Replaces LSTM encoder/decoder with Transformer
• Combines self-attention with pointer mechanism
• Enhanced coverage for long documents

Strengths:

• Handles OOV words effectively
• Reduces hallucination by grounding in source
• Combines extractive precision with abstractive flexibility
• Strong baseline for abstractive summarization

Limitations:

• Still prone to repetition without careful coverage
• Single document summarization; struggles with multi-document
• Can be computationally expensive

22. SummaC: Summary Consistency Detection

22.1 Overview

SummaC (Summary Consistency) is a metric for evaluating factual consistency between summaries and source documents. Published in TACL 2022, it addresses the critical issue of hallucination in summarization.

Key Paper: arXiv:2111.09525 | ACL Anthology: 2022.tacl-1.10

22.2 Methodology

SummaC uses Natural Language Inference (NLI) to detect inconsistencies:

1. Sentence-Level Consistency:
• Split summary into sentences
• Check each sentence against source document
• Label as: ENTAILED, NEUTRAL, or CONTRADICTED

1. NLI Models Used:
• Originally: Sentence-Transformers fine-tuned on NLI datasets
• SummaC-Conv: Uses conversation-based NLI model
• Works without requiring training on summarization-specific data

1. Aggregation:
• Compute consistency score per summary sentence
• Average or max-pool across all sentences
• Final score: percentage of consistent (non-contradictory) sentences

22.3 SummaC Benchmark

Six datasets in the benchmark:

• XSum Faithfulness (BBC articles → one-sentence summaries)
• CNN/DailyMail (news articles → highlights)
• SummEval (professional summaries)
• FRANK (factuality annotations)
• QAGS (question answering-based)
• TruthfulQA (for detecting hallucinations)

Evaluation: Binary classification (consistent vs. inconsistent)

22.4 Performance

SummaCConv Results:

• Balanced accuracy: 74.4% on benchmark
• 5% improvement over previous NLI-based methods
• Strong correlation with human factuality judgments

22.5 Comparison with Other Factual Consistency Metrics

22.6 Practical Implementation

Using SummaC:

from summac.model_summac import SummaCConv

# Load model
model = SummaCConv(verbs="exec", max_chunk=512)

# Score summary against source
score = model.score(source_doc, generated_summary)
# Returns consistency score (0-1)

Best Practices (2025-2026):

• Use alongside ROUGE/BERTScore for comprehensive evaluation
• Particularly important for abstractive summarization
• Combine SummaC with other factual metrics (QAFactEval) for robustness

23. Latest Research Updates (March 2026 - Week 1)

23.1 Embeddings - Newest Models and MTEB Updates

Perplexity pplx-embed (Released March 2026)

• Open-source multilingual embedding models in two sizes: 0.6B and 4B parameters
• Flagship 4B model outperforms Anthropic and Voyage equivalents on MTEB and ConTEB benchmarks
• Built on Qwen architecture with late chunking support
• Rivaling Google and Alibaba at a fraction of the memory cost

Qwen3-Embedding-8B (March 2026)

• Tops the MTEB multilingual leaderboard at 70.58
• Outperforms all open-source alternatives
• 32K context (small models) / 128K context (large/MoE models)
• Apache 2.0 license for commercial use

PTEB: Stochastic Paraphrasing for Embedding Evaluation (arXiv:2510.06730, February 2026)

• New approach using LLMs for stochastic evaluation-time benchmarks
• Shifts focus from static datasets to dynamic evaluation
• Addresses limitations of static MTEB benchmarks

Legal RAG Bench (March 2026)

• New benchmark for legal retrieval
• Identifies data leakage issues in commercial embedding models
• Reveals poor label quality in public legal evaluation sets

23.2 Semantic Representation - Knowledge Graphs and RAG Advances

GraphRAG (March 2026)

• Combines knowledge graphs with RAG for improved reasoning over complex relationships
• Enables AI systems to reason over complex relationships
• Provides more accurate answers to complex queries
• Offers transparency in information retrieval and usage

Enterprise RAG Adoption (Gartner Survey, March 2026)

• 67% of Fortune 500 companies have deployed or are actively building RAG systems
• Most widely adopted enterprise AI architecture
• Hybrid RAG combining extractive and generative processes gaining traction
• Active RAG continuously updates knowledge base and learns from new interactions

Knowledge Graph Conference 2026 (March 2026)

• Cornell Tech NYC & Online event
• Focus on evolution of semantic technologies
• ISWC 2026 workshops announced for semantic materials science

23.3 Traditional NLP - Classical Approaches

BM25 Renaissance (March 2026)

• BM25 seeing renewed interest as complementary to vector search
• Full-text search for RAG: BM25 as "precision layer"
• Three patterns for hybrid retrieval:
• Full-text as primary retriever
• Full-text as filter for vector search
• Full-text as backstop
• RRF (Reciprocal Rank Fusion) merges BM25 and vector results

spaCy and Classical NLP Libraries

• Continuing use for tokenization, POS tagging, dependency parsing
• Python libraries for sentiment analysis: Pattern, NLTK, TextBlob, spaCy
• Hybrid approaches: BM25 + semantic similarity for keyword extraction

23.4 Deep Learning - Transformer Variants, SSMs, Summarization

MAMBA-Transformer Hybrids (arXiv:2603.01288, March 2026)

• First hybrid Transformer-State Space Model (SSM) architecture for extractive summarization
• Components: Transformer encoder + Mamba state space model + Linear classifier
• Achieves +0.23 ROUGE-1 improvement over BERTSUM and MATCHSUM on ArXiv dataset
• 24-27% faster inference on CNN/DailyMail

Mamba for NLP (March 2026)

• Selective State Space Models offering O(L) complexity vs O(N²) for transformers
• 500x reduction in per-token operations for sequence processing
• Recurrent inference with parallelizable training
• Challenges: redundant directional modeling, low parameter efficiency in direct applications

Qwen3.5 and DeepSeek V4 (March 2026)

• Hybrid mixture-of-experts (MoE) structure
• Strong performance with lower deployment cost
• Cost efficiency as important as raw benchmark performance

23.5 Evaluation Methods - Latest Metrics, LLM-as-Judge

LLM-as-Judge Evolution (March 2026)

• Two approaches: prompt-based general-purpose LLMs and task-specific fine-tuned models
• When criteria clearly defined, LLMs deliver reliable assessments at scale
• Subjective evaluation: primarily human-led
• Objective evaluation: more often automated using LLMs

RAG Evaluation Best Practices (March 2026)

• Multi-layer evaluation: retrieval metrics (NDCG, MRR) → generation metrics (BERTScore, ROUGE, RAGAs)
• LLM-as-judge pass with calibrated rubric
• Human review for flagged items and random sampling
• Three reporting levels: executive, team, developer dashboards

RAGAs Framework Updates (2026)

• Automates scoring across: faithfulness, answer relevance, context precision/recall
• Evaluates RAG systems without requiring ground-truth labels
• LLM-as-judge to next level with automated scoring

RAGChecker (March 2026)

• Practitioner's guide published for comprehensive RAG evaluation
• Integrates retrieval and generation metrics

SEAL LLM Leaderboards (Scale AI, March 2026)

• Expert-driven LLM benchmarks
• Updated AI model leaderboards ranking top models across coding, reasoning

G-Eval and SummQ (2025-2026)

• G-Eval: LLM-based evaluation with chain-of-thought for summarization
• SummQ: Quiz-based evaluation framework (September 2025)

Section 23 added: March 6, 2026

24. Classical Word Embeddings (Foundation)

24.1 Word2Vec

Overview: Introduced by Mikolov et al. (2013) at Google, Word2Vec revolutionized NLP by learning dense vector representations for words from large unlabeled corpora.

Architecture Variants:

• CBOW (Continuous Bag of Words): Predicts a target word from surrounding context words
• Skip-gram: Predicts surrounding context words from a target word

Key Characteristics:

• Produces static embeddings (same vector regardless of context)
• Captures semantic relationships via vector arithmetic (e.g., "king" - "man" + "woman" ≈ "queen")
• Efficient training using negative sampling and hierarchical softmax

Limitations:

• Cannot handle out-of-vocabulary (OOV) words
• Ignores word order in context
• Single representation per word (polysemy problem)

2024-2025 Usage: Still used as baselines and for efficient token-level features, but largely superseded by contextual embeddings for most tasks.

24.2 GloVe (Global Vectors)

Paper: Pennington et al. (2014) - "GloVe: Global Vectors for Word Representation"

Approach: Combines global matrix factorization with local context window methods:

• Builds word co-occurrence matrix from corpus
• Learns embeddings that encode co-occurrence statistics
• Optimization objective: weighted least squares on log co-occurrence counts

Advantages over Word2Vec:

• Incorporates global corpus statistics (not just local context)
• Often better at analogy tasks
• More stable training

Limitations:

• Still produces static embeddings
• Memory-intensive for large vocabularies
• Requires careful handling of rare words

2024-2025 Usage: Used in embeddings ensembles and when interpretable co-occurrence features are needed.

24.3 FastText

Paper: Bojanowski et al. (2017) - "Enriching Word Vectors with Subword Information"

Key Innovation: Uses subword (character n-gram) representations:

• Each word represented as bag of character n-grams
• Enables handling of OOV words by composing from subwords
• Particularly effective for morphologically rich languages

Advantages:

• Handles OOV words gracefully
• Captures morphological patterns
• Better for languages with rich inflection (Arabic, Finnish, Turkish)

Limitations:

• Larger embedding matrices due to subword vocabulary
• Still static embeddings
• Slower inference than Word2Vec/GloVe

2024-2025 Usage:

• Foundation for modern subword tokenization (BPE, SentencePiece)
• Used in production systems requiring OOV robustness
• facebook/fastText library still maintained

24.4 Comparison Summary

24.5 Evolution to Contextual Embeddings

The key limitation of all classical embeddings: static - same word always has same vector regardless of context.

Modern Solution - Contextual Embeddings:

• ELMo (2018): BiLSTM-based contextual representations
• BERT (2018): Transformer-based, deep bidirectional context
• GPT family: Unidirectional, autoregressive context
• RoBERTa, ALBERT, DeBERTa: Optimized BERT variants

Key Advantage: Same word gets different embedding based on context ("bank" = river vs. financial institution)

25. Advanced Traditional NLP Methods

25.1 Part-of-Speech (POS) Tagging

Definition: Assigns grammatical tags to each word (noun, verb, adjective, etc.)

Methods:

1. Rule-based: Dictionary lookup + contextual rules
2. Statistical: HMM (Hidden Markov Model), CRF (Conditional Random Fields)
3. Neural: BiLSTM-CRF, Transformer-based taggers

2024-2025 State:

• spaCy (en_core_web_lg): transformer-based POS tagging with 99%+ accuracy
• StanfordNLP: Multi-language support
• Transformers: BERT-based models achieve near-human performance

Applications in Text Extraction:

• Identifying key content words
• Phrase chunking
• Sentence boundary detection

25.2 Dependency Parsing

Definition: Analyzes grammatical structure by establishing relationships between words (subject, object, modifier, etc.)

Popular Parsers:

• spaCy: Fast, neural network-based, 90%+ accuracy
• Stanford Parser: Rule-based + neural, multi-language
• Stanza: Universal Dependencies framework

2024-2025 Advances:

• UDPipe: Trainable pipeline for tokenization, tagging, lemmatization, dependency parsing
• BiAffine parser: State-of-the-art attention mechanism for dependency parsing
• Pretrained language models: BERT-enhanced parsing achieves new benchmarks

Applications:

• Relationship extraction between entities
• Question answering
• Information retrieval

25.3 Coreference Resolution

Definition: Determining which mentions in text refer to the same entity

Challenges:

• Pronouns ("he", "she", "it")
• Named entities ("John" → "the CEO")
• Zero anaphora (implicit subjects in some languages)

2024-2025 Methods:

Neural Approaches:

• Span-based models: Treat spans as potential mentions
• Entity-level models: Use entity representations for clustering
• End-to-end neural coreference (Lee et al., 2017+): Dominant approach

LLM-Enhanced (2025):

• Hybrid neuro-symbolic pipeline: Combines transformers with symbolic reasoning
• Ontonotes-based models: Fine-tuned on academic datasets
• spanBERT: Pre-trained for span representation

Tools:

• spaCy: en_core_web_lg includes coreference
• NeuralCoref: Hugging Face wrapper for coreference
• Stanford CoreNLP: Full coreference pipeline

Applications in Extraction:

• Event extraction across sentences
• Document-level entity tracking
• Cross-document entity resolution

25.4 Practical Pipeline Integration

Modern Extractive Pipeline (2025):

Raw Text → Sentence Split → Tokenize → POS Tag → Dependency Parse → NER → Coreference → Relation Extraction

Python Libraries:

• spaCy: All-in-one, production-ready
• Stanza: Research-quality, 70+ languages
• Hugging Face Transformers: Fine-tuned NER/RE models

26. Semantic Similarity Methods

26.1 Traditional Approaches

Word Overlap Metrics:

• Jaccard Similarity: |A ∩ B| / |A ∪ B|
• Dice Coefficient: 2|A ∩ B| / |A| + |B|
• Cosine Similarity (TF-IDF): Classic baseline

Limitations:

• Ignore word order
• Can't handle paraphrases
• No semantic understanding

26.2 Embedding-Based Similarity

Sentence Embeddings (see Section 7):

• Generate embeddings using sentence transformers
• Compute cosine similarity between vectors

2025-2026 Best Practices:

• Use task-specific fine-tuned models
• Combine multiple embeddings (ensemble)
• Consider late interaction for precision

26.3 Semantic Similarity Benchmarks

STS (Semantic Textual Similarity):

• STS Benchmark (2017): Standard train/dev/test splits
• STS3k (2025): New dataset with systematic word combinations
• Multiingual STS: Cross-language similarity

Benchmarks:

• MTEB: Includes STS tasks
• SBERT.net: Easy evaluation on STS datasets

26.4 Cross-Encoder vs Bi-Encoder Similarity

Bi-Encoder (for ranking):

• Pre-compute document embeddings
• Fast cosine similarity
• Used in retrieval pipelines

Cross-Encoder (for reranking):

• Joint encoding of sentence pair
• Higher accuracy, slower
• Used after bi-encoder retrieval

27. Fine-Tuning Strategies for Summarization

27.1 Full Fine-Tuning

Process:

1. Initialize with pretrained weights
2. Update all parameters on domain data
3. Risk: Catastrophic forgetting

Best Practices:

• Lower learning rate than pretraining (1e-5 to 5e-5)
• Warmup steps for stability
• Gradient clipping (max_norm=1.0)

27.2 Parameter-Efficient Fine-Tuning (PEFT)

2024-2025 Dominant Approaches:

LoRA (Low-Rank Adaptation):

• Inject trainable rank decomposition matrices
• Keep pretrained weights frozen
• 2025: LoRA+, GaLore, DoRA improvements

Prefix Tuning:

• Prepend trainable continuous vectors
• Lightweight, effective for few-shot

Adapter Modules:

• Small bottleneck layers inserted in transformer
• Popular: AdapterFusion, AdapterHub

Quantization + PEFT:

• QLoRA: Quantized LLMs + LoRA
• Enables fine-tuning 70B+ models on single GPU

27.3 Domain Adaptation

Continued Pretraining:

1. Continue pretraining on domain corpus (domain-adaptive pretraining)
2. Then fine-tune on summarization task

Data Selection:

• Curate domain-specific training data
• Use quality filters for noisy datasets

27.4 Multi-Task Fine-Tuning

Approach: Train on multiple related tasks simultaneously

• Summarization + QA + classification
• Shared representations improve generalization

2025 Trend:

• Instruction-tuned models for summarization
• GRPO (Group Relative Policy Optimization) for alignment

28. Note on BLOND Metric

The original task mentioned "BLOND" metric. Based on comprehensive research:

BLOND Status:

• No widely-adopted "BLOND" metric was found in academic literature
• May be confused with:
• BLEU: Standard machine translation metric
• BLONDE: BLEU for automatic video captioning (2016)
• BERT-based metrics (BERTScore, etc.)

Recommendations:

• Use BERTScore for semantic evaluation
• Use BLEU for translation tasks
• Use ROUGE for summarization
• Use RAGAs for RAG evaluation
• Use LLM-as-Judge for comprehensive quality

If "BLOND" refers to a specific proprietary metric, please provide additional context.

References (Sections 24-28)

• Mikolov et al. (2013): Word2Vec - "Efficient Estimation of Word Representations in Vector Space"
• Pennington et al. (2014): GloVe - "GloVe: Global Vectors for Word Representation"
• Bojanowski et al. (2017): FastText - "Enriching Word Vectors with Subword Information"
• spaCy Documentation: https://spacy.io/
• Stanza: Stanford NLP Group - https://stanfordnlp.github.io/stanza/
• ACL 2025: Coreference Resolution advances
• LoRA Paper: "LoRA: Low-Rank Adaptation of Large Language Models" (Hu et al., 2021)
• QLoRA: "QLoRA: Efficient Finetuning of Quantized LLMs" (Dettmers et al., 2024)

Sections 24-28 added: March 6, 2026

Summary: Complete Coverage

This document now covers all five research angles:

23. LLM-Based Extraction and Summarization: Advances 2025-2026

This section covers the latest advances in text extraction and summarization using Large Language Models, focusing on developments from 2025-2026.

23.1 LLM-Based Extraction Methods

23.1.1 Generative Information Extraction with LLMs

Recent surveys (Xu et al., 2025) have comprehensively reviewed LLM-based generative IE methods, which convert plain text into structured knowledge through auto-regressive generation rather than traditional discriminative approaches.

Key Paradigms:

• Supervised Fine-tuning: Further training LLMs on IE tasks using labeled data
• Few-shot Learning: Generalization from a small number of labeled examples via in-context learning
• Zero-shot Generation: Generating structured output without any training examples for specific IE tasks
• Data Augmentation: Enhancing information extraction by applying transformations using LLMs

Typical IE Tasks Addressed:

• Named Entity Recognition (NER): Entity identification + Entity typing
• Relation Extraction: Classification, triplet extraction, and strict relation extraction
• Event Extraction: Event detection and event argument extraction

23.1.2 In-Context Learning for Extraction

In-context learning (ICL) has emerged as a powerful paradigm for rapid domain adaptation in extraction tasks.

Key Developments (2025):

• SAIL (Sample-Centric In-Context Learning): A fine-tuning-free method for Document Information Extraction that leverages textual and layout similarity to guide LLMs, achieving strong performance without full training.

• Domain Adaptation via ICL: By providing instructions and a few examples to guide extraction, methods avoid the need for extensive, domain-specific datasets required by traditional supervised techniques.

• Prompt Engineering for IE: Studies show that automatic prompt search with fine-tuning methods improves few-shot learning performance for classification and regression tasks in document information extraction.

Best Practices for ICL in Extraction:

• Concise examples that clearly demonstrate the essential pattern work better than verbose ones
• Include enough detail to make the task unambiguous
• For data extraction, show relevant fields clearly rather than including extensive surrounding text

23.1.3 Universal IE Frameworks

Two main approaches have emerged for universal information extraction:

1. NL-LLMs (Natural Language LLM-based): Uses natural language prompts to define extraction schemas
2. Code-LLMs: Represents extraction tasks as code (e.g., Python classes with docstrings), leveraging LLMs' code generation capabilities

Key Frameworks:

• InstructUIE: Universal IE framework using instruction tuning
• Code4UIE: Code-based approach for unified IE tasks

23.2 Long-Context LLMs for Document Summarization

23.2.1 Context Window Advances

State-of-the-art models now support context windows of 16K to 1M+ tokens, dramatically improving their ability to summarize lengthy documents while maintaining coherence.

Key Capabilities:

• Capturing nuanced relationships across extended spans of text
• Maintaining coherence in summaries of lengthy documents
• Leveraging dependencies across extended spans for reasoning, code generation, and document summarization

23.2.2 Summarization Strategies

1. Extractive vs. Abstractive Approaches:

• Extractive: Identifies and extracts key sentences directly from source text; preserves original wording
• Abstractive: Generates entirely new text capturing essential meaning; excels at readability
• Hybrid: Combines both techniques for balanced factual accuracy and readability

2. Map-Reduce Approaches:
For documents exceeding token limits, MapReduce remains the dominant approach:

• Map step: Chunk document into pieces fitting within token limits, generate summary for each chunk
• Reduce step: Generate final summary from chunk summaries

3. Chain-of-Thought (CoT) Summarization:

• Breaks summarization into steps: identify main topics → extract key points → synthesize into coherent summary
• Improves logical flow and factual retention for complex/technical content

4. Hierarchical/Structured Approaches:

• CoTHSSum (2025): Chain-of-thought reasoning with hierarchical segmentation for structured long-document summarization
• Uses hierarchical encoder-decoders to better capture long-range context in scientific papers

23.2.3 Efficient Context Management

Recent advances in context management for summarization include:

• LLM Summarization: Another AI model generates short summaries
• Observation Masking: Older, less important information is hidden
• Both approaches preserve important context through fundamentally different mechanisms

23.3 Evaluation Metrics and Benchmarks (2025-2026)

23.3.1 Traditional Metrics Still in Use

• ROUGE: Measures overlap of n-grams between generated and reference summaries
• BLEU: N-gram precision-based metric
• F1 Scores: For extraction-oriented tasks

23.3.2 New Evaluation Frameworks

Systematic Reviews (2025):

• A comprehensive review of 113 peer-reviewed studies on long document summarization methods and evaluation metrics
• Focus on LDS (Long Document Summarization) methods and metrics capturing quality, coherence, and factual accuracy
• Rising adoption of hybrid models combining extractive and abstractive strategies

Benchmark Datasets:

• CNN/DailyMail: Standard benchmark for extractive and abstractive summarization
• XSum: Extreme single-sentence summaries of BBC articles
• Samsum: Conversational dialogues
• NewsSumm (2025): World's largest human-annotated multi-document news summarization dataset for Indian English
• QMSum: Query-driven summarization benchmark

Biomedical Literature Summarization:

• Benchmarking 62 text summarization methods (2026): ranging from frequency-based and TextRank extractors to modern encoder-decoder models

23.3.3 LLM-as-Judge Evaluation

• Using LLMs to evaluate summarization quality through pairwise comparisons
• Assessing coherence, relevance, and factual consistency
• Alignment with human judgments

23.3.4 Domain-Specific Benchmarks

Key Developments:

• Legal RAG Bench (2026): Domain-specific retrieval benchmark for legal applications
• Medical/Healthcare: Specialized benchmarks for clinical note summarization
• Scientific Literature: Methods specifically designed for academic paper summarization

23.3.5 RAG-Specific Evaluation

For retrieval-augmented extraction/summarization:

• Faithfulness: Whether generated answers remain faithful to retrieved context
• Answer Relevancy: Appropriateness to question
• Context Recall: Coverage of ground truth
• Context Precision: Relevance of retrieved chunks
• RAGAs Framework: Automated evaluation using LLM-as-judge

23.3.6 Emerging Evaluation Platforms (2026)

23.4 Key Research Papers and Resources (2025-2026)

Essential Papers:

1. "Large Language Models for Generative Information Extraction: A Survey" (Xu et al., arXiv 2025) - Comprehensive survey of LLM-based IE
2. "CoTHSSum: Structured long-document summarization via chain-of-thought reasoning and hierarchical segmentation" (2025)
3. "A systematic review of long document summarization methods: Evaluation metrics and approaches" (ScienceDirect, Aug 2025)
4. "Evaluation of Prompt Engineering on the Performance of a Large Language Model in Document Information Extraction" (MDPI, 2025)

Curated Resources:

• Awesome-LLM4IE-Papers: https://github.com/quqxui/Awesome-LLM4IE-Papers - Regularly updated repository of LLM for IE papers

23.5 Practical Recommendations

For Extraction Tasks:

1. Start with few-shot in-context learning for rapid prototyping
2. Use Code-LLMs for structured schema extraction
3. Fine-tune with LoRA (rank 8-16) for domain-specific extraction

For Summarization:

1. Choose extractive for legal/medical contexts requiring exact phrasing
2. Choose abstractive for readability-focused applications
3. Use MapReduce for documents exceeding context window
4. Apply Chain-of-Thought prompting for complex technical content

For Evaluation:

1. Combine traditional metrics (ROUGE, BLEU) with LLM-as-judge
2. Use domain-specific benchmarks when available
3. Implement faithfulness checks for RAG-based systems

Section 23 added: March 2026 - LLM-based extraction and summarization advances

Document complete - All 5 research angles covered with 2024-2026 methods and trends

Latest Research Findings (March 6, 2026 - Morning Update)

LLM-Based Extraction Methods (2026)

• Google's langextract: New Python library for LLM-powered structured text extraction with high precision in source localization and interactive refinement capabilities.
• Token Alignment for Verification: New research (bioRxiv, Feb 2026) shows text-alignment methods form a partially ordered set, with ordered alignment being most practical for verifying LLM-extracted text - implemented in the taln library.
• HTML-to-Text Extraction: New framework (arXiv:2602.19548, Feb 2026) rethinks extraction for LLM pretraining, showing that combining multiple extractors outperforms single approaches.
• Nemotron-CC-Math: State-of-the-art method using lynx (text-based browser) for robust math/code handling, with LLM-based cleaning of extractions.
• Clinical Research Extraction: Springer published comprehensive guidelines (March 2026) on operationalizing LLMs for clinical data extraction with quality control and governance frameworks.

Hybrid Extraction + RAG Approaches

• A-RAG (Agentic RAG): New framework (arXiv:2602.03442, Feb 2026) scaling RAG via hierarchical retrieval interfaces, moving beyond single-shot passage retrieval.
• Hybrid Search as Default (2026): All major vector databases (Pinecone, Qdrant, Weaviate) now support native hybrid search - typically returns 20-50 candidate documents.
• MA-RAG: Multi-agent framework with specialized agents (Planner, Step Definer, Extractor, QA) collaborating through chain-of-thought reasoning.
• RAGentA: Multi-agent system with hybrid sparse-dense retrieval, iterative document filtering, and citation-attributed answer generation.
• Context Rot Phenomenon: Research shows LLM performance degrades with long contexts - hierarchical summarization emerging as solution to replace context with summaries when approaching limits.

Evaluation Benchmarks for Summarization

• DeepEval: Emerging evaluation platform with 30+ metrics for RAG/agentic/conversational evaluation, including summarization-specific tests.
• Custom LLM Evaluation Pipelines: New frameworks (March 2026) combining BLEU, ROUGE, BERTScore, and LLM-as-judge for comprehensive summarization evaluation.
• G-Eval: Continues to be prominent for summarization quality assessment using chain-of-thought prompting.
• 2026 LLM Evaluation Trend: Stanford researchers predicted 2026 as "year of AI evaluation" - multilingual safety and guardrails becoming critical focus areas.
• Enterprise LLM Benchmarks: Integration of task-based checks (summarization, translation, sentiment) with intrinsic metrics (BLEU, ROUGE, F1) in unified evaluation harnesses.

Fine-Tuning for Extraction (2026 Best Practices)

• Structured Data Extraction: Typically needs 200-500 examples for schema learning, formatting variation handling, and context understanding.
• QLoRA for Extraction: Combining quantization with LoRA enables fine-tuning of 70B+ models on single GPU for domain-specific extraction tasks.
• Code-LLMs: Emerging approach representing extraction schemas as Python classes with docstrings, leveraging code generation capabilities.

Morning update added: March 6, 2026

23. Fresh Research Findings (March 6, 2026 - Final Update)

23.1 MAMBA-Transformer Hybrids for Low-Resource Extractive Summarization

Source: arxiv.org/abs/2603.01288 (March 1, 2026)

• Key Innovation: First hybrid Transformer-State Space Model (SSM) architecture for extractive summarization
• Components:
• Transformer encoder for sentence-level semantics
• Mamba state space model for efficient inter-sentence dependencies
• Linear classifier for sentence relevance prediction
• Benefits:
• Significant ROUGE improvements in low-resource scenarios
• Addresses quadratic complexity bottleneck in traditional transformers
• Processes full documents without truncation
• 24-27% faster inference on CNN/DailyMail
• Application: Resource-constrained environments, long document processing

23.2 Wayin AI - Technical Video to Searchable Notes

Source: ucstrategies.com (March 5, 2026)

• Key Innovation: Advanced OCR to decode on-screen data from technical videos
• Capabilities:
• Processes Python scripts directly from video streams
• Handles complex calculus equations
• Character identification from video stream
• Application: Technical video content extraction, educational content indexing, technical documentation

23.3 Semantic Search with LLM Embeddings

Source: machinelearningmastery.com (March 2, 2026)

• Coverage: Building semantic search systems using LLM embeddings
• Key Models: Hugging Face sentence transformers (e.g., "all-MiniLM-L6-v2")
• Methodology: Translating text into numerical vectors capturing semantic characteristics
• Best Practices: Hybrid approaches combining keyword and semantic search

23.4 Vector Databases 2026 - Complete Guide

Source: calmops.com (March 3, 2026)

• Comprehensive coverage: Vector database landscape for 2026
• Key Capability: Semantic matching (e.g., "automobile" → "cars" without exact keyword)
• Trends: Hybrid indexes combining vector embeddings and keyword search
• Leading Platforms: Pinecone, Chroma, Weaviate, Milvus, Qdrant

23.5 Snowflake Intelligence - Hybrid Search Integration

Source: flexera.com (March 3, 2026)

• Key Feature: Hybrid index combining vector embeddings and keyword search
• Capabilities: Automatic semantic reranking within Snowflake
• Integration: Cortex AI Functions for LLM calls directly in SQL
• Embedding Function: snowflake-arctic-embed-m-v2 for document embeddings

23.6 Cognitive Architectures for AI Agents - Memory Systems

Source: tredence.com (March 4, 2026)

• Focus: Long-term storage in Databricks
• Key Concepts:
• Agentic AI memory in 2026
• Procedural, semantic, and episodic memory based on human cognition
• Tools: Databricks Lakebase, Mosaic AI Vector Search
• Application: Building agents with human-like memory architectures

23.7 OpenSearch Semantic Search with Amazon Bedrock Titan

Source: docs.opensearch.org (March 3, 2026)

• Integration: Amazon Bedrock Titan embedding model
• Configuration: 1536-dimensional vectors, cosine similarity
• Pipeline: Text embedding processor for automatic vectorization
• Use Case: Enterprise semantic search applications

23.8 LLM Evaluation Beyond BLEU and ROUGE

Source: wandb.ai (March 2, 2026)

• Key Trend: Moving beyond traditional n-gram metrics for LLM evaluation
• New Approaches:
• BERTScore for semantic similarity
• LLM-as-a-judge for quality assessment
• Chain-of-thought evaluation (G-Eval)
• Benefits: Higher correlation with human evaluation

23.9 Production RAG System Best Practices

Source: gauraw.com (March 2, 2026)

• Orchestration: LangChain vs LlamaIndex comparison
• Vector Databases: Chroma (dev), Pinecone (prod), Weaviate (hybrid)
• Domain-Specific: Match embedding model to domain (legal, medical, multilingual)
• Recommendation: Specialized embeddings significantly outperform general models

23.10 Interpretable Text Embeddings Framework

Source: ICLR 2025 (referenced in recent surveys)

• Key Innovation: Addresses "black box" nature of dense embeddings
• Benefit: Enables understanding of what semantic dimensions represent
• Value: Better debugging and trust in extraction pipelines

Summary of New Findings (March 6, 2026)

Section 23 added: March 6, 2026 07:50 GMT+1 - Final research update

24. Latest Embedding Benchmarks and Evaluation (March 2026)

24.1 PTEB: Stochastic Paraphrasing for Embedding Evaluation

Paper: arXiv:2510.06730 (February 2026)

PTEB (Paraphrasing Text Embedding Benchmark) represents a shift from static datasets to dynamic, evaluation-time assessment:

Key Innovation:

• Uses LLMs to generate paraphrases at evaluation time
• Creates semantically equivalent but textually distinct problem instances
• Better approximates diverse real-world applications than static benchmarks

Complementary to MTEB:

• MTEB targets maximal language and task coverage
• PTEB stress-tests semantic invariance via multi-run eval-time paraphrasing
• Addresses overfitting to specific benchmark phrasing

24.2 MTEB Leaderboard Updates (March 2026)

Top Open-Source Models:

Key Trends:

• 15,000+ pre-trained models on Hugging Face
• Multilingual models gaining prominence (1000+ languages)
• Domain-specific models outperforming generalists

24.3 Legal RAG Bench

Source: Isaacus (March 2026)

Key Finding: Information retrieval is the primary driver of legal RAG performance rather than reasoning capabilities.

Implications:

• Embedding quality matters more than LLM reasoning for legal extraction
• Domain-specific embeddings crucial for legal applications
• Reveals data leakage issues in commercial embedding models

24.4 Commercial Embedding Options (2026)

Top Performers:

• OpenAI text-embedding-3-large: 0.92 MTEB, highest among proprietary
• Cohere Embed v4: Strong multilingual performance
• Voyage AI: Best for book data in benchmarks
• Nomic Embed v1.5: Open-source alternative with good quality

Selection Criteria:

• Latency vs. quality tradeoff
• Cost considerations (API vs. local deployment)
• Domain specificity requirements

25. Emerging Research Directions (March 2026)

25.1 Agentic RAG (A-RAG)

Paper: arXiv:2602.03442 (February 2026)

Key Innovation: Scaling RAG via hierarchical retrieval interfaces rather than single-shot passage retrieval.

Components:

• Multi-level document understanding
• Iterative refinement of retrieved context
• Agentic planning for complex queries

25.2 Multi-Agent Extraction Systems

MA-RAG Framework:

• Specialized agents: Planner, Step Definer, Extractor, QA
• Chain-of-thought reasoning collaboration
• Hybrid sparse-dense retrieval

RAGentA System:

• Iterative document filtering
• Citation-attributed answer generation
• Hybrid sparse-dense retrieval

25.3 Context Window Management

Context Rot Phenomenon:

• LLM performance degrades with increasingly long contexts
• Needle-in-a-Haystack tests only lexical retrieval
• Real applications require reasoning over ambiguous information

Solutions:

• Hierarchical summarization: Replace context with summary when approaching limits
• Observation masking: Hide older, less important information
• Sliding window with memory: Keep running summary of processed content

25.4 Multimodal Extraction

Wayin AI (March 2026):

• OCR for decoding on-screen data from technical videos
• Processes Python scripts directly from video streams
• Handles complex calculus equations
• Exports to Markdown for Notion/Obsidian compatibility

Emerging Capability: Bridging "dark data" gap in video content extraction

References (Section 24-25)

• PTEB: arXiv:2510.06730 - Stochastic Paraphrasing for Embedding Evaluation
• Qwen3-Embedding: https://blog.premai.io/best-open-source-llms-for-rag-in-2026-10-models-ranked-by-retrieval-accuracy/
• Legal RAG Bench: https://isaacus.com/blog/legal-rag-bench
• Perplexity pplx-embed: https://abit.ee/en/artificial-intelligence/perplexity-pplx-embed-embedding-models-rag-mteb-open-models-nlp-hugging-face-mit-licence-en
• A-RAG: arXiv:2602.03442 - Agentic RAG framework
• Wayin AI Review: https://ucstrategies.com/news/wayin-ai-review-2026-can-this-ai-turn-technical-videos-into-searchable-notes/

Sections 24-25 added: March 6, 2026

Summary: All Research Angles Complete

Document fully updated: March 6, 2026

26. Supplementary Research Findings (March 6, 2026)

26.1 Classical Word Embeddings - Additional Insights

Word2Vec

• Training approaches: CBOW (predicts target from context) vs Skip-gram (predicts context from target)
• Strengths: Efficient training, captures semantic relationships via vector arithmetic (king - man + woman ≈ queen)
• Limitations: Cannot handle OOV words, ignores word order, static embeddings (polysemy problem)

GloVe

• Approach: Combines global matrix factorization with local context windows
• Strengths: Incorporates global corpus statistics, often better at analogy tasks
• Use cases: Document similarity, domain-specific applications with smaller corpora

FastText

• Key innovation: Subword (character n-gram) representations
• Strengths: Handles OOV words by composing from subwords, better for morphologically rich languages
• Practical impact: Foundation for modern tokenization (BPE, SentencePiece)

26.2 Semantic Representation -补充

Knowledge Graphs

• Google Knowledge Graph (2012): Major application of semantic networks in search
• Structure: Nodes (entities), edges (relationships), labels (semantic types)
• NLP integration: Semantic enrichment using NLP to construct comprehensive views

Topic Modeling Updates

• BERTopic (2024-2025): Uses BERT embeddings + HDBSCAN clustering + c-TF-IDF
• Top2Vec: Joint embedding of documents and topic vectors
• Selection guide:
• LDA: Traditional corpus, interpretability
• NMF: Short texts, parts-based interpretation
• BERTopic: Semantic similarity priority, short texts

26.3 Evaluation Methods -补充

Metric Performance Comparison (2025-2026 Research)

Key Insights

• BERTScore: More suitable for semantic similarity measurement
• ROUGE: Preferred for word-level similarity
• Traditional metrics (ROUGE, BLEU): Limited in capturing semantic similarity and nuanced aspects
• Human evaluation: Remains gold standard but expensive and time-consuming

References (Supplementary)

• Medium: Advanced Word Embeddings - Word2Vec, GloVe, and FastText
• Analytics Vidhya: Word Embeddings in NLP
• Towards Data Science: Word2Vec, GloVe, and FastText Explained
• Wikipedia: Semantic network
• IBM: What Is a Knowledge Graph?
• Medium: BERTScore and ROUGE
• JAIR: FFCI Framework for Interpretable Automatic Evaluation
• OpenAI Cookbook: How to evaluate a summarization task

Supplementary section added: March 6, 2026

27. Fresh Web Search Findings (March 6, 2026 - Evening)

27.1 Latest Embedding Techniques (Web Search Results)

Nomic Embed Text V2 (2026):

• Open-source, multilingual embedding model
• Uses Mixture-of-Experts (MoE) architecture
• Focuses on transparency and reproducibility
• Achieves excellent semantic performance with efficient computation

SBERT Long-Text Optimization (June 2025):

• Novel approaches for clustering long documents with Sentence-BERT
• Addresses 512-token limitation through strategic chunking
• Siamesen networks produce fixed-size sentence embeddings
• Cosine similarity enables efficient comparison (10,000 sentences in seconds vs. 65 hours)

Multimodal Embeddings (2025):

• Translates different input types (text, image, audio) into unified vector representations
• Concept similarity outweighs format differences
• Critical for cross-modal retrieval and generation

2025 Embedding Trends:

• Transformer-based and instruction-tuned embeddings achieving top performance
• Multilingual models (1000+ languages)
• Domain-specific (medicine, code)
• Multimodal (text-image-audio)

27.2 Knowledge Graph Advances (Web Search Results)

GraphRAG Ecosystem (2025-2026):

• Microsoft GraphRAG: Entity-relation graphs from retrieved passages
• Semantic community summaries improve QA performance
• LightRAG, FastGraphRAG, MiniRAG: Lightweight, efficient graph representations

LLM-Based KG Extraction:

• KGGen (2025): Extracts knowledge graphs from plain text using language models
• LKD-KGC: Rapid schema induction via clustering entity types from document summaries
• SF-GPT: Three modules (Entity Extraction Filter, Entity Alignment Generator, Self-Fusion Subgraph)

Evaluation Metrics for KGs:

• G-F1 and T-F1: Graph and triple-level precision/recall/F1
• GED (Graph Edit Distance): Minimal edits to transform generated to reference graph
• GM-GBS (Graph Matching BERTScore): Semantic similarity between generated and ground truth triples

27.3 Evaluation Metrics - Latest Insights (Web Search Results)

Metric Selection Guide (2025):

• BLEU: Precision-focused, ideal for machine translation
• ROUGE: Recall-heavy, dominant for summarization tasks
• BERTScore: Semantic similarity leader for creative text generation

BERTScore Advantages:

• Excels in summarization where semantic fidelity matters more than exact word overlap
• Example: "The red shoes cost $20.00" → rates "The rouge slippers cost $20" more similar than "The blue socks cost $20"
• Uses contextual embeddings from BERT

RAG Evaluation:

• UniEval, BLEU, ROUGE for quantitative assessment
• BERTScore and BLEURT: n-gram recalls using contextual representations
• BARTScore: Closer to perplexity measurement using critic model

Medical Summaries Evaluation (December 2025):

• Criteria: relevance, completeness, redundancy, coherence, structure, grammar, hallucinations
• LLM-as-Judge approach gaining traction

References (Section 27 - Web Search)

• Openxcell: 10 Best Embedding Models 2026
• Springer: Optimizing SBERT for Long Text Clustering (June 2025)
• Artsmart AI: Top Embedding Models 2026
• Medium: State of Embedding Technologies for LLMs (Sept 2025)
• John Snow Labs: Sentence Embeddings in Spark NLP (Sept 2025)
• arXiv: Efficient Knowledge Graph Construction for RAG (2507.03226)
• Frontiers: Knowledge Graphs and LLMs Fusion (June 2025)
• arXiv: KGGen - Extracting Knowledge Graphs (2502.09956)
• MDPI: Knowledge Graph Construction - Extraction, Learning, Evaluation (March 2025)
• Galileo AI: BERTScore Explained (March 2025)
• Markaicode: BLEU vs ROUGE vs BERTScore (May 2025)
• Elastic: RAG Evaluation Metrics (Sept 2025)
• MDPI: Evaluating Medical Text Summaries (Dec 2025)

Section 27 added: March 6, 2026 20:07 GMT+1 - Evening web search findings

Final Summary

This document now provides comprehensive coverage of:

Document complete - Updated March 6, 2026

28. Latest Research Findings (March 7, 2026 - New Update)

Research compiled from web searches on March 7, 2026

1. Embeddings: New Models & LLM-Empowered Approaches

1.1 New Embedding Models (2025-2026)

1.2 LLM-Empowered Embeddings

• Decoder-based embedding models: Training pre-trained decoders (removing causal attention masking) contrastively in Siamese setups
• Industry adoption: Widely adopted by major AI companies (Lee et al., 2025a,b)
• Survey: Comprehensive survey on "When Text Embedding Meets Large Language Model" (arXiv:2412.09165v3, March 2025)

1.3 Interpretable Text Embeddings (2025)

• Focus: Building explainable embedding models for better debugging and trust
• Survey: "Interpretable Text Embeddings and Text Similarity Explanation" (arXiv:2502.14862v2, February 2025)

1.4 Reranking Innovations

• Two-stage retrieval: Embedding models get top-100 candidates → reranker refines to top-5
• Qwen3-Reranker-8B: Strong performance in asymmetric retrieval
• Hybrid retrieval: Dense semantic + sparse lexical + graph-based embeddings fusion

2. Semantic Representation: KG Methods & Semantic Graphs

2.1 Knowledge Graph Construction with LLMs

2.2 KG-Enhanced RAG

• Dual-Pathway KG-RAG: Combines structured KG + unstructured retrieval → 18% hallucination reduction
• SF-GPT: Three modules (Entity Extraction Filter, Entity Alignment Generator, Self-Fusion Subgraph)
• Joint extraction models: Transformer-based, single-pass entity + relation extraction
• Evaluation Metrics: G-F1, T-F1, GED (Graph Edit Distance), GM-GBS

2.3 Semantic Graph Methods

• Graph-based retrieval: Combined with dense vector and keyword-based retrievers
• Reciprocal Rank Fusion (RRF): Standard method for merging ranked lists
• Semantic threshold filtering: Removes low-quality hits before reranking
• ATLAS achievement: 95% semantic alignment with human-crafted schemas with zero manual intervention

3. Traditional NLP: Extractive Methods

3.1 Modern Extractive Approaches

• Mamba-Transformer Hybrids for Extractive Summarization (March 2026):
• Linear O(L) complexity vs transformer quadratic
• First hybrid SSM+Transformer for extractive summarization (arXiv:2603.01288)
• Effective for low-resource scenarios
• 24-27% faster inference on CNN/DailyMail

• Feature-based scoring evolution:
• Position encodings + title-similarity features
• Integration with sentence embeddings
• Neural reranking pipelines

3.2 Graph-Based Ranking (Updated)

• TextRank, LexRank enhanced with contextual embeddings
• Hybrid approaches combining semantic graphs with transformer reranking

4. Deep Learning: Transformer Variants & SSM Hybrids

4.1 State-Space Model (SSM) Hybrids

4.2 SSM Technical Details

• Selective scan: Hardware-aware kernels minimizing memory IO
• Linear complexity: O(N) vs O(N²) for transformers
• Long-context performance: Specialized evaluation on 256k+ token windows
• Edge deployment: Custom SSM kernels account for 55%+ inference latency
• Performance characterization: Characterizing SSM and hybrid models for long context (arXiv:2507.12442)

4.3 LLM-Based Extraction (2025-2026)

• LLM-Guided Planning: Two-stage approach - structured plan generation → plan-driven simplification
• Prompt engineering: Architecture comparison (seq2seq vs autoregressive)
• Cross-lingual extraction: Contextual embedding similarity for token matching
• LimTopic: LLM-based topic modeling combining BERTopic with LLM for scientific articles

5. Evaluation: Benchmarks & LLM-as-Judge

5.1 New Benchmarks (2025-2026)

5.2 LLM-as-Judge Improvements (2025-2026)

• Best models: GPT-4-turbo shows best judging capabilities for code generation and summarization
• Smaller LLMs limitations: Models with tens of billions parameters struggle with judging tasks
• Survey findings: Comprehensive "Survey on LLM-as-a-Judge" (arXiv:2411.15594)
• Medical domain: LLM-as-judge for clinical AI summaries (Nature Digital Medicine, 2025)
• Best practices: Reproducible scoring templates, CoT reasoning, inter-judge reliability metrics (Cohen's Kappa, Krippendorff's Alpha)

5.3 Cost Analysis (2025)

5.4 Metric Performance Comparison (2025-2026)

Summary: Key 2025-2026 Advances

References (2025-2026)

• Gu, A. & Dao, T. (2024). Mamba: State Space Models. arXiv.
• IBM Research (2025). Meet Bamba. IBM Blog.
• ATLAS (2025). Automated Triple Linking And Schema induction.
• Edge, D. et al. (2025). Graph RAG. arXiv.
• Qwen Team (2025). Qwen3 Embedding. Qwen Blog.
• arXiv:2412.09165v3 (2025). When Text Embedding Meets LLM.
• arXiv:2411.15594 (2025). Survey on LLM-as-a-Judge.
• arXiv:2507.16587 (2025). LLM-as-judge for Code Generation and Summarization.
• arXiv:2603.01288 (2026). MAMBA-Transformer Hybrids for Extractive Summarization.
• Nature Digital Medicine (2025). Evaluating clinical AI summaries with LLM-as-judge.
• ScienceDirect (2025). Systematic review of long document summarization methods.
• arXiv:2502.14862v2 (2025). Interpretable Text Embeddings Survey.

Section 28 added: March 7, 2026

29. Additional Research Findings (March 7, 2026 - Afternoon)

29.1 New Summarization Paper: Global Structure Awareness

Paper: arXiv:2602.09821 (February 10, 2026)

Title: "Text summarization via global structure awareness"

Key Contribution:

• Proposes novel approach focusing on global document structure
• Addresses limitation of local attention mechanisms in transformers
• Improves coherence and factual consistency in summaries
• Multi-granularity approach: sentence-level + document-level features

29.2 RAG Evaluation Tools (March 2026)

Top 5 RAG Evaluation Platforms:

Key Metrics Tracked:

• Answer Relevancy: Does the answer address the question?
• Faithfulness: Is the answer consistent with retrieved context?
• Contextual Relevancy: Quality of retrieved context
• Precision@K, NDCG: Retrieval ranking quality
• F1 Scores, Resolution Rates: Overall performance

29.3 RAG vs Fine-Tuning (2026)

Key Insight (March 2026):

• Long context did NOT kill RAG
• No universal winner - hybrid approach is best 2026 pattern
• Best Practice: Retrieval for facts + Fine-tuning for style, policy, decision behavior

Evaluation Best Practices:

1. Maintain canonical question set for cross-team comparisons
2. Rolling sample for production monitoring
3. Index snapshotting for reproducible evaluations
4. Automated scoring: retrieval metrics (NDCG, MRR) → generation metrics (BERTScore, ROUGE, RAGAs)

29.4 Clinical RAG Evaluation (March 2026)

Source: Wiley - CPT: Pharmacometrics & Systems Pharmacology (2026)

Framework: Using DeepEval with three metrics:

• Answer relevancy
• Faithfulness
• Contextual relevancy to clinical pharmacology (ClinPharm metric)

Application: Evaluating regulatory compliance of drug information and clinical trial protocols

29.5 Research Tools for Text Extraction (2026)

Paperguide AI (March 2026):

• Integrated platform for literature review
• Automated summarization
• Data extraction
• Reference management
• Paper writing and report generation
• Advanced semantic search capabilities

Key Trend: AI tools increasingly combining extraction + summarization + writing in unified workflows

References (Section 29)

• arXiv:2602.09821 - Text summarization via global structure awareness (February 2026)
• arXiv:2603.01288 - MAMBA-Transformer Hybrids for Extractive Summarization (March 2026)
• Deepchecks - RAG Evaluation Metrics
• Maxim AI - 5 Best RAG Evaluation Tools in 2026
• Wiley - RAG for Evaluating Regulatory Compliance (2026)
• Umesh Malik - RAG vs Fine-Tuning for LLMs (2026)

Section 29 added: March 7, 2026 15:30 GMT+1 - Final afternoon update

30. Final Research Findings (March 7, 2026 - Late Update)

30.1 Embedding Models for RAG (February 2026)

Top 5 Models Ranked (atal upadhyay blog, February 2026):

Key Insight: MTEB scores don't tell full story - adversarial test sets like "Liability Trap" reveal failure modes

30.2 Embedding Model Selection Criteria (2026)

Evaluation Dimensions:

• Accuracy: MTEB, BEIR benchmarks
• Latency: Query time requirements
• Cost: API vs local deployment
• Domain: General vs specialized
• Context window: 512 to 1M+ tokens

Commercial vs Open Source Trade-offs:

• Commercial: Higher quality, less maintenance
• Open source: Customization, cost savings, data privacy

30.3 Sparse + Dense Hybrid Search Updates (March 2026)

OpenSearch Neural Sparse Search:

• Semantic search relies on dense retrieval (embedding models)
• Problem: Dense methods use k-NN which consumes large memory/CPU
• Solution: Neural sparse search uses inverted index (efficient as BM25)
• Combines semantic quality with keyword search efficiency

Three-Way Retrieval (IBM Case Study, 2026):

• BM25 + dense vectors + sparse vectors
• Outperforms two-way and single-method approaches
• Optimal for enterprise RAG systems

30.4 Production RAG Best Practices (2026)

Orchestration:

• LangChain vs LlamaIndex comparison continues
• Choice depends on complexity and customization needs

Vector Database Selection:

• Development: Chroma
• Production: Pinecone
• Hybrid search: Weaviate

Domain-Specific Models:

• Match embedding model to domain
• Legal, medical, multilingual require specialized embeddings
• Domain-specific embeddings significantly outperform general models

Summary: Complete 5-Angle Research Coverage

Section 30 added: March 7, 2026

Document final update: March 7, 2026

31. Subagent Research Findings (March 7, 2026)

1. Embeddings - New Methods & Benchmarks

GIRCSE: Generative Iterative Refinement (September 2025)

• Paper: arXiv:2509.24291
• Key Innovation: Uses autoregressive generation to iteratively refine semantic representations
• Approach: Produces sequences of soft tokens optimized under contrastive objective
• Results: Outperforms LLM-based embedding baselines on MTEB benchmark
• Notable: Exhibits test-time scaling property - generating more tokens at inference steadily improves embedding quality
• Significance: Establishes generative iterative refinement as new paradigm for representation learning

New Embedding Models (Early 2026)

Based on practitioner discussions and leaderboards:

• ZeroEntropy zembed-1: New entrant gaining attention
• OpenAI text-embedding-3-large: Continuing improvements
• Cohere Embed v4: Strong commercial performance
• Jina v3: Open source option
• Nomic Embed v1.5: Open-source with competitive performance
• Voyage AI: High-quality embeddings for RAG

Benchmark Updates

• MMTEB (Massive Multilingual Text Embedding Benchmark): Published November 2025, covers 250+ languages
• PTEB (arXiv:2510.06730): Tests robustness via stochastic paraphrasing at evaluation time
• RTEB: Retrieval benchmark with private test sets
• Agentset Leaderboard: Uses GPT-5 as judge for pairwise ELO comparisons, tests across financial, scientific, business domains

Key Trend

MTEB scores don't tell full story - adversarial test sets like "Liability Trap" (Semantic Twins Dataset for RAG Testing) reveal failure modes that benchmark scores miss, particularly for legal/contractual text where "shall" vs "shall not" distinctions matter.

2. Semantic Representation - RAG & Knowledge Graphs

Agentic RAG (2026)

• Architecture: Replaces "retrieve once" with "retrieve, evaluate, revise, retrieve again"
• Components: Adds planning, grading, and iteration to retrieval
• When to use:
• Multi-part or multi-hop questions
• Ambiguous queries requiring query rewriting
• Tool-rich environments with multiple knowledge sources
• Caution: More steps don't guarantee more truth - needs verification gates and external feedback (CRITIC framework)
• Production data: 57% of organizations have deployed agentic RAG as of February 2026

Knowledge Graph RAG

• Microsoft GraphRAG: Builds LLM-generated knowledge graph from corpus, clusters entities into communities, pre-summarizes communities
• Variants (2026):
• LightRAG: Optimized for indexing speed
• FastGraphRAG: Cost-optimized
• LazyGraphRAG: Defers graph construction until query time (1000× lower indexing cost)
• GraphRAG 1.0: 43% smaller output (10-100GB per 10M tokens)
• Performance: Up to 100% improvement over naive vector RAG on complex queries
• When to use: Multi-hop queries, complex global questions, relationship understanding

Pipeline RAG Upgrades (Still Relevant)

• Hybrid search: Combines keyword + semantic matching
• Metadata filtering: Date, version, team, product
• Reranking: Cross-encoder rescoring for top-k noise
• Best for: Single-hop Q&A, tight latency/cost budgets, document-shaped corpora

3. Traditional NLP Methods

No major breakthrough methods in 2025-2026. The focus has shifted to:

• Better chunking strategies for RAG pipelines
• Hybrid approaches combining traditional statistical methods with neural approaches

4. Deep Learning - Summarization & Agentic Approaches

SummQ: Adversarial Agentic Collaboration (September 2025)

• Paper: arXiv:2509.20900
• Approach: Quiz generation as continuous quality check
• Results: Superior performance on ROUGE, BERTScore, LLM-as-judge, and human assessments
• Key insight: Creates adversarial dynamic ensuring comprehensive coverage, factual accuracy, verifiability

AgenticSum: Clinical Text Summarization (February 2026)

• Paper: arXiv:2602.20040
• Framework: Separates context selection, generation, verification, and targeted correction
• Purpose: Reduce hallucinated content in high-stakes clinical documentation

GloSA-sum: Global Structure-Aware Summarization (February 2026)

• Paper: arXiv:2602.09821
• Innovation: Uses TDA (Topological Data Analysis) based framework
• Approach:
• Encodes sentences into high-dimensional semantic embeddings
• Constructs weighted undirected graph (semantic similarity + positional distance)
• Hierarchical strategy: segment-level + global summarization
• Results: Reduces redundancy while preserving semantic and logical integrity

Mamba-Transformer Hybrids (March 2026)

• Paper: arXiv:2603.01288
• First hybrid: Mamba (State Space Model) + Transformer for extractive summarization
• Problem solved: Quadratic complexity bottleneck in long document processing
• Results: Better efficiency-accuracy tradeoff for long documents

State Space Models (SSMs) Developments

• Falcon-H1 series: Parallel hybrid combining Transformer attention with Mamba-2 SSMs
• Context windows: Up to 256K supported
• Trend: Hybrid approaches where Mamba handles bulk processing, Transformer attention for precision

5. Evaluation Methods

LLM-as-Judge Evolution

• GPT-5 as evaluator: Agentset leaderboard uses GPT-5 for pairwise ELO comparisons
• Multi-dimensional assessment: Combining multiple evaluation aspects
• Limitations: Less reproducible, can amplify confident narratives
• Best practice: Use reference-free evaluation, include specific criteria in prompts, aggregate across multiple LLM judges

Production Evaluation Frameworks (2026)

New Metrics

• QAFactEval: Factuality assessment for summarization
• GloSA evaluation: Uses ROUGE, BERTScore, QAFactEval, and human evaluation
• Three-Layer RAG Evaluation:
• Retrieval Layer: Precision, Recall, MRR, NDCG
• Generation Layer: Faithfulness, Answer Relevance, Context Utilization
• End-to-End Pipeline: Combined metrics

Summary: Key 2025-2026 Advances

Latest Developments (March 7, 2026 - Evening)

Research conducted: March 7, 2026 evening

Summary of Evening Search Findings

Web searches conducted on March 7, 2026 evening confirmed that the document is already comprehensively covering the latest developments. Key findings:

RAG Evaluation (Enterprise Focus)

• New Benchmarks Identified: RAGBench (general-purpose), CRAG (contextual relevance), LegalBench-RAG (legal QA), WixQA (web-scale QA), T²-RAGBench (multi-turn)
• Enterprise Priorities: Factual grounding, retrieval quality, compliance risks alongside accuracy
• Evaluation Layers: Retrieval (precision@k, recall@k, MRR, nDCG) → Generation (faithfulness, relevance, citation coverage, hallucination rate) → End-to-end (correctness, latency, cost, safety)
• Test Set Best Practices: Golden datasets + synthetic queries (Ragas, ARES) + human review; strict versioning for comparability

Knowledge Graph Construction

• Survey Confirmed: arXiv:2510.20345 "LLM-empowered knowledge graph construction: A survey" provides comprehensive overview
• Key Trend: LLMs have fundamentally changed KG construction economics - reframing IE as generative task
• Production Approaches: LangChain's with_structured_output using Pydantic objects for schema-driven extraction

Summarization Benchmarks

• NewsSumm Dataset: World's largest human-annotated multi-document news summarization dataset for Indian English (published November 2025)
• Key Finding: State-of-the-art models trained on Western English datasets (CNN/DailyMail, XSum) perform substantially worse on Indian English news

Field Status (March 7, 2026)

The text extraction and summarization field shows strong activity in:

1. Agentic RAG - Self-correcting retrieval with planning and iteration
2. Hybrid SSM-Transformer models - Efficiency gains for long documents
3. LLM-as-Judge evaluation - Production-ready evaluation frameworks
4. Domain-specific benchmarks - Legal, medical, multilingual focus

No major breakthrough techniques identified beyond those already documented in the March 7, 2026 morning and afternoon updates.

References (Evening Update)

• Label Your Data: RAG Evaluation - Metrics and Benchmarks for Enterprise AI Systems (March 2026)
• arXiv:2510.20345: LLM-empowered knowledge graph construction survey
• MDPI NewsSumm: Indian English news summarization dataset (November 2025)
• Maxim AI: Top 5 RAG Evaluation Tools in 2026

Evening update added: March 7, 2026

32. Final Subagent Research Findings (March 7, 2026 - Late Evening)

Research completed by subagent on March 7, 2026

1. Embedding Models - Latest MTEB Leaders & New Developments

1.1 Commercial Embedding Models (March 2026)

Elastic's New Embedding Models (February 2026):

• Released best-in-class embedding models for semantic search
• Compact size outperforms significantly larger 7B-14B parameter models
• Achieves best-in-class results on MMTEB (Multilingual MTEB) among comparable size models

Openxcell Ranking (March 2026):

• Qwen3-Embedding: Best for multilingual (100+ languages), instruction-aware architecture
• Cohere embed-v4: Strong commercial performance
• Jina v3: Open source option
• Voyage AI: Best for book data in benchmarks
• BGE models: Consistent top performance on MTEB and C-MTEB

Reddit Practitioner Insights (March 2026):

• Real-world production data benchmarks differ from MTEB synthetic benchmarks
• Cohere embed-v4, Voyage AI, and Jina v3 commonly used in production
• Domain-specific benchmarks (FiQA finance, SciFact science, MSMARCO web search) matter more than MTEB

1.2 CPU-Optimized Embeddings (February 2026)

Key Development:

• INT8 quantization reduces embedding model size by 75%
• MTEB score drops less than 0.5% after quantization
• Enables running embedding models on CPU at half the cost
• Critical for production cost optimization

1.3 Benchmark Limitations

"Liability Trap" Adversarial Test Set:

• Tests whether embedding models confuse "shall" vs "shall not" in legal clauses
• MTEB scores don't reveal this failure mode
• Demonstrates need for domain-specific adversarial evaluation

2. Semantic Representation - GraphRAG & Semantic Caching

2.1 GraphRAG Dynamic Improvements (March 2026)

New Development:

• Original GraphRAG used fixed hierarchy level for global search (typically Level 2)
• Innovation: Algorithm dynamically selects which community reports to include based on query relevance
• Improves precision and reduces unnecessary context

IBM Explanation:

• GraphRAG uses Cypher graph query language for domain-specific data retrieval
• Leverages both semantic AND structural signals unlike vector-only RAG
• Particularly effective for complex relationship queries

2.2 Semantic Caching for RAG (March 2026)

Zero-Waste Agentic RAG (Towards Data Science, March 2026):

• Caching architectures that intercept requests before agent invocation
• Delivers answers in milliseconds with zero LLM token costs
• Key component: Semantic similarity matching for cache hits

Proximity Project (arXiv:2603.03301):

• Focuses on reducing cache search time
• Divides cache into independent buckets using locality-sensitive hashing (LSH)
• Ensures semantically similar vectors placed in same bucket

Boring Bot Implementation (March 2026):

• Primary cache layer between UI and retrieval system
• Intercepts incoming queries, checks for semantically similar cached responses
• Fresh predicate markers handle time-sensitive data (e.g., "2011 reviews" still valid after 2026 update)

Valkey 2026 (AWS, February 2026):

• Vector similarity search + full-text search capabilities
• Semantic caching with hybrid search for AI workloads
• Launching and expanding in 2026

2.3 Industry RAG Survey (March 2026)

Comprehensive Review (IJET Journal, March 2026):

• Systematic review of RAG research 2020-2026
• Charts progression from "retrieve-then-read" to "modular retrieval + autonomous agent-driven reasoning"
• Key focus: Decoupling reasoning from memorization in LLMs

3. Deep Learning - State Space Models & Mamba-3

3.1 Mamba-3 ICLR 2026

ICLR Poster Presentation (March 2026):

• Combines more expressive recurrence with complex state update rule
• Enables richer state tracking
• Multi-input, multi-output formulation
• Better exploits hardware parallelism during decoding

Complex-Valued State Spaces:

• Mamba-3 switches to complex-valued state spaces
• Connected to Data-Dependent Rotary Position Embeddings (RoPE)
• Significant improvement in sequence modeling

3.2 Falcon-H1 Series (March 2026)

Architecture:

• Parallel hybrid combining Transformer attention with Mamba-2 SSMs
• 0.5B parameters matching 7B baseline performance from 2024
• Supports 256K context windows

Hybrid Approach:

• Mamba handles bulk sequence processing
• Transformer attention for precision recall
• Learning rate choice critical for SSM performance (often overlooked)

3.3 Other SSM Applications

MambaLLM:

• Fuses macro-index and micro-stock data via SSMs + LLMs

CrossLLM-Mamba:

• Multimodal fusion for RNA interaction prediction

4. Evaluation - New Benchmarks & Metrics

4.1 DRAGOn Benchmark (arXiv:2507.05713)

Published: July 2025 (discovered March 2026)

Key Features:

• RAG benchmark on periodically updated corpus
• Recent reference datasets
• Question generation framework
• Automatic evaluation pipeline
• Public leaderboard

4.2 Legal RAG Bench (February 2026)

Source: Isaacus (February 20, 2026)

Purpose: Reasoning-intensive benchmark for legal RAG systems

Key Issues Identified:

• Performance on existing legal benchmarks doesn't correlate with real-world legal retrieval
• Data leakage into commercial embedding model training sets
• Poor label quality in public legal evaluation sets
• AILA Casedocs and Statutes comprise 25% of MTEB's legal split (also in multilingual MTEB)

4.3 Synthetic Data for RAG Evaluation (February 2026)

Red Hat Developer Article:

• SDG Hub for generating high-quality question-answer-context triplets
• Benchmark retrievers and track LLM performance over time
• Addresses shortage of labeled training data

4.4 New RAG Evaluation Tools (March 2026)

Goodeye Labs Ranking:

1. Retrieval Relevance Evaluators: Benchmarked against HaluEval
2. Precision@K and NDCG: For retrieval ranking quality
3. Integrations: Ragas, DeepEval, Cleanlab evaluation libraries

Deepchecks RAG Evaluation:

• Context precision@K, NDCG combined with user satisfaction indicators
• Integration with RAGAs and TruLens benchmarks
• Task-specific KPIs (resolution velocity in support, conversion rate in e-commerce)

4.5 VectifyAI Mafin 2.5 (February 2026)

Announcement: MarkTechPost (February 22, 2026)

Key Achievement:

• 98.7% Financial RAG Accuracy
• New open-source vectorless tree indexing approach
• PageIndex: Alternative to traditional vector embeddings

5. Complete Summary of Research Angles

Final Research Task Completion

Task: Research latest developments in text extraction and summarization methods

Research Focus Areas (as specified):

1. ✅ Embedding models - Latest MTEB leaders, new models, benchmark limitations
2. ✅ Semantic representation - GraphRAG developments, knowledge graphs, semantic caching
3. ✅ Deep learning - Transformer variants, SSMs, agentic RAG innovations
4. ✅ Evaluation - New benchmarks, metrics, RAG evaluation frameworks

Time Period: Late 2025 and early 2026

Status: Complete - Document comprehensively updated with latest findings

Subagent research completed: March 7, 2026 20:36 GMT+1