Beyond viewing · Planned

Retrieval pipelines

Splitting a document by character count cuts tables in half and separates a heading from what it heads. A chunker that can read the document model does not have to.

Not shipped yet

This page describes work that is planned, not an API you can call today. It is here so you can judge whether the direction fits before adopting the viewer. Follow the roadmap for where it stands.

Why

Retrieval quality is decided long before the model is called, by how the document was cut up. The default everywhere is a fixed window over a flat string, because that is all a flat string allows — and it produces chunks that begin mid-sentence, tables split across three of them and headings orphaned from their sections.

The parse already knows where the headings, the tables and the list boundaries are. Chunking on that structure instead of on a character count is not a research problem; it is only unavailable because the extractor usually throws the structure away first.

The expected shape

Sketched to show the direction. Names and signatures will change before any of it is released.

chunk.ts

import { chunk } from '@apertura/extract';

const chunks = chunk(document, {
  target: 1200,          // characters, as a goal rather than a rule
  splitOn: ['heading', 'table', 'section'],
  keepTogether: ['table', 'list'],
  carryHeadings: true,   // each chunk knows the headings above it
});

// Every chunk carries the anchor that points back into the document.
chunks[0].anchor;
chunks[0].headings;  // ['3. Termination', '3.2 Notice period']

What this is not

Not embeddings, not a vector store, not a retriever and not an agent. Those are well served already, and adding another would be work spent where nothing is missing. This is the step before them — the one where the quality is actually lost — and the anchors it carries are what let the answer point back at the original document.