Preprocessing: what survived and what did not
The classical preprocessing pipeline — lowercase, remove punctuation, strip stop words, stem to root forms — was designed for bag-of-words models which couldn't use word order or morphology at all. Applied to a modern transformer, which is specifically built to use both, most of those steps remove information that the model is designed to exploit.
| Technique | Status | Reasoning |
|---|---|---|
| Lowercasing | Usually skip | Case carries meaning: acronyms, proper nouns, product codes |
| Stop-word removal | Skip | “Not”, “without”, and “before” are stop words that invert meaning |
| Stemming / lemmatisation | Skip for neural, keep for keyword index | Subword tokenisers handle morphology; BM25 still benefits |
| Punctuation stripping | Skip | Sentence boundaries and negation depend on it |
| Deduplication | Essential | Duplicate passages crowd out diverse evidence in retrieval |
| Layout-aware parsing | Essential | A two-column PDF read straight across produces nonsense |
| Language detection | Keep | Routing to the right model or index still matters |
What replaced them is unglamorous and in fact far more important: getting clean text out of the source format in the first place. A table flattened into a wall of numbers, or a scanned page read badly, is a defect which no subsequent technique can overcome.
Chunking: the decision that sets the ceiling
If a system retrieves over documents, then how those documents are broken up into passages determines pretty much everything about its accuracy. That's by far the most neglected technique in applied NLP.
- Split on structure, not character count. Sections, clauses and headings are meaningful breakpoints. A fixed 500-character split will cut sentences in half and separate a definition from the term it defines.
- Carry context into every chunk. Prefix each passage with its document title and section path. A retrieved fragment that doesn't state what it belongs to is often of very little use.
- Overlap deliberately. A small overlap between adjacent chunks ensures that an answer which straddles a boundary isn't lost, at the expense of some index size.
- Keep tables intact. Splitting a table across chunks destroys it. Extract the tables separately and represent them in a form which will survive retrieval.
- Size to the question, not to a rule. Reference lookups favour small, precise chunks; narrative or procedural content is better served by larger chunks which preserve the sequence.
If a retrieval system is giving you confidently wrong answers, check the chunks before blaming the model. In our experience, most 'hallucination' reports are actually a chunking or parsing defect two steps up the chain.
Embeddings, hybrid retrieval, and reranking
Those three together are the core of modern applied NLP, and improving them will generally get you more accuracy than any change to the generation model.
Embeddings
Text is mapped to vectors which are positioned by meaning, so passages that are semantically similar sit close to each other in the space regardless of whether they share any words. That gives you semantic search, clustering, deduplication and topic grouping from a single representation.
Hybrid retrieval
Vector search is excellent at meaning and poor at exact tokens; keyword search (e.g. BM25) is the reverse. The things users search for — part numbers, error codes, clause references and internal acronyms — are precisely the things embeddings blur. Running both and merging the results covers each blind spot. That should be the default, not an optimisation.
Reranking
Retrieval is all about optimising for recall: cast a wide net, perhaps 50 candidates. A cross-encoder reranker then reads each candidate alongside the query and assigns a relevance score, promoting the small number that are actually relevant. Adding a reranker to a plain vector pipeline is routinely the biggest single accuracy boost you can achieve, for a few tens of milliseconds.
Classification, extraction, and tagging
Text classification
Labelling a document (ticket category, sentiment, urgency, intent). You can get a general model to do this straight away, and that's ideal while the label set is still in flux. Once you've got high volume and the labels are stable, a small fine-tuned classifier costs a fraction as much per call, responds much more quickly and is far more predictable.
Named entity recognition and structured extraction
The real business value lies in pulling specific facts out of prose: the parties and dates from contracts, the amounts and references from invoices, the findings from clinical notes. Schema-constrained generation has changed this dramatically — the model is required to output in accordance with a declared schema, so extraction produces validated structured data rather than prose to parse.
- Declare the schema, do not parse the prose. Constraining the output to a schema removes an entire class of brittle post-processing.
- Model absence explicitly. Give each field a way to be reported as missing. Otherwise, models will make up plausible values for fields which the document does not in fact contain.
- Keep a pointer to the source span. Extraction that records where in the document the value came from is auditable; extraction that doesn't is just a claim.
- Validate against business rules. Dates in range, totals that reconcile, references that resolve. Cheap checks pick up most extraction errors before they become anything serious.
Sentiment and aspect analysis
Overall sentiment is rarely of much use by itself. It's aspect-based analysis — working out what the sentiment is about — that turns a pile of reviews into something you can actually make a product decision with, as in our comment insights case study.
Techniques for making sense of a whole corpus
- Clustering over embeddings. Grouping documents by meaning in order to identify the themes present in a corpus — the practical successor to classical topic modelling for the vast majority of applications.
- Near-duplicate detection. Finding passages that say the same thing in different words, which is important for retrieval quality and for identifying contradictory versions of a policy.
- Summarisation at two levels. Summarising individual documents and then summarising across them, which is how you get an answer that draws on hundreds of documents without exceeding your context window.
- Query rewriting. Rewriting the user's question for retrieval — resolving pronouns from the conversation history and breaking a compound question down into targeted sub-queries. Cheap, and one of the most effective retrieval improvements available.
- Multi-hop retrieval. Running several dependent retrievals, where the second query is informed by the first result. Essential for questions that no single passage can answer on its own.
How SoftmaxAI selects NLP techniques
We build our language systems the same way we build everything else. We have a written, client-approved specification prior to any code. We have an agent-accelerated build led by senior engineers, and every decision that matters is owned by a human.
- Cheapest technique that clears the bar. We size against real throughput and latency, as at volume the difference between a prompted model and a small fine-tuned one is the difference between viable and unaffordable.
- Retrieval before generation. We tune the parsing, chunking, hybrid search and reranking until retrieval recall is high, and only then pick the generation model. Doing it the other way around just wastes budget.
- Evaluation written first. The golden set is developed with your domain experts as part of the specification, so "better" is something that can be measured rather than argued about.
- Deployed to your cloud. With production evaluation and monitoring in your own environment — natively on Vertex AI and Gemini where that makes sense, as a Google Cloud Partner.
New to the field? Start with our beginner's guide to NLP, then read what retrieval-augmented generation is for an understanding of how these techniques fit together into a system.
NLP techniquesChunkingHybrid searchRerankingExtractionClassificationEmbeddingsQuery rewriting