Opens in a new tab
Home / Blog / Artificial Intelligence

Essential NLP Techniques: The Practical Toolkit

Most NLP technique lists were written before the advent of general-purpose language models and have never been updated. Here's the toolkit that still earns its place today, and what each technique is actually for.

SoftmaxAI EngineeringPublished March 3, 2026Updated August 25, 202613 min read
Topic
Practical NLP techniques and when each applies
Still essential
Chunking, embeddings, hybrid retrieval, reranking
Largely superseded
Stemming, stop-word removal, hand-built parsing
Selection rule
Cheapest technique that meets the accuracy bar

Preprocessing: what survived and what did not

The classical preprocessing pipeline — lowercase, remove punctuation, strip stop words, stem to root forms — was designed for bag-of-words models which couldn't use word order or morphology at all. Applied to a modern transformer, which is specifically built to use both, most of those steps remove information that the model is designed to exploit.

TechniqueStatusReasoning
LowercasingUsually skipCase carries meaning: acronyms, proper nouns, product codes
Stop-word removalSkip“Not”, “without”, and “before” are stop words that invert meaning
Stemming / lemmatisationSkip for neural, keep for keyword indexSubword tokenisers handle morphology; BM25 still benefits
Punctuation strippingSkipSentence boundaries and negation depend on it
DeduplicationEssentialDuplicate passages crowd out diverse evidence in retrieval
Layout-aware parsingEssentialA two-column PDF read straight across produces nonsense
Language detectionKeepRouting to the right model or index still matters
Classical preprocessing steps today

What replaced them is unglamorous and in fact far more important: getting clean text out of the source format in the first place. A table flattened into a wall of numbers, or a scanned page read badly, is a defect which no subsequent technique can overcome.

Chunking: the decision that sets the ceiling

If a system retrieves over documents, then how those documents are broken up into passages determines pretty much everything about its accuracy. That's by far the most neglected technique in applied NLP.

  • Split on structure, not character count. Sections, clauses and headings are meaningful breakpoints. A fixed 500-character split will cut sentences in half and separate a definition from the term it defines.
  • Carry context into every chunk. Prefix each passage with its document title and section path. A retrieved fragment that doesn't state what it belongs to is often of very little use.
  • Overlap deliberately. A small overlap between adjacent chunks ensures that an answer which straddles a boundary isn't lost, at the expense of some index size.
  • Keep tables intact. Splitting a table across chunks destroys it. Extract the tables separately and represent them in a form which will survive retrieval.
  • Size to the question, not to a rule. Reference lookups favour small, precise chunks; narrative or procedural content is better served by larger chunks which preserve the sequence.

If a retrieval system is giving you confidently wrong answers, check the chunks before blaming the model. In our experience, most 'hallucination' reports are actually a chunking or parsing defect two steps up the chain.

Embeddings, hybrid retrieval, and reranking

Those three together are the core of modern applied NLP, and improving them will generally get you more accuracy than any change to the generation model.

Embeddings

Text is mapped to vectors which are positioned by meaning, so passages that are semantically similar sit close to each other in the space regardless of whether they share any words. That gives you semantic search, clustering, deduplication and topic grouping from a single representation.

Hybrid retrieval

Vector search is excellent at meaning and poor at exact tokens; keyword search (e.g. BM25) is the reverse. The things users search for — part numbers, error codes, clause references and internal acronyms — are precisely the things embeddings blur. Running both and merging the results covers each blind spot. That should be the default, not an optimisation.

Reranking

Retrieval is all about optimising for recall: cast a wide net, perhaps 50 candidates. A cross-encoder reranker then reads each candidate alongside the query and assigns a relevance score, promoting the small number that are actually relevant. Adding a reranker to a plain vector pipeline is routinely the biggest single accuracy boost you can achieve, for a few tens of milliseconds.

Classification, extraction, and tagging

Text classification

Labelling a document (ticket category, sentiment, urgency, intent). You can get a general model to do this straight away, and that's ideal while the label set is still in flux. Once you've got high volume and the labels are stable, a small fine-tuned classifier costs a fraction as much per call, responds much more quickly and is far more predictable.

Named entity recognition and structured extraction

The real business value lies in pulling specific facts out of prose: the parties and dates from contracts, the amounts and references from invoices, the findings from clinical notes. Schema-constrained generation has changed this dramatically — the model is required to output in accordance with a declared schema, so extraction produces validated structured data rather than prose to parse.

  • Declare the schema, do not parse the prose. Constraining the output to a schema removes an entire class of brittle post-processing.
  • Model absence explicitly. Give each field a way to be reported as missing. Otherwise, models will make up plausible values for fields which the document does not in fact contain.
  • Keep a pointer to the source span. Extraction that records where in the document the value came from is auditable; extraction that doesn't is just a claim.
  • Validate against business rules. Dates in range, totals that reconcile, references that resolve. Cheap checks pick up most extraction errors before they become anything serious.

Sentiment and aspect analysis

Overall sentiment is rarely of much use by itself. It's aspect-based analysis — working out what the sentiment is about — that turns a pile of reviews into something you can actually make a product decision with, as in our comment insights case study.

Techniques for making sense of a whole corpus

  • Clustering over embeddings. Grouping documents by meaning in order to identify the themes present in a corpus — the practical successor to classical topic modelling for the vast majority of applications.
  • Near-duplicate detection. Finding passages that say the same thing in different words, which is important for retrieval quality and for identifying contradictory versions of a policy.
  • Summarisation at two levels. Summarising individual documents and then summarising across them, which is how you get an answer that draws on hundreds of documents without exceeding your context window.
  • Query rewriting. Rewriting the user's question for retrieval — resolving pronouns from the conversation history and breaking a compound question down into targeted sub-queries. Cheap, and one of the most effective retrieval improvements available.
  • Multi-hop retrieval. Running several dependent retrievals, where the second query is informed by the first result. Essential for questions that no single passage can answer on its own.

How SoftmaxAI selects NLP techniques

We build our language systems the same way we build everything else. We have a written, client-approved specification prior to any code. We have an agent-accelerated build led by senior engineers, and every decision that matters is owned by a human.

  • Cheapest technique that clears the bar. We size against real throughput and latency, as at volume the difference between a prompted model and a small fine-tuned one is the difference between viable and unaffordable.
  • Retrieval before generation. We tune the parsing, chunking, hybrid search and reranking until retrieval recall is high, and only then pick the generation model. Doing it the other way around just wastes budget.
  • Evaluation written first. The golden set is developed with your domain experts as part of the specification, so "better" is something that can be measured rather than argued about.
  • Deployed to your cloud. With production evaluation and monitoring in your own environment — natively on Vertex AI and Gemini where that makes sense, as a Google Cloud Partner.

New to the field? Start with our beginner's guide to NLP, then read what retrieval-augmented generation is for an understanding of how these techniques fit together into a system.

NLP techniquesChunkingHybrid searchRerankingExtractionClassificationEmbeddingsQuery rewriting

FAQ

Questions we get asked.

Which NLP preprocessing steps are still necessary today?
Deduplication, layout-aware document parsing and language detection are all still essential. Lowercasing, stop-word removal, stemming and punctuation stripping are generally counterproductive with current models, as they remove the case, negation and sentence-boundary information that the model is specifically designed to use. Stemming is still useful for a keyword index.
What is the most important NLP technique for accuracy?
For any system that retrieves over documents, the combination of structure-aware chunking, hybrid retrieval and cross-encoder reranking. That combination typically improves answer quality more than upgrading the generation model, and reranking on its own is often the biggest single gain you can achieve.
Why is chunking so important?
Because retrieval can only return what's in the chunks. Splitting on fixed character counts cuts sentences apart, separates definitions from their terms and destroys tables. Splitting on document structure (carrying the title and section into each chunk) and keeping tables intact sets the ceiling for everything that comes after.
Should I fine-tune a model or use prompting?
Prompt a general model while the requirements are still evolving — it requires no labelled data and can iterate in hours. Switch to a small fine-tuned model once you've got sufficient volume and the label set or output schema is stable, as the cost, latency and predictability benefits become decisive at scale.
What is hybrid search and why does it matter?
Hybrid search runs a semantic vector search and a keyword search in parallel and then fuses the results. This matters because embeddings capture meaning but blur exact tokens, so part numbers, error codes and acronyms — precisely what users are looking for — are found by the keyword search and missed by vectors alone.
How do I stop an extraction model inventing values?
Constrain output to a declared schema, give every field an explicit way of being reported as absent, require a pointer to the source span the value came from, and validate the results against business rules (e.g. date ranges, totals that must reconcile).
Start here

Building something with language data? Let’s pressure-test it — free.

A 30-minute call with the engineers who build these systems. Bring your corpus and your toughest question, and you'll get an honest assessment of which techniques your problem actually requires.