Automatic video metadata that creators can rely on

Client Industry

Media · video content platform

Key Challenge

Manual, inconsistent video summaries, titles & chapters

Core Tech Stack

LLM · LangChain · Qdrant · Langfuse

Deployment Model

API-integrated metadata pipeline

SoftmaxAI incorporated production-quality AI into a media platform's product, the purpose of which is to convert raw video into structured, publish-ready metadata by automatically creating summaries, titles, and timestamped chapters for each uploaded video. The video is first transcribed, then an LLM analyzes the transcript using LangChain, and a Qdrant vector store — having been moved from Pinecone with optimized batch upserts — ensures that retrieval remains quick even as the catalogue increases. Observability is provided by Langfuse throughout the pipeline so that the team can examine the outputs and adjust the prompting based on actual feedback, while custom APIs deliver the results directly into the client's product.

• Cataloguing was manual and slow. The creators prepared the summaries, titles, and chapter markers themselves, carrying out work that was repetitive and which increased in amount in direct proportion to the volume of the content.
• Quality was inconsistent. Hand-written metadata varied in tone, length, and completeness from one creator to the next, which made the catalogue harder to browse and search.
• Long transcripts are messy. Genuine video transcripts are long and full of noise; if simple prompting is used it will either drift off topic or lose the thread, which is why the system needed to reason reliably about unstructured and imperfect text.
• Retrieval had to scale cleanly. The vector infrastructure supporting retrieval had to increase in size as the library grew without causing a sharp rise in cost or operational overhead.

The system operates as a pipeline that runs from left to right — starting with the video uploaded by a creator, then going through transcription and LLM reasoning, before being stored in a Qdrant vector store and finally coming out as generated metadata supplied via custom APIs — with Langfuse observing each step.

• Spec before code. We prepared a specification that had been approved by the client, outlining the transcription pipeline, each metadata output, and the method by which quality would be assessed before any production code was written.
• LLM reasoning over transcripts. An LLM, controlled by LangChain, transcribes and reasons about video, and prompt engineering has been strengthened in order to avoid problems caused by long and messy transcripts and thus maintain the coherence of the summaries, titles, and chapters.
• Vector store migration. We moved the retrieval from Pinecone to Qdrant, using optimized batch upserts, which has enabled the platform to have a vector database that scales well as the amount of content increases.
• Observability-driven iteration. Langfuse keeps track of each generation so that humans can review the outputs, detect any regressions, and improve their prompting based on actual feedback — decisions regarding quality have remained the responsibility of humans.

• Automatic video summaries. Each video that has been uploaded is reduced to a clear and consistent summary without the need for the creator to write it by hand.
• Generated titles and chapters. The system proposes titles and timestamped chapter names automatically, turning a long recording into a navigable, structured asset.
• Consistent metadata at scale. Since all the outputs have the same structure in the catalogue, browsing and search remain consistent as the library expands.
• Scalable retrieval on Qdrant. The Qdrant store that has been migrated and which uses batch upserts is able to maintain fast and cost-effective vector retrieval even as the amount of content grows — without having the operational overhead associated with the previous setup.
• Continuous quality visibility. With Langfuse observability the team is able to obtain a continuous overview of generation quality, which in turn makes it easy to review, debug and improve the system based on actual usage.