Live CCTV surveillance that detects, reasons, and alerts in real time

Client Industry

Security technology · video surveillance platform

Key Challenge

Real-time AI on live camera streams, without alert noise

Core Tech Stack

Python · FastAPI · ZeroMQ · YOLO · CLIP · VLMs

Deployment Model

Multi-process pipeline · GPU or edge accelerators

SoftmaxAI designed the core backend systems along with the computer vision and generative AI components for a production-level live CCTV surveillance platform that runs AI on the camera streams as they are received. The cameras connect via RTSP; each stream is continuously decoded and goes through motion-gated object detection and tracking, with face recognition, license plate recognition, and vision-language reasoning added onto this. Alerts are triggered within seconds of an event, each detection is indexed so that it can be searched by means of text, image, or face, and the monitoring logic is expressed using simple language rules or visual workflows — it is a live surveillance system, not a recorder with search added on.

• Answers only came after the fact. A recorder answers 'what was recorded?' and leaves a person to watch hours of footage — the operators needed answers as events happened: for example, who is on site without a helmet, how many people are waiting at the entrance, and which vehicles entered today.
• Real time is unforgiving. Detection, tracking, recognition, and reasoning had to be carried out on many concurrent streams frame by frame, and no slow model, provider call, or network timeout was permitted to cause the video processing to stall.
• Raw detections make unusable alerts. A single person lingering in a doorway can trigger the system continuously; without cooldowns, de-duplication, and crossing debounce, the operators end up muting the system and it ceases to matter.
• Monitoring logic could not require engineers. The operators had to be able to state what they cared about — such as zones, thresholds, PPE rules, and watchlists — either in plain language or on a visual canvas, without the need for configuration files or model training.

The platform operates using a decoupled, multi-process pipeline consisting of the stages of capture, perception, understanding and action; the RTSP streams are decoded with motion detection used to gate the object detector, the tracked objects are asynchronously enriched with recognition and embeddings, and a ZeroMQ event bus sends each detection on to the rule engines, the alerting system and the searchable event index.

• Embedded product engineering. We designed and implemented the FastAPI layer, including the CRUD operations for rules, workflows, zones, assets, and dashboards; the multi-modal search endpoints; and the metric and analytics queries — with database access being separated behind a persistence layer so that the endpoints remain thin and can be reused by background processors.
• A decoupled multi-process pipeline. The video frames are transferred between processes through shared memory without being copied, and each detection from ZeroMQ's pub/sub mechanism is sent separately to the rules, the workflows, the metrics, and the embeddings — the more resource-intensive recognition models operate in processes that are supervised by a watchdog and lie off the critical path.
• Processors that turn detections into alerts. We developed the real-time, periodic vision-language, metric, and workflow processors, using zone, confidence, object-age, cooldown, and active-hours filters together with per-track de-duplication and crossing debounce so that each alert is worth reading.
• Plugin-registry model integrations. The YOLO-family, RF-DETR, and open-vocabulary detectors; the OC-SORT, BoT-SORT, and Norfair trackers; the face and CLIP-family embeddings; and the LLM/VLM providers, from Gemini and Vertex AI to self-hosted vLLM, each sit behind a common interface, enabling deployments to switch models without having to modify the call sites.

• Alerts within seconds, not reviews after the fact. Real-time, scheduled vision-language, metric-threshold, and workflow rules continuously monitor each stream and generate alerts shortly after the event has occurred, retaining with each alert the frame that triggered it, the confidence level, and the dispatch latency so that a noisy rule can be diagnosed on the basis of evidence.
• Everything searchable afterwards. Each object that is tracked becomes an event stored in a vector index and can be searched using a natural-language description, a similar image, a face, or the name of a person, vehicle, or plate, there being no need for those concepts to be trained as specific classes.
• Plain-language configuration. It is possible to create rules, workflows, dashboards, and search filters based on a natural-language description and then check them against a schema before saving — there being a marketplace offering prebuilt workflow templates for situations relating to PPE, loitering, capacity, and license plates.
• Role-based, camera-scoped access. There are four hierarchical roles together with per-user camera permissions, and these are enforced on the server at the data level — if a user does not have permission for a particular camera, that camera is not included in the API response rather than just being hidden by the interface.
• Degrades gracefully under load. Instead of halting the cameras, a slow recognition model causes only one feature to degrade; when supervised processes fail they restart, backpressure is applied using bounded queues, and latency metrics at each stage trace a regression to a single model.