Video analytics that reveal and remove assembly-line bottlenecks

Client Industry

Manufacturing · assembly-line operations

Key Challenge

Subjective, manual time-and-motion observation on the line

Core Tech Stack

SlowFast · NVIDIA Jetson · Edge inference · MLOps

Deployment Model

On-prem edge inference at the workstation

SoftmaxAI developed LineSense, a production-quality edge video-analysis platform which monitors assembly-line workstations and converts worker movements into objective and unbiased time-and-motion data. The cameras send their footage to an on-premises NVIDIA Jetson device, where a SlowFast human-action-recognition model is run directly at the edge — no video leaves the floor — while a dedicated MLOps pipeline is used for annotation, augmentation, experiment tracking and deployment in order to keep the model up to date. The analytics system detects bottlenecks in real time, triggers immediate alerts, and stores the archived footage so that teams can carry out root-cause analysis regarding exactly what occurred · when · where.

• Manual observation was subjective. Since time-and-motion studies required a person equipped with a stopwatch, the results differed from observer to observer and it was impossible for them to continuously cover every workstation.
• Cycle times drifted without explanation. Although bottlenecks, quality-related rework, and inconsistent cycle times continued to be a problem, the team did not have an accurate, up-to-date view of where the time was actually being lost.
• Recognizing worker actions is hard. Telling the difference between significant assembly actions and normal motion in live video requires a temporal model rather than frame-by-frame classification — and it has to be fast.
• Footage could not leave the floor. Because of the latency and data-governance constraints it was not possible to stream the raw video to the cloud and so the inference had to take place on the premises, at the workstation, in real time.

LineSense operates as an edge-native pipeline on the factory floor, comprising a process that starts with the live cameras at the workstations, proceeds via on-device SlowFast inference, and then feeds into a time-and-motion analytics engine which provides bottleneck insights, issues real-time alerts, and supplies archived footage for root-cause analysis, all while an MLOps pipeline continuously trains and deploys the model.

• Spec before code. We prepared a specification that had been approved by the client, outlining which workstations were to be monitored, the specific worker actions that the model had to recognize, and the way in which the insights were to be passed down to the floor — all before any development took place.
• Inference at the edge. The senior engineers deployed on-premise inferencing on NVIDIA Jetson devices using a SlowFast action-recognition network, enabling the live video to be analyzed at the workstation without relying on a cloud round-trip.
• An MLOps pipeline behind the model. Data annotation, augmentation, experiment tracking, and deployment were put together in the form of a repeatable pipeline, so that the recognition model could be retrained and re-shipped to the edge as the production line evolved.
• Human-owned evaluation. The engineers had control over the evaluation of the model and the interpretation of the results, which ensured that the analytics remained trustworthy and were based on what actually takes place on the line rather than on a demonstration.

• Objective time-and-motion data. Instead of relying on subjective manual observations, continuous measurements taken by camera systems provide each workstation with an objective and impartial account of how time is used.
• Bottlenecks made visible. By converting the identified actions into a real-time display of the points where the line slows down, the analytics engine enables teams to focus on the actual causes of variation rather than having to guess them.
• Instant alerting on the floor. If the system detects a stall or an anomaly it sends out an immediate alert and thus moves the team from carrying out a review after the fact to responding at the time it happens.
• Root-cause analysis from archived footage. Since each incident can be traced back to the specific recorded clip, operators are able to replay what took place and identify the causes instead of having to reconstruct the events from memory.
• A model that keeps pace with the line. Using the MLOps pipeline, new actions and workstations can be annotated, trained, and redeployed to the edge, enabling the platform to adapt whenever products and processes change.