Cobalt Systems

Cobalt systems large and small

A 3 camera Cobalt based system

Cobalt is an engine that performs advanced video scene analysis. It is a piece of software that runs under the Windows Operating System (32-bit and 64-bit). It takes its video from a pre-recorded video sequence, or from a live feed. Multiple Cobalt engines can run simultaneously, each being fed with its own video stream. The output from the engine is analysed video in a variety of forms.

The installer or the user directs Cobalt as to what to do and which results to store or transmit. It does this through the Cobalt engine's own user interface. Since Cobalt is an open system, the results of the analysis can be redirected to any other third party system that either runs on the Cobalt computer, or that is connected to the Cobalt computer via TCP/IP.

Small Cobalt rack system

The system displayed in the diagram above is among the simplest systems that can be put together. However, Cobalt computers can be linked via a local area network and combined into substantial systems handling data from large numbers of cameras. Such large systems will also need to run a Cobalt Server in order to collate and redistribute all the information generated by the Cobalt computers to the required places. Cobalt is a client-server system, though it does not need a server in the single computer case illustrated.

The system shown on the left is a rack system with an engineering console.The number of cameras handled per computer depends on the cpu horsepower of the computer. A netbook might handle one or two cameras, while a dual Xeon machine might handle 16 or more, depending on the tasks demanded of the Cobalt engine.

A system like that shown to the left would perhaps be found in an air conditioned computer room. There would be remotely connected engineering consoles located in the system manager's room and operator consoles placed in a control room.

Almost any number of systems like that shown can be hooked together, even if they are on different sites.

The Cobalt engine can also handle the transcoding and analysis of DVR video recordings: this creates an archive of automatically generated video metadata that can be rapidly searched. The requirement here is that the video recording to be transcoded should be in some standard (non-proprietary) format, otherwise it cannot be read. We might take "standard" to mean "playable on Windows Media Player, or failing that, on some more powerful generic media player such as VLC or MPlayer".

Return to top of page

What is it?

The Cobalt Engine is a software suite that runs under the Windows operating system. The "Professional" versions of these are recommended if the system is to be used as part of a network of computers acquiring and handling video.

The Cobalt Engine configures and runs video analysis on a variety of video sources:

Live video sources

These are IP video cameras and USB Webcams. The IP camera must support the RTSP streaming protocol. If the camera provides only one channel, that is devoted to MJPEG acquisition. If it is multi-channel the additional channels can be dedicated to real time video streaming and recording.
Live video is presented to the Cobalt Engine using streaming software. In particular, Cobalt supports Open Source programs such as MPlayer and VLC.

Recorded video

The video source in this case is a pre-recorded video file such as an MP4, FLV or MPEG-2 recording.
Access to the archive, if it is not in the MJPEG format, is via a stream transcoder. Cobalt supports Open Source programs such as MPlayer and VLC for this purpose.

Cobalt archive

During video acquisition, Cobalt can analyse the incoming stream in its entirety and store metadata associated with the stream as a reference to the recording. The metadata archive can be queried and the results of the query recovered from the recording.

DVD video

These are recordings made to DVD, usually in MPEG-2 format VOB files. They often have audio.

There is a Cobalt interface called "Cobalt MPViewer" that provides a simple user interface with which to access these video sources. Alternatively, the engine can be run via a command line. Information from the Engine is made generally available to other applications via messages sent using TCP/IP. The message format is an open protocol described later.

In particular the information from the engine can be sent to a Cobalt Server. This is a separate version of the Engine that can manage data from a distributed system and pass information to other applications. The Cobalt Server is in effect a classic Application Server and can be used to drive an SQL database.

Return to top of page

What does it do?

The Cobalt Engine has the following modes of operation:

Live Mode

The Cobalt Engine is fed with live video streams which are analysed in real time. Events defined by the user are sent to an event handling console, a VMS system, or transmitted to remote monitoring sites.

Recording Mode

The live video streams are analysed and stored locally. The result of the analysis of the scene is stored as metadata which can be accessed for query. Access can be done remotely and while the system is recording.

In Recording Mode the Cobalt Engine can either analyse or transcoder the recorded data:

Analyse

Analyses video and generates video metadata for later query. The metadata analysis is for the entire scene so subsequent queries can make queries of different areas of the camera view.

Transcode

Convert all or part of video to another format. The conversion is not done by the Cobalt Engine itself, but by the RTSP acquisition sub-system.

When the input source is an IP camera, both modes can run simultaneously if the camera delivers multiple RTSP streams. However, this comes with the inevitable cost of a greater demand on CPU and memory resources.

Return to top of page

Acquiring video streams

The Cobalt Engine handles an input stream in the form of JPEG files. Any incoming live video stream or recorded video has to be converted into a sequence of JPEGS before being fed into the Cobalt Engine. If the files are not already available in that format they must be transcoded from their native format. There are a number of software programs that can do this, notably the Open Source products MPlayer and VLC and programs derived from these, such as SMplayer or GOM.

Video that is streamed from a camera can be converted on the fly using such programs. The JPEGS are analysed and deleted as they pass through the Cobalt Engine.

Multi-stream acquisition

The Cobalt Engine can handle one stream at a time. So to handle multiple streams it is necessary to run multiple copies of the Cobalt Engine, each handling its own stream. The number of streams that can be simultaneously handled by a single computer depends on the CPU power and memory available. A typical Intel Atom powered Netbook can certainly handle one or two streams (depending on the delivered data rate), while a "Sandy Bridge i7" can typically handle 8 inputs, or even 16 inputs if the processing demands are light.

Streams from different computers can be merged and managed using the Cobalt Server. Hence a distributed Cobalt system can handle very large numbers of cameras and other data streams.

Cobalt Processing

Cobalt processes image sequences through a series of filters, most, but not all of which, are user selectable. There are three stages to the filtering: per-processing, video analysis, post-processing. There is a further, application related, stage which comes in the form of enhancements or add-ons to the Cobalt engine. These are not delivered as part of the standard package.

Pre-processing

Video Analysis

Post-processing

Noise removal
Smoothing
Edge generation
Activity masks
Undersampling
Colour plane selection
Camera shake
Frame skip
Snapshots

Video Event Detection
Non-motion detection
Morphology analysis
Strobing
Tomographic

Hitrate control
Direction
Size
Colour
Edge check
Memory

Return to top of page

These are discussed individually following on from the discussion of the user interface.

The "Standard" Cobalt Engine implements only the noise pre-filter and the Video Event Detection module, neither of which is under user control. While this is a relatively naive approach to event detection it is nonetheless, under most circumstances, highly efficient and extremely easy to set up. This is the basis of what is referred to as the Cobalt "Zero Training" delivery.

In the "Expert" Cobalt Engine all available filters come under user control, with the exception of noise removal during the pre-processing stage. This is done automatically by a series of internal filters that are specially designed to handle artifacts in JPEG and MJPEG encoded images during low light conditions.

Enhancements

There are two kinds of optional filter which provide enhancements to the system functionality: Plug-ins and Applications.

Plug-ins are enhancements to the video stream that require no special user interface modifications. Any changes to the user interface required by a plug-in is automatically handled by Cobalt. So, for example, the expert mode enhancement of adding control zones to the detection is a part of the user interface to set up Cobalt detection. Other plug-ins, like the ability to save video snapshots at specified times do not require a special user interface: the snapshots are stored locally as JPEG files that can be picked up by any JPEG file viewer.

Applications are programmes in their own right with their own application specific user interface. The applications merely use the Cobalt engine for video analysis and set the results of the analysis within a project specific context. Examples of such applications would be Crowd Monitoring and Vehicle Inspection.

These require substantial user interfaces and substantial system integration. Since Cobalt is an open system, enough data is available to those who wish to write their own applications or interface Cobalt with an existing application.

Client-Server

The Cobalt Engine application provides full network support for both input and output functions. Each video acquisition computer must run one or more client instances of the Cobalt Engine: one is required per camera. Event Viewers or Video Management Systems (VMS) can connect to the output ports of the Clients and receive data about the video streams being analysed.

Hence it is straightforward to implement an event handling console that runs on an individual acquisition computer and that handles all inputs from the cameras connected to that computer.

The outputs from multiple acquisition computers can be sent to a single computer running the Cobalt Engine in Server mode. This acts as an application server and can relay all messages from the clients to a single user interface, a VMS, or an SQL database.

Return to top of page

System Architecture I:

Let us look into a Cobalt computer to see what software is running and what it does. The sketch below shows the various components that are working in and around the Cobalt Acquisition Computer. Each element has been assigned a letter to simplify the following discussion. The symbols Cobalt blue ellipses represent Cobalt processes running in the computer, while thedark green disk-like shapes represent important places where data is stored. All data except the H.264 storage (G) is due to Cobalt. The H.264 storage is done by the Camera (A) or by the VMS system (B).

Cobalt Engine Internals - what is saved


The components A, B, D and E are external to the Cobalt System running on the Cobalt acquisition computer, C. The video can be provided to the acquisition computer, C, either by live cameras or supplied from a VMS system.

Many of the major Video Management Systems (VMS's) take control of the cameras and provide third party users with video buffers from which they can take their live streams. If the site has a VMS then the results of the Cobalt analysis can be viewed within the VMS context provided the VMS interfacing to Cobalt has been implemented.

Return to top of page

Item Function Comments
A Input video streams: IP cameras In the absence of a VMS system, cameras should support RTSP and have at least two streams if H.264 DVR recording is required.
B Connection to VMS system Many VMS systems lock the cameras and provide applications with the video streams.
C Network switch If there is a connection to outside of the local area this should embody a hardware firewall and set access rights.
D Firewall Hardware firewall, could be part of the router C
E The Internet Communication to the outside world and remote sites and control rooms. Must be a VPN connection
F Cobalt Acquisition Computer Runs Windows "Professional". Both 32-bit and 64-bit versions are supported.
G H.264 DVR storage Comes from the camera if H.264 DVR recording required. Storage quality and format set in camera (A)
H MPviewer A Cobalt H.264 viewer based on the MPlayer media player., which comes under the GPL license. Details of the license are given elsewhere here and on the distribution media
I MJPEG temporary storage Comes from the camera if video analysis required. Image quality and format set in the camera, but may be changed internally by Cobalt.
J Cobalt VMS Modest VMS system to display and handles events. Shows live video streams and handles replay of H.264 recording via the MPviewer (H).
K Transcoder A Cobalt program to generate appropriately sized MJPEG streams for low bandwidth connections (3G). The transcoder also generates a recorded video archive in the ICE formats.
L Cobalt Analyser Analyses the incoming MJPEG stream without prejudice as regards areas of interest. Tests for alarms as defined by user configuration of the scene using baseline detection algorithms or user selected enhanced detection.
M Video data transmission Transmits remotely specified video data. This could be either recorded DVR data, ongoing alarms data, or archival alarms. The transmission adjusts to bandwidth limitations (3G).
N ICE recording repository Generated by the transcoder, this is an alternative to H.264 that is more compact and that allows rapid search on metadata criteria. Can generate time-lapse DVR recording.
O Analysis repository The outcome of the analysis is a searchable database of image metadata. This is stored and works in conjunction with both the ICE recording repository (N) and the H.264 DVR recording (G).
P Event archive The archive of events generated in accordance with the user settings of the Cobalt analyser (L).

The table presents a brief commentary on all of these components, some or all of which may be present in a Cobalt based system.

Return to top of page

System Architecture II:

The way a Cobalt system is put together can best be described in a diagram showing the hierarchical structure of the elements that make up a deliverable application.

Cobalt engine hierarchy
The table shows the layered architecture of the Cobalt system starting at the bottom with the hardware, operating system and inputs, going to the top with the specific applications and their user interfaces.

The above scheme illustrates, at a detailed technical level, the way in which the Cobalt engine fits into an application that is an end user installation. As can be appreciated, this is quite complex.

The left hand column shows which functions are provided by each row, or set of rows, in the table. The right hand column shows the relevance of this to the end user and delineates which functions have to be undertaken by the installer versus the functions used by the system operator(s). Of course, the installer and system operator may be one and the same person or group.

The parts in the darkest shades of cobalt blue are the basic functionality provided by the Cobalt engine. The lighter shades of cobalt blue are expert level enhancements in the video analysis methodology that may be required to solve specific problems or to handle especially difficult situations.

It is up to the installer or user to provide and test a properly configured video source. This is particularly the case if the video source comes from a third party VMS system when the installer must already understand how to provide an MJPEG video stream. Likewise, the data from Cobalt is made available to all applications and if that data is to be interfaced to a third party video management system, the installer must already understand how to do this.

Cobalt comes with a pre-packaged event viewer and a modest video management system.

Return to top of page