Performing real-time analytics using a network processing solution able to directly ingest IP camera video streams

• US 8,325,228 B2
• Filed: 07/25/2008
• Issued: 12/04/2012
• Est. Priority Date: 07/25/2008
• Status: Active Grant

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Mariadoss, Pandian
• Primary Examiner(s): Chan, Wing
• Assistant Examiner(s): YI, DAVID
• Application Number:
  US12/180,431
• Publication Number:
  US 20100020172A1
• Time in Patent Office: 1,593 Days
• Field of Search: 348/143
• US Class Current: 348/143
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19669 Event triggers storage or change of storage policy
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/443 OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention discloses a solution for an intelligent video surveillance (IVS) system able to perform real-time analytics on internet protocol (IP) video streams which are directly ingested by the IVS system. An internet protocol camera can be identified and connected to the intelligent video surveillance system which lacks an intermediate video management component. The raw video stream can be received directly from the camera in the form of a real-time video stream such as real-time streaming protocol (RTSP). One or more real-time analytics can be performed on said video stream based on one or more system criteria and user established criteria. When an incident event is detected one or more programmatic actions can be performed in response to analytics enacted.

Claims

1. A system for performing video analytics comprising:

an intelligent video surveillance system, comprising one or more processors for directly ingesting video streams from at least one network connected IP camera, wherein said intelligent video surveillance system is further configured to perform real-time video analytics on the directly ingested video stream;

a filter specific to the network connected IP camera that is an application programming interface (API) configured to decode the video stream from the network connected IP camera before conveying the decoded video stream to the intelligent video surveillance system, wherein the filter specific to the network connected IP camera is configured to receive the video streams from the at least one network connected IP camera over the network, wherein the at least one network connected IP camera comprises a plurality of network connected IP cameras geographically proximate to each other, wherein the intelligent video surveillance system is configured to perform the real time analytics on multiple video streams from different ones of the plurality of network connected IP cameras; and

a software engine, comprising program instructions stored in at least one non-transitory medium, said program instructions able to be executed by at least one processor to perform at least one programmatic action responsive to results produced by an analytics engine that performs the real-time analytics, wherein at least one action that the software engine is configured to perform is a notification action that conveys a notification to a previously designated user when a previously established condition is detected by the analytics engine.

2. The system of claim 1, wherein the real-time video analytics comprise at least one of face recognition, object tracking, and motion detection.

3. The system of claim 1, further comprising:

a plurality of camera modules stored on a storage medium, wherein each camera module comprises programmatic instructions permitting the intelligent video surveillance system to interface with a vender specific IP camera.

4. The system of claim 1, wherein the intelligent video surveillance system is configured to convey commands to the at least one network connected IP camera, where the commands result in an IP camera adjustment, which affects an aspect of the video stream captured by the IP camera.

5. The system of claim 1, wherein the intelligent video surveillance system provides at least one Web service that provides the real-time video analytics on the directed ingested video stream.

6. The system of claim 1, further comprising:

a non-transitory storage media configured to store profiles for entities responsible for a geographic region, wherein images of the geographic region are captured within the directly ingested video stream, wherein entity specific configuration settings are included in the stored profiles, wherein the entity specific configuration settings specify conditions to be detected by the analytics engine and a programmatic actions to be automatically performed when the specified conditions are detected, wherein each network connected IP camera is indexed within a database system against a stored profile.

14. The system of claim 1, wherein the intelligent video surveillance system is implemented in a middleware software platform.

15. The system of claim 1, further comprising:

a Middleware for Large Scale Surveillance (MILS) server to provide consolidation, management, and access for the analyzed video stream processed by the intelligent video surveillance system.

7. A method for intelligent video surveillance comprising:

identifying a raw, real-time video stream from a remotely located internet protocol camera connected to a network;

processing the raw real-time video stream using an application program interface, wherein the processing comprises decoding the raw real-time video stream from the network connected IP camera, and wherein the application program interface is a filter specific to the internet protocol camera;

conveying the processed video stream to an analytics engine, wherein the analytics engine is implemented in a middleware software platform; and

performing at least one video analytics programmatic action against the processed video stream.

comparing a metric extracted from the processed video stream responsive to the video analytics programmatic action against at least one previously established condition;

when comparing results match the metrics and the previously established condition, automatically initiating at least one resulting programmatic action, wherein the at least one previously established condition and the at least one resulting programmatic action are user configurable by an authorized entity, wherein said authorized entity is an entity having responsibility over a region captured within the processed video stream.

8. The method of claim 7, wherein the analytics engine is a component of an intelligent video surveillance system that is unable to process the raw real-time video stream in absence of the application program interface.

9. The method of claim 7, wherein the video analytics programmatic action comprises at least one of face recognition, object tracking, and motion detection.

10. The method of claim 7, wherein said programmatic action is at least one of a notification, an alert, and an action altering the state of a localized entity.

11. The method of claim 7, further comprising:

identifying a different raw, real time video stream from a different remotely located internet protocol camera connected to the network;

processing the different raw, real-time video stream using the application program interface;

conveying the different processed video stream to the analytics engine; and

performing at least one video analytics programmatic action against the processed video stream and the different processed video stream.

16. The method of claim 7, further comprising:

exporting metadata from the processed video stream; and

indexing the metadata within a database system.

12. A computer program product for intelligent video surveillance comprising:

a non-transitory computer usable medium having computer usable program code embodied therewith, the computer usable program code comprising;

computer usable program code configured to identify a raw, real-time video stream from a remotely located internet protocol camera connected to a network;

computer usable program code configured to process the raw real-time video stream using an application program interface, wherein the process comprises decoding the raw real-time video stream from the network connected IP camera, and wherein the application program interface is a filter specific to the internet protocol camera;

computer usable program code configured to convey the processed video stream to an analytics engine, wherein the application program interface is a filter specific to the internet protocol camera, wherein the analytics engine is implemented in a middleware software platform;

computer usable program code configured to perform at least one video analytics programmatic action against the processed video stream,

computer usable program code configured to compare a metric extracted from the processed video stream responsive to the video analytics programmatic action against at least one previously established condition; and

computer usable program code configured to, when comparing results match the metrics and the previously established condition, automatically initiating at least one resulting programmatic action, wherein the at least one previously established condition and the at least one resulting programmatic action are user configurable by an authorized entity, wherein said authorized entity is an entity having responsibility over a region captured within the processed video stream.

13. The computer program product of claim 12, wherein the analytics engine is a component of an intelligent video surveillance system that is unable to process the raw real-time video stream in absence of the application program interface.