Network based real-time analytics of raw video streams

• US 8,941,735 B2
• Filed: 09/14/2012
• Issued: 01/27/2015
• Est. Priority Date: 07/25/2008
• Status: Expired due to Fees

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Mariadoss, Pandian
• Primary Examiner(s): YI, DAVID
• Application Number:
  US13/618,154
• Publication Number:
  US 20130271598A1
• Time in Patent Office: 865 Days
• Field of Search: 348/143
• US Class Current: 348/143
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19669 Event triggers storage or change of storage policy
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/443 OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
• CPC Class Codes: H04N 21/4431 characterized by the use of Application Program Interface [API] libraries
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/6175 involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/814 comprising emergency warnings arrangements specially adapted for emergency or urgency in broadcast systems H04H20/59; arrangements for providing alarms, notifications, alerts to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/8352 : involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A video stream is received over a network from a remotely located video capture device. The video stream is processed in real-time or near real time. The processing of the video stream extracts at least one metric from content of the video stream. The extracted metric is compared against at least one previously established value to generate either a TRUE or a FALSE result. Responsive to a TRUE result, at least one programmatic action is automatically initiated. The previously established value, the programmatic action, or both are user configurable. The programmatic action is a real time or a near real time action resulting from analyzing content of the video stream. Responsive to a FALSE comparison result, the at least one programmatic action is not automatically initiated.

Claims

1. A method comprising:

one or more computers receiving a video stream over a network from a remotely located video capture device;

at least one of the one or more computers processing the video stream in real-time or near real time, wherein the processing of the video stream extracts at least one metric from content of the video stream, wherein the one or more computers process the video stream using an application program interface (API), wherein the processing comprises decoding the video stream from the remotely located video capture device, and wherein the application program interface is a filter specific to the remotely located video capture device, wherein different types, versions, and varieties of video capture devices are associated with different filters or APIs, wherein the processing occurs using an analytics engine that is a component of an intelligent video surveillance system that is unable to process the video stream in absence of the application program interface;

at least one of the one or more computers comparing the extracted metric against at least one previously established value to generate either a TRUE or a FALSE result;

responsive to a TRUE result, at least one of the one or more computers automatically initiating at least one programmatic action, which comprises a notification action that conveys a notification to at least one previously designated user when a previously established condition is automatically detected by the analytics engine, wherein the previously established value, the programmatic action, or both are user configurable, and wherein the programmatic action is a real time or a near real time action resulting from analyzing content of the video stream; and

responsive to a FALSE comparison result, the one or more computers not initiating the at least one programmatic action.

2. The method of claim 1, wherein the video stream captures input from a geospatial region, wherein an authorized entity is able to configure the established value, the programmatic action, or both is has responsibility and authority over the geospatial region.

3. The method of claim 1, wherein the video capture device is a network connected IP camera.

4. The method of claim 1, wherein the received video stream is raw, unprocessed video captured by a video capture device.

5. The method of claim 1, further comprising:

at least one of the one or more computers determining a processing filter for the video capture device from a plurality of different processing filters, each of the different processing filters being mapped to different video capture devices or to different types of video capture devices, wherein the processing uses the determined processing filter.

6. The method of claim 1, wherein said at least one programmatic action further comprises is at least one of an alert and an action altering the state of a localized entity.

7. The method of claim 1, wherein the processing of the video stream and the comparing of the extracted metric are performed as part of an object tracking analysis of the video stream performed by the one or more computers.

8. The method of claim 1, Wherein the processing of video stream and the comparing of the extracted metric are performed as part of a face recognition analysis of the video stream performed by the one or more computers.

9. The method of claim 1, wherein the processing of the video stream and the comparing of the extracted metric are performed as part of a motion detection analysis of the video stream performed by the one or more computers.

10. The method of claim 1, wherein the one or more computers are part of a middleware implemented solution for processing received video streams.

11. The method of claim 1, wherein the received video stream is a raw, real-time video stream from a network connected IP camera, which is the remotely located video capture device.

12. The method of claim 1, wherein the comparing is performed as part of one of more of:

a face recognition process, an object tracking process, and a motion detection process.

13. The method of claim 1, wherein the initiated programmatic action conveys an alert state, an alert, or the notification to an entity owning or controlling the geographic region from which the video stream was captured.

14. The method of claim 1, wherein the processing of the video stream comprises:

exporting metadata from the video stream; and

indexing the metadata within a database system.

15. A computer program product comprising:

a non-transitory computer usable storage medium storing computer usable program code;

computer usable program code being executable by at least one processor to receive a video stream over a network from a remotely located video capture device;

computer usable program code being executable by at least one processor to process the video stream in real-time or near real time, wherein the processing of the video stream extracts at least one metric from content of the video stream, wherein the processing of the video stream using an application program interface (API), wherein the processing comprises decoding the video stream from the remotely located video capture device, and wherein the application program interface is a filter specific to the remotely located video capture device, wherein different types, versions, and varieties of video capture devices are associated with different filters or APIs, wherein the processing occurs using an analytics engine that is a component of an intelligent video surveillance system that is unable to process the video stream in absence of the application program interface;

computer usable program code being executable by at least one processor to compare the extracted metric against at least one previously established value to generate either a TRUE or a FALSE result;

computer usable program code being executable by at least one processor to, responsive to a TRUE result, automatically initiate at least one programmatic action, which comprises a notification action that conveys a notification to at least one previously designated user when a previously established condition is automatically detected by the analytics engine, wherein the previously established value, the programmatic action, or both are user configurable, and wherein the programmatic action is a real time or a near real time action resulting from analyzing content of the video stream; and

computer usable program code being executable by at least one processor to, responsive to a FALSE comparison result, not initiate the at least one programmatic action.

16. The computer program product of claim 15, wherein the video stream captures input from a geospatial region, wherein an authorized entity is able to configure the established value, the programmatic action, or both is has responsibility and authority over the geospatial region.

17. A system for performing video analytics comprising:

a video surveillance system, comprising one or more processors for ingesting video streams from a remotely located video capture device, wherein said video surveillance system performs real-time or near-real time video analytics on the video stream;

a filter specific to the remotely located video capture device that is an application programming interface (API), wherein the video surveillance system is unable to process the video stream received from the video capture device in absence of the application program interface; and

a software engine, comprising program instructions stored in at least one non-transitory medium, said program instructions able to be executed by at least one processor to perform at least one programmatic action responsive to results produced by an analytics engine that performs the real-time analytics, wherein at least one action that the software engine is configured to automatically perform is a notification action that conveys a notification to at least one previously designated user when a previously established condition is automatically detected by the analytics engine, wherein the analytics engine;

compares the extracted metric against at least one previously established value to generate either a TRUE or a FALSE result;

responsive to a TRUE result, automatically initiates the at least one programmatic action, wherein the previously established value, the programmatic action, or both are user configurable, and wherein the programmatic action is a real time or a near real time action resulting from analyzing content of the video stream; and

responsive to a FALSE comparison result, does not initiate the at least one programmatic action.