Event triggered location based participatory surveillance

• US 9,740,940 B2
• Filed: 03/12/2013
• Issued: 08/22/2017
• Est. Priority Date: 03/26/2012
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Chattopadhyay, Dhiman, Pal, Arpan, Banerjee, Rohan, Dasgupta, Ranjan, Sinha, Priyanka, Sinha, Aniruddha, Bhaumik, Chirabrata
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US14/386,518
• Publication Number:
  US 20150070506A1
• Time in Patent Office: 1,624 Days
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 25/08 using communication transmission lines G08B13/19658, G08B21/0286, G08B25/016 take precedence
• CPC Class Codes: G08B 27/006 with transmission via telephone network
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention provides the multimodality filtration surveillance comprising of a plurality of filtration stages executed at the backend server to confirm nature of anomaly in an event, the filtration stages comprising: a first filter of a video anomalies detection in the event for a specified time-place value, a second filter of a city soundscape adapted to provide a localized decibel maps of a city, a third filter of a geocoded social network adapted to semantically read and analyze data from one or more social media corresponding to the specified time-place value, and a fourth filter of an event triggered or proactive local participatory surveillance adapted to provide augmented information on the detected anomalies.

Claims

1. A system for surveillance of distributed geographical sites for a plurality of event, each event associated with at least one predefined characteristics and accessible to a plurality of event subscribers, the system comprises:

a plurality of sensing device flexibly coupled to a plurality of geographically distributed site;

a sensory information processing unit electrically coupled with the each sensing device and a combination thereof, the sensory information processing unit comprising;

a video analytics module configured to identify and index the events in the captured sensory information and generate a cumulative metadata therefor;

a plurality of participatory surveillance tool comprising mobile data gathering and communication devices, each participatory tool being selectively identified based on the event metadata and configured to share gathered sensory information corresponding to the said event, wherein the plurality of participatory surveillance tool comprises a proactive and an event triggered surveillance information input applications installed thereon the sensory devices, social network posts referring to a geo-tagged social network data, video and sound recordings received from mobile devices and text/voice call recordings from the plurality of local sites;

a backend analytics server, coupled with a messaging middleware and a central storage, is configured to perform a multimodality filtration surveillance for anomaly detection in the events; and

a search module adapted for a real threat detection comprising a temporal search for identification of a distinct spacio-temporal character the each event and a geo-spatial search providing a color graded heat-map illustrating geographical distribution of potential security events along with timestamp.

2. The system of claim 1, wherein at an instance during surveillance period, each sensing device is configured to continuously capture audio-visuals of at least one site and transmit the captured sensory input over a local network to a gateway device.

3. The system of claim 1, wherein the sensory information processing unit is operably coupled with a local storage and a local web server, the local storage is adapted to compress and store the captured audio-visuals, indexed events and the corresponding metadata thereof, wherein the local storage is further coupled with an event publishing tool configured to asynchronously transmit each recorded event information to a messaging middleware, wherein the messaging middleware is further configured to receive a plurality of event subscription request from the event subscribers.

4. The system of claim 1, wherein a central storage exist along with a central web server and is configured to store the sensory information and a consolidated event indexes and corresponding metadata thereof from the plurality of local storages connected therewith.

5. The system of claim 1, wherein an event rendering module, upon selection of the event, is configured to asynchronously publish the each selected event onto a user dashboard and intuitively suggest and rank a plurality of threat prone events for user subscription.

6. The system according to claim 1, wherein the relevance parameters for selection of the participating tools comprise of relative proximity of the tool to a recorded geospatial location of the event and time of occurrence of the event.

7. The system according to claim 1, wherein the multimodality filtration surveillance module comprises of a plurality of filtration stages executed at the backend server to confirm nature of anomaly in an event, the multimodality filtration surveillance module further comprising:

a module for first filter of a video anomalies detection in the event for a specified time-place value, a module for second filter of a city soundscape adapted to provide a localized decibel maps of a city visualized as different color hues indicating different classes of sound, a module for third filter of a geocoded social network adapted to semantically read and analyze data from one or more social media corresponding to the specified time-place value, and a module for fourth filter of an event triggered local participatory surveillance adapted to provide augmented information on the detected anomalies.

8. The system according claim 1, wherein the sensory inputs of one or more surveillance cameras are coupled with at least one social networking application, the said sensory input comprising a spatiotemporal data having attributes and is adapted to maintain an implicit coherence in semantics to preserve an ambient context with other sensory inputs, the coupled surveillance cameras forming a social network among themselves based on one or more common intents.

9. The system according to claim 1, wherein an impact of the each participatory surveillance tool is classified through audio analysis and natural language processing and is represented in a heat map on top of a Geographical Information System (GIS) map.

10. The system according to claim 1, wherein the each geographical site under surveillance is pre coded and the information emanating therefrom is semantically analyzed.

11. The system according to claim 1, wherein the backend analytics server is further configured to execute the spacio-temporal and geo-spatial characteristics search on the data mining and business intelligence tools to extract consolidated rich information about the each selected event identified through a multimodality filtration of a plurality of participatory surveillance tools, surveillance cameras, and audio recordings.

12. The system according to claim 1, wherein the backend analytics server is further comprise of a data reduction module configured to split a continuous video streams into a plurality of particular video clips, each clip corresponding to one particular event selected by the event subscriber enabling a fast retrieval of the selected individual event.

13. The system according to claim 1, further comprising an Image indexing module configured to stitch each detected particular event and form a chronological sequence video of the particular events.

14. A system for an event triggered participatory surveillance, the system comprising:

an event metadata generation module coupled with a plurality of sensory devices located at a distributed sites in a predefined geography;

a plurality of participatory surveillance tool flexibly located at arbitrary sites, each participatory tool adapted to gather sensory data and communicate;

a participating tool selector module configured to select at least one participating tool on at least one relevance parameter; and

a multimodality filtration surveillance module installed at a backend analytics server, is configured for a plurality of modality based filtration of the gathered sensory information for anomaly detection in the events.

15. A method for surveillance of distributed geographical sites for a plurality of event, each event associated with at least one predefined characteristics and accessible to a plurality of event subscribers, the method comprises:

coupling flexibly a plurality of sensing devices to a plurality of geographically distributed sites to capture a sensory information from the each site;

processing the captured sensory information to index the each event and generate a cumulative metadata therefor;

compressing and storing the captured sensory information, indexed events and the corresponding metadata thereof in a local storage;

storing the sensory information and metadata of the each indexed event from the plurality of local storages into a central storage;

performing a multimodality filtration surveillance for anomaly detection in the events by a backend analytics server, coupled with a messaging middleware, a central storage and a plurality of participatory surveillance tool;

searching spacio-temporal and geo-spatial characters of the events identifying a distinct temporal occurrence of events and a geographically distributed color graded heat-map of the city sounds; and

wherein the plurality of participatory surveillance tool comprises;

a proactive and an event triggered surveillance information input applications installed thereon the sensory devices, social network posts referring to a geo-tagged social network data, video and sound recordings received from mobile devices and text/voice call recordings from the plurality of local sites.

16. The method of claim 15, wherein the backend analytics server is coupled with the messaging middleware, the central storage and the plurality of participatory surveillance tools for distinctively filtering and identifying an event from at least one source thereof, each multimodality filtration offering a greater confidence score of real threat detection resulting in reduced false alarms.