Systems and methods for creating a story board with forensic video analysis on a video repository

• US 10,311,305 B2
• Filed: 03/20/2017
• Issued: 06/04/2019
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Janakiraman, Kirupakar, KandhaMunisamy, Naga Sundar, Sankarapandian, Rajesh Kulandaivel, Arumaikani, Ramesh, Patel, Keyurbhai, Muthusamy, Baskaran
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/463,208
• Publication Number:
  US 20180268223A1
• Time in Patent Office: 806 Days
• Field of Search: None
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel

Abstract

Systems and methods for creating a story board with forensic video analysis on a video repository are provided. Some methods can include storing a plurality of video data streams in a data repository, storing asynchronous streams of metadata of each of the plurality of video data streams in the data repository, identifying a first object captured by at least one of the plurality of video data streams, using the asynchronous streams of metadata to identify correlations or interactions between the first object and a plurality of other objects over time, and replicating a story of the first object.

Claims

1. A method comprising:

storing a plurality of video data streams of a monitored region in a data repository;

storing respective asynchronous streams of metadata of each of the plurality of video data streams in the data repository;

identifying a first physical object present in the monitored region and depicted in at least one of the plurality of video data streams;

using the respective asynchronous streams of metadata of each of the plurality of video data streams to identify correlations or interactions between the first physical object and a plurality of other physical objects over time, the plurality of other physical objects being present in the monitored region and depicted in at least some of the plurality of video streams; and

replicating a story of the first physical object using the correlations or interactions between the first physical object and the plurality of other physical objects over time.

2. The method of claim 1 wherein the first physical object includes a person or a structure in the monitored region.

3. The method of claim 1 further comprising using the respective asynchronous streams of metadata of each of the plurality of video data streams to classify, detect, identify, locate, or track the first physical object or to identify a relationship between the first physical object and the plurality of other physical objects.

4. The method of claim 1 further comprising refining the respective asynchronous streams of metadata of each of the plurality of video data streams stored in the data repository based on the correlations or the interactions between the first physical object and the plurality of other physical objects identified over time.

5. The method of claim 1 wherein replicating the story of the first physical object includes generating an incident management report on top of the respective asynchronous streams of metadata of each of the plurality of video data streams, and wherein the incident management report summarizes the correlations or the interactions between the first physical object and the plurality of other physical objects over time.

6. The method of claim 1 wherein replicating the story of the first physical object includes generating a storyboard on top of the respective asynchronous streams of metadata of each of the plurality of video data streams, and wherein the storyboard summarizes the correlations of the interactions between the first physical object and the plurality of other physical objects over time.

7. The method of claim 6 wherein the storyboard orders the plurality of video data streams to present an intelligent view of the story of the first physical object irrespective of an order of the plurality of video data streams in the data repository.

8. The method of claim 6 further comprising streaming the respective asynchronous streams of metadata of each of the plurality of video data streams to a user interface device displaying the storyboard.

9. A method comprising:

storing a plurality of video data streams of a monitored region in a data repository;

storing respective metadata of each of the plurality of video data streams in the data repository;

a classification module asynchronously analyzing the plurality of video data streams and the respective metadata of each of the plurality of video data streams to classify physical objects present in the monitored region and depicted in at least some of the plurality of video data streams;

an interaction module asynchronously analyzing the plurality of video data streams and the respective metadata of each of the plurality of video data streams to classify interactions of the physical objects;

a relationship builder asynchronously analyzing the plurality of video data streams and the respective metadata of each of the plurality of video data streams to build relationships of the physical objects; and

based on the physical objects as classified, the interactions of the physical objects, and the relationships of the physical objects, presenting a story of at least one of the physical objects, at least one of the interactions of the at least one of the physical objects, or at least one of the relationships of the at least one of the physical objects.

10. The method of claim 9 further comprising:

based on the physical objects as classified, the interactions of the physical objects, and the relationships of the physical objects, identifying the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, the at least one of the relationships of the at least one of the physical objects, or an event sequence of the at least one of the physical objects as suspicious; and

presenting the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, the at least one of the relationships of the at least one of the physical objects, or the event sequence of the at least one of the physical objects identified as suspicious.

11. The method of claim 9 wherein the physical objects include a person or a structure in the monitored region.

12. The method of claim 9 further comprising refining the respective metadata of each of the plurality of video data streams stored in the data repository based on the physical objects as classified, the interactions of the physical objects, and the relationships of the physical objects identified over time.

13. The method of claim 9 wherein presenting the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, or the at least one of the relationships of the at least one of the physical objects includes generating an incident management report on top of the respective metadata of each of the plurality of video data streams, and wherein the incident management report summarizes the at least one of the physical objects as classified, the at least one of the interactions of the at least one of the physical objects, and or the at least one of the relationships of the at least one of the physical objects.

14. The method of claim 9 wherein presenting the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, or the at least one of the relationships of the at least one of the physical objects includes generating a storyboard on top of the respective metadata of each of the plurality of video data streams, and wherein the storyboard summarizes the at least one of the physical objects as classified, the at least one of the interactions of the at least one of the physical objects, and the at least one of the relationships of the at least one of the physical objects.

15. The method of claim 14 wherein the storyboard orders the plurality of video data streams to present an intelligent view of the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, or the at least one of the relationships of the at least one of the physical objects irrespective of an order of the plurality of video data streams in the data repository.

16. The method of claim 14 further comprising streaming the respective metadata of each of the plurality of video data streams to a user interface device displaying the storyboard.

17. A method comprising:

storing a plurality of video data streams of a monitored region in a data repository;

storing respective metadata of each of the plurality of video data streams in the data repository;

a classification module asynchronously analyzing the respective metadata of each of the plurality of video data streams to classify physical objects present in the monitored region and depicted in at least some of the plurality of video data streams;

an interaction module asynchronously analyzing the respective metadata of each of the plurality of video data streams to classify interactions of the physical objects;

a relationship builder asynchronously analyzing the respective metadata of each of the plurality of video data streams to build relationships of the physical objects; and

based on the physical objects as classified, the interactions of the physical objects, and the relationships of the physical objects, presenting a story of at least one of the physical objects, at least one of the interactions of the at least one of the physical objects, or at least one of the relationships of the at least one of the physical objects.

18. The method of claim 17 further comprising refining the respective metadata of each of the plurality of video data streams stored in the data repository based on the physical objects as classified, the interactions of the physical objects, and the relationships of the physical objects identified over time.

19. The method of claim 17 wherein presenting the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, or the at least one of the relationships of the at least one of the physical objects includes generating an incident management report on top of the respective metadata of each of the plurality of video data streams, and wherein the incident management report summarizes the at least one of the physical objects as classified, the at least one of the interactions of the at least one of the physical objects, and the at least one of the relationships of the at least one of the physical objects.

20. The method of claim 17 wherein presenting the story of the at least one of the physical objects, the at least one of the interactions of the at least one of the physical objects, or the at least one of the relationships of the at least one of the physical objects includes generating a storyboard on top of the respective metadata of each of the plurality of video data streams, and wherein the storyboard summarizes the at least one of the physical objects as classified, the at least one of the interactions of the at least one of the physical objects, and the at least one of the relationships of the at least one of the physical objects.