System and method for generating semantic annotations

• US 9,251,421 B2
• Filed: 06/15/2013
• Issued: 02/02/2016
• Est. Priority Date: 09/13/2012
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Tu, Peter Henry, Rittscher, Jens, Guan, Li
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US13/918,905
• Publication Number:
  US 20140072171A1
• Time in Patent Office: 962 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

In accordance with one aspect of the present technique, a method is disclosed. The method includes receiving a new video from one or more sensors and generating a new content graph (CG) based on the new video. The method also includes comparing the new CG with a plurality of prior CGs. The method further includes identifying a first portion of the new CG matching a portion of a first prior CG and a second portion of the new CG matching a portion of the second prior CG. The method further includes analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG. The method further includes generating a sequence of SAs for the new video based on the analysis of the first and the second set of SAs.

Claims

1. A method, comprising:

receiving a new video from one or more sensors;

generating a new content graph (CG) based on the new video;

comparing the new CG with a plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos;

identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs;

analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG; and

generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

2. The method of claim 1, wherein the new CG includes a plurality of nodes interconnected via one or more edges.

3. The method of claim 2, wherein the plurality of nodes represent at least one of a plurality of objects and a plurality of activities in the new video.

4. The method of claim 2, wherein the one or more one or more edges represent at least one of a spatial relationship, a temporal relationship, and a dynamic relationship between the plurality of nodes.

5. The method of claim 2, wherein identifying the portion of the first prior CG further comprises determining at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

6. The method of claim 1, wherein analyzing the first and the second set of SAs further comprises analyzing statistical data associated with the first and the second set of SAs.

7. The method of claim 1, further comprising:

determining whether the sequence of SAs satisfies a notification category; and

sending a notification in response to determining that the sequence of SAs satisfies the notification category.

8. The method of claim 7, wherein the notification category includes at least one of safety, theft, vandalism, and business opportunity.

9. A system, comprising:

at least one processor;

a graph module stored in a memory and executable by the at least one processor, the graph module configured for receiving a new video from one or more sensors and generating a new content graph (CG) based on the new video;

a comparison module stored in the memory and executable by the at least one processor, the comparison module communicatively coupled to the graph module for comparing the new CG with a plurality of prior CGs and identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos; and

a narrative module stored in the memory and executable by the at least one processor, the narrative module communicatively coupled to the comparison module for analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG and generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

10. The system of claim 9, wherein the new CG includes a plurality of nodes interconnected via one or more edges.

11. The system of claim 10, wherein the comparison module is further configured to determine at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

12. The system of claim 9, wherein the narrative module is further configured to analyze statistical data associated with the first and the second set of SAs.

13. The system of claim 9, further comprising a notification module for determining whether the sequence of SAs satisfies a notification category and sending a notification to a user in response to determining that the sequence of SAs satisfies the notification category.

14. A computer program product comprising a non-transitory computer readable medium encoded with instructions that, in response to execution by at least one processor, cause the processor to perform operations comprising:

receiving a new video from one or more sensors;

generating a new content graph (CG) based on the new video;

comparing the new CG with a plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos;

identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs;

analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG; and

generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

15. The computer program product of claim 14, wherein the new CG includes a plurality of nodes interconnected via one or more edges.

16. The computer program product of claim 15, wherein the plurality of nodes represent at least one of a plurality of objects and a plurality of activities in the new video.

17. The computer program product of claim 15, wherein the one or more one or more edges represent at least one of a spatial relationship, a temporal relationship, and a dynamic relationship between the plurality of nodes.

18. The computer program product of claim 14, further causing the processor to perform operations comprising determining at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

19. The computer program product of claim 14, further causing the processor to perform operations comprising analyzing statistical data associated with the first and the second set of SAs.

20. The computer program product of claim 14, further causing the processor to perform operations comprising:

determining whether the sequence of SAs satisfies a notification category;

sending a notification to a user in response to determining that the sequence of SAs satisfies the notification category.