Visual similarity for video content

• US 9,489,577 B2
• Filed: 07/23/2010
• Issued: 11/08/2016
• Est. Priority Date: 07/27/2009
• Status: Active Grant

• Current Assignee: Cxense ASA
• Original Assignee: Cxense ASA
• Inventors: Wilde, Thomas
• Primary Examiner(s): He, Jialong
• Application Number:
  US12/804,518
• Publication Number:
  US 20110022394A1
• Time in Patent Office: 2,300 Days
• Field of Search: 704/270, 704/251, 704/278, 704/235, 725/40, 725/45, 725/46
• US Class Current: 1/1
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Methods and apparatus, including computer program products, for visual similarity. A method includes receiving a stream of video content, generating interpretations of the received video content using speech/natural language processing (NLP), associating the interpretations of the received video content with images extracted from video content based on timeline, and using the interpretations to obtain interpretations of other images or other video content.

Claims

1. A method comprising:

receiving a stream of video content;

generating speech to text for the received video content;

generating passage level annotations from the generated text using natural language processing (NLP);

associating the passage level annotations with a timeline; and

associating imagery with the text to generate thumbnails at periodic time intervals resulting in a database of annotations to imagery and imagery to annotations.

2. The method of claim 1 further comprising using database of annotations to imagery and imagery to annotations to obtain interpretations of other digital images to improve the interpretation derived from the speech.

3. The method of claim 1 wherein the annotations comprise names, concepts, and/or logical substructure.

4. The method of claim 1 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals comprises applying a machine learning technique.

5. The method of claim 4 wherein the machine learning technique includes one of a neural networks approach or a classification and regression tree (CART) approach to classify an image into an annotation or to classify an annotation into an image.

6. The method of claim 4 wherein the machine learning technique is a principal component analysis (PCA) of facial imagery.

7. The method of claim 4 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals further comprises clustering the images.

8. The method of claim 1 further comprising generating a master catalog of images for selected annotations.

9. The method of claim 8 further comprising comparing videos to determine whether the videos contain speech.

10. The method of claim 9 wherein comparing videos comprises pixel comparison.

11. An apparatus comprising:

a local computing system linked to a network of interconnected computer systems, the local computing system comprising a processor, a memory and a storage device;

the memory comprising an operating system and a visual similarity process, the visual similarity process comprising;

receiving a stream of video content;

generating speech to text for the received video content;

generating passage level annotations from the generated text using natural language processing (NLP);

associating the passage level annotations with the text time aligned to result in text, annotations and a time stamp; and

associating imagery with the annotated text to generate thumbnails at periodic time intervals resulting in a database of annotations to imagery and imagery to annotations.

12. The apparatus of claim 11 wherein the annotations comprise names, concepts, and/or logical substructure.

13. The apparatus of claim 11 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals comprises applying a machine learning technique.

14. The apparatus of claim 13 wherein the machine learning technique includes one of a neural networks approach or a classification and regression tree (CART) approach to classify an image into an annotation or to classify an annotation into an image.

15. The apparatus of claim 13 wherein the machine learning technique is a principal component analysis (PCA) of facial imagery.

16. The apparatus of claim 11 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals further comprises clustering the images.

17. The apparatus of claim 11 wherein the visual similarity process further comprises generating a master catalog of images for selected annotations.

18. The apparatus of claim 17 wherein the visual similarity process further comprises comparing videos to determine whether the videos contain speech.

19. The apparatus of claim 18 wherein comparing videos comprises pixel comparison.