Visual similarity

• US 20110022394A1
• Filed: 07/23/2010
• Published: 01/27/2011
• Est. Priority Date: 07/27/2009
• Status: Active Grant

• Current Assignee: Cxense ASA
• Original Assignee: Cxense ASA
• Inventors: Wide, Thomas
• Granted Patent: US 9,489,577 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Methods and apparatus, including computer program products, for visual similarity. A method includes receiving a stream of video content, generating interpretations of the received video content using speech/natural language processing (NLP), associating the interpretations of the received video content with images extracted from video content based on timeline, and using the interpretations to obtain interpretations of other images or other video content.

Claims

1. A method comprising:

receiving a stream of video content;

generating interpretations of the received video content using speech/natural language processing (NLP);

associating the interpretations of the received video content with images extracted from video content based on timeline; and

using the interpretations in the database to obtain interpretations of other images or other video content.

2. The method of claim 1 wherein the associated interpretations and images are stored in a database.

3. The method of claim 1 wherein generating interpretations comprises key frames in scene transitions.

4. The method of claim 1 wherein generating interpretations comprises clustering images.

5. The method of claim 1 wherein using the interpretations to obtain interpretations of other digital images utilizes neural networks.

6. The method of claim 1 wherein using the interpretations to obtain interpretations of other digital images utilizes Principal Component Analysis (PCA).

7. The method of claim 1 wherein using the interpretations to obtain interpretations of other digital images utilizes pixel similarity.

8. The method of claim 1 wherein the speech/Speech Natural Language (NLP) processing includes using closed captions.

9. The method of claim 1 wherein the speech/Speech Natural Language (NLP) processing includes using passages.

10. A method comprising:

receiving a stream of video content;

generating speech to text for the received video content;

generating passage level annotations from generated text using natural language processing (NLP);

associating the passage level annotations with the text from the speech time aligned to result in text, annotations and a time stamp; and

associating imagery with the annotated text to generate thumbnails at periodic time intervals resulting in a database of annotations to imagery and imagery to annotations.

11. The method of claim 10 further comprising using database of annotations to imagery and imagery to annotations to obtain interpretations of other digital images to improve the interpretation derived from the speech.

12. The method of claim 10 wherein the annotations comprise names, concepts, and/or logical substructure.

13. The method of claim 10 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals comprises applying a machine learning technique.

14. The method of claim 13 wherein the machine learning technique includes one of a neural networks approach or a classification and regression tree (CART) approach to classify an image into an annotation or to classify an annotation into an image.

15. The method of claim 13 wherein the machine learning technique is a principal component analysis (PCA) of facial imagery.

16. The method of claim 13 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals further comprises clustering the images.

17. The method of claim 10 further comprising generating a master catalog of images for selected annotations.

18. The method of claim 17 further comprising comparing videos to determine whether the videos contain speech.

19. The method of claim 18 wherein comparing videos comprises pixel comparison.

20. An apparatus comprising:

a local computing system linked to a network of interconnected computer systems, the local computing system comprising a processor 18, a memory and a storage device;

the memory comprising an operating system and a visual similarity process, the visual similarity process comprising;

receiving a stream of video content;

generating speech to text for the received video content;

generating passage level annotations from generated text using natural language processing (NLP);

associating the passage level annotations with the text from the speech time aligned to result in text, annotations and a time stamp; and

associating imagery with the annotated text to generate thumbnails at periodic time intervals resulting in a database of annotations to imagery and imagery to annotations.

21. The apparatus of claim 20 wherein the annotations comprise names, concepts, and/or logical substructure.

22. The apparatus of claim 20 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals comprises applying a machine learning technique.

23. The apparatus of claim 22 wherein the machine learning technique includes one of a neural networks approach or a classification and regression tree (CART) approach to classify an image into an annotation or to classify an annotation into an image.

24. The apparatus of claim 22 wherein the machine learning technique is a principal component analysis (PCA) of facial imagery.

25. The apparatus of claim 20 wherein associating imagery with the annotated text to generate thumbnails at periodic time intervals further comprises clustering the images.

26. The apparatus of claim 20 wherein the visual similarity process further comprises generating a master catalog of images for selected annotations.

27. The apparatus of claim 26 wherein the visual similarity process further comprises comparing videos to determine whether the videos contain speech.

28. The apparatus of claim 27 wherein comparing videos comprises pixel comparison.