VIDEO VISUAL AND AUDIO QUERY

• US 20120008821A1
• Filed: 05/10/2011
• Published: 01/12/2012
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: Sharon, Eitan, Moshe, Asael, Borenstein, Eran, Brandt, Achi, Srinivasan, Praveen, Tek, Mehmet
• Granted Patent: US 9,508,011 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A video visual and audio query system for quickly identifying video within a large known corpus of videos being played on any screen or display. In one embodiment, the system can record via a mobile phone camera and microphone a live video clip from the TV and transcode it into a sequence of frame-signatures. The signatures representative of the clips can then be matched against the signatures of the TV content in a corpus across a network to identify the correct TV show or movie.

Claims

1. A method comprising:

receiving, by a computing device, content;

analyzing, by a computing device, a slice of the content;

generating, by a computing device, a bit-wise signature of the slice; and

using, by a computing device, the generated signature in a search query.

2. The method of claim 1, wherein the content is received by the computing device while playing on a different device.

3. The method of claim 1, wherein the content is video.

4. The method of claim 3, wherein the slice of the content is a frame of the video.

5. The method of claim 4, wherein each bit of the bit-wise signature is associated with a brightness of a region of the frame.

6. The method of claim 5, wherein each bit is assigned a first value if a predetermined half of the region is brighter than the other half of the region, and each bit is assigned a second value if the predetermined half of the region is not brighter than the other half of the region.

7. The method of claim 6, wherein the predetermined half of the region is a right half or a left half of the region.

8. The method of claim 6, wherein the predetermined half of the region is a top half or a bottom half of the region.

9. The method of claim 1, wherein the content is audio.

10. The method of claim 9, wherein the slice of the content is a spectrogram of the audio.

11. The method of claim 10, wherein each bit of the bit-wise signature is associated with a change in energy in a frequency band of the spectrogram.

12. The method of claim 11, wherein each bit is assigned a first value if the change in energy represents an increase in energy, and each bit is assigned a second value if the change in energy represents a decrease in energy.

16. A method of claim 1, wherein the search query is used to determine an identity of the video.

17. A method of claim 1, wherein the determining of the boundaries includes looking for lines that have a significant intensity variance along an inward edge relative to the screen and an insignificant intensity variance along an external edge relative to the screen.

18. A method of claim 1, wherein the determining of the boundaries includes filtering out lines that cut through a continuous object.

19. A method of claim 1, wherein the determining of the boundaries includes selecting lines located within the video.

13. A computing device comprising a processor configured to:

receive content;

analyze a slice of the content;

generate a bit-wise signature of the slice; and

use the generated signature in a search query.

14. A non-transitory computer-readable storage medium embodied with instructions to cause, when executed, a computing device to:

receive content;

analyze a slice of the content;

generate a bit-wise signature of the slice; and

use the generated signature in a search query.

15. A method comprising:

receiving, by a computing device, a recording of a video playing on a screen of a different device;

determining, by a computing device, boundaries of the screen in the recording;

extracting, by a computing device, the video within the determined boundaries from the recording; and

using, by a computing device, the extracted video in a search query.

20. A computing device comprising a processor configured to:

receive a recording of a video playing on a screen of a different device;

determine boundaries of the screen in the recording;

extract the video within the determined boundaries from the recording; and

use the extracted video in a search query.

21. A non-transitory computer-readable storage medium embodied with instructions to cause, when executed, a computing device to:

receive a recording of a video playing on a screen of a different device;

determine boundaries of the screen in the recording;

extract the video within the determined boundaries from the recording; and

use the extracted video in a search query.