Video visual and audio query

• US 9,508,011 B2
• Filed: 05/10/2011
• Issued: 11/29/2016
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: Tek, Mehmet, Sharon, Eitan, Moshe, Asael, Srinivasan, Praveen, Borenstein, Eran, Brandt, Achi
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Kholdebarin, Iman K
• Application Number:
  US13/104,793
• Publication Number:
  US 20120008821A1
• Time in Patent Office: 2,030 Days
• Field of Search: 382100-119
• US Class Current: 1/1
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A video visual and audio query system for quickly identifying video within a large known corpus of videos being played on any screen or display. In one embodiment, the system can record via a mobile phone camera and microphone a live video clip from the TV and transcode it into a sequence of frame-signatures. The signatures representative of the clips can then be matched against the signatures of the TV content in a corpus across a network to identify the correct TV show or movie.

Claims

1. A method, comprising:

receiving, by a computing device, content comprising a media clip including audio and video;

analyzing, by the computing device, at least a portion of the content, said analyzing including preparing query frames of the content by selecting representative frames in the content and selecting at least one of the audio and video in the content to generate a signature for a search query;

generating, by the computing device, the signature, the signature including a set of bits identifying the content, wherein the set of bits are determined from a plurality of regions of a video frame of the representative frames in the content, a bit of the set of bits associated with a region of the plurality of regions, the bit being assigned a first value if a predetermined half of the region is brighter than the other half of the region, and the bit being assigned a second value if the predetermined half of the region is not brighter than the other half of the region; and

receiving video search results resulting from a comparison of the generated signature in a search query against a search corpus to identify the media clip in the search corpus.

2. The method of claim 1, wherein the receiving by the computing device occurs while the media clip is playing on a different device.

3. The method of claim 1, wherein the content is video and the analyzing including detecting an active play screen area by analyzing intensity edges of the at least a portion of the content.

4. The method of claim 1, wherein the comparison includes randomly selecting multiple collections of a fixed number of bits and comparing the multiple collections against a search corpus.

5. The method of claim 1, wherein the predetermined half of the region is a right half or a left half of the region.

6. The method of claim 1, wherein the predetermined half of the region is a top half or a bottom half of the region.

7. The method of claim 1, wherein the at least a portion of the content is a spectrogram of the audio.

8. The method of claim 7, wherein the analyzing further includes creating a signature from audio in the at least a portion of the content, each bit therein being associated with a change in energy in a frequency band of the spectrogram.

9. The method of claim 8, wherein each bit is assigned a first value if the change in energy represents an increase in energy, and each bit is assigned a second value if the change in energy represents a decrease in energy.

10. A computer-readable storage device embodied with instructions to cause, when executed, a computing device to:

receive content comprising a query frame including video content and audio content;

analyze the query frame;

generate a signature of the video content of the query frame wherein each bit of the signature is associated with a brightness of a region of the frame and each bit is assigned a first value if a predetermined half of the region is brighter than the other half of the region, and each bit is assigned a second value if the predetermined half of the region is not brighter than the other half of the region; and

use the signature of the audio content and the signature of the video content in a search query including randomly selecting multiple collections of a sub-set of bits in the signature of the video content and comparing the multiple collections against a search corpus.

11. A method comprising:

receiving, by a first computing device from a second computing device, a signature of a portion of a media clip, the signature representing a recording of a video playing on a screen of a different device which is captured by the second computing device, the signature generated from selected query frames of either or both of audio content or video content in the media clip;

using, by a first computing device, the signature in a search query by comparing the signature against a search corpus to reduce a set of result candidates for the search query the comparing including a coarse direct comparison of the signature to ground truth signatures and, if needed, a second comparison to an index of the search corpus, the comparing to identify the video playing on the screen of the different device; and

 

outputting results of the search query from the first computing device to the second computing device.

12. A method of claim 11, wherein the search query matches the signature to a corpus of ground truth signatures specifying an identity of the selected query frames as a video.

13. A method of claim 11, wherein the query frame is created by determining boundaries of the screen by looking for lines that have an intensity variance indicating one or more of the lines is not dark along an inward edge relative to the screen and an intensity variance indicating the one or more lines are dark along an external edge relative to the screen.

14. A method of claim 13, wherein the determining of the boundaries includes filtering out lines that cut through a continuous object.

15. A method of claim 13, wherein the determining of the boundaries includes selecting lines located within the video.

16. A computing device comprising a processor configured to:

receive a recording of a video playing on a screen of a different device, the recording including a video portion and an audio portion;

determine boundaries of the screen in video portion of the recording by determining lines that have an intensity variance indicating one or more of the lines is not dark along an inward edge relative to the screen and an intensity variance indicating the one or more lines are dark along an external edge relative to the screen;

extract the video within the determined boundaries from the recording;

extract the audio portion from the recording;

generate a signature for the audio portion and a bit wise signature for the video portion and use the signatures in one or more search queries to identify the recording.

17. A method comprising:

receiving a recording of a video playing on a screen of a different device, the recording including a video portion and an audio portion;

determining boundaries of the screen in the video portion of the recording, the determining of the boundaries includes looking for lines that have an intensity variance indicating one or more of the lines is not dark along an inward edge relative to the screen and an intensity variance indicating the one or more lines are dark along an external edge relative to the screen; and

extracting the video within the determined boundaries from the recording;

extracting the audio portion from the recording;

generating a signature for the extracted audio portion and a bit wise signature for the extracted video portion and

using the extracted video in a search query identify the content as part of a broadcast media content.