Apparatus and software system for and method of performing a visual-relevance-rank subsequent search

• US 8,364,660 B2
• Filed: 07/13/2009
• Issued: 01/29/2013
• Est. Priority Date: 07/11/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: Sharon, Eitan, Brandt, Achiezer, Borenstein, Eran, Moshe, Asael, Delgo, Lior
• Primary Examiner(s): Alam, Shahid
• Assistant Examiner(s): ASPINWALL, EVAN S
• Application Number:
  US12/502,206
• Publication Number:
  US 20100070483A1
• Time in Patent Office: 1,296 Days
• Field of Search: 707600-899, 709200-253, 714 1- 57, 714699-824, 719310-332
• US Class Current: 707/706
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/329 on a disc [VTOC]

Abstract

A method analyzes the visual content of media such as videos for collecting together visually-similar appearances in their constituent images (e.g. same scenes, same objects, faces of the same people.) As a result, the most relevant and salient (of clearest and largest presence) visual appearances depicted in the videos are presented to the user, both for the sake of summarizing the video content for the users to “see before they watch” (that is, judge by the depicted video content in a filmstrip-like summary whether they want to mouse-click on the video and actually spend time watching it), as well as for allowing to users to further refine their video search result set according to the most relevant and salient video content returned (e.g. largest screen-time faces).

Claims

1. A method comprising the steps of:

receiving a search query specifying search criteria;

searching for videos satisfying the search criteria;

identifying one or more visual objects in each of the videos of video search results returned by the search query;

determining a visual relevance for each of the visual objects in each of the videos within the video search results, wherein determining the visual relevance for each of the visual objects comprises ranking the visual objects and increasing a ranking of the videos of the video search results containing the ranked visual objects based on the rankings of the visual objects;

grouping the visual objects present in multiple of the videos of the video search results;

displaying an image for each visual object grouping along with the video search results;

receiving a selection of one of the visual objects from a user; and

displaying a reorder of the video search results responsive to the selection comprising generating a new set of video search results comprising a new set of videos and a new set of visual objects, in accordance with the visual relevance of the visual object selected by the user.

2. The method according to claim 1 wherein the image for each visual object grouping comprises a face.

3. The method according to claim 1 wherein the step of identifying includes determining which of the visual objects appear most frequently in the videos of the video search results.

4. The method according to claim 1 wherein the step of identifying includes determining a ranking of the visual objects to determine the visual relevance for each of the visual objects.

5. The method according to claim 4 wherein the step of ranking the visual objects is based on prominency of appearance and duration of each of the visual objects within the videos of the video search results.

6. The method according to claim 1 wherein determining the visual relevance for each of the visual objects comprises ranking the visual objects and increasing a ranking of the videos of the video search results containing the ranked visual objects based on the rankings of the visual objects.

7. The method according to claim 6 further comprising displaying links to the videos of the video search results and the new set of video search results in an order determined by the rankings of the videos of the video search results and the new set of video search results.

8. An apparatus comprising:

an interface operating to receive a search query specifying search criteria;

a search engine operating to search for videos satisfying the search criteria;

a recognition engine operating to identify one or more visual objects in each of the videos of a first set of video search results returned by the search query and determine a visual relevance for each of the visual objects in each of the videos within the first set of video search results;

a formatting engine operating to group the visual objects present in multiple of the videos of the first set of video search results; and

a display engine operating to;

display an image for each visual object grouping along with the first set of video search results;

receive a selection of one of one of the visual objects from a user;

generate a second set of video search results comprising a second set of videos

responsive to the selection, wherein the visual relevance of the second set of videos appearing in the second set of video search results is different from the visual relevance of the videos appearing in the first set of video search results, the visual relevance for each of the visual objects being determined by ranking the visual objects and increasing a ranking of the videos of the video search results containing the ranked visual objects based on the rankings of the visual objects; and

display the second set of video search results comprising the second set of videos to the user.

9. The apparatus according to claim 8 wherein the image for each visual object grouping comprises a face.

10. The apparatus according to claim 8 wherein the recognition engine further operates to determine which of the visual objects appear most frequently in the videos of the first set of video search results.

11. The apparatus according to claim 8 wherein the formatting engine further operates to determine a ranking of the visual objects.

12. The apparatus according to claim 11 wherein the ranking of the visual objects is based on prominency of appearance and duration of each of the visual objects within the videos of the first set of video search results.

13. The apparatus according to claim 8 wherein the formatting engine further operates to rank the visual objects and increase a ranking of the videos of the first set of video search results containing the ranked visual objects based on the rankings of the visual objects.

14. The apparatus according to claim 13 further wherein the display engine further operates to display links to the videos of the first and second set of video search results in an order determined by the rankings of the videos of the first and second set of video search results.

15. A non-transitory computer usable medium having computer readable program code embodied therein, the computer readable program code including:

computer readable program code for causing the computer to receive a search query specifying search criteria;

computer readable program code for causing the computer to search for videos satisfying the search criteria;

computer readable program code for causing the computer to analyze video search results returned by the search query to identify one or more visual objects in each of the videos of the video search results returned by the search query;

computer readable program code for causing the computer to determine a visual relevance for each of the visual objects in each of the videos based on a visual significance of the visual objects within the video search results, the visual relevance for each of the visual objects being determined by ranking the visual objects and increasing a ranking of the videos of the video search results containing the ranked visual objects based on the rankings of the visual objects;

computer readable program code for causing the computer to group the visual objects present in multiple of the videos of the video search results;

computer readable program code for causing the computer to display an image for each visual object grouping along with the video search results;

computer readable program code for causing the computer to receive a selection of one of the images from a user; and

computer readable program code for causing the computer to display a new set of video search results comprising a new set of videos to the user, in accordance with the visual relevance of the image selected by the user.