APPARATUS AND SOFTWARE SYSTEM FOR AND METHOD OF PERFORMING A VISUAL-RELEVANCE-RANK SUBSEQUENT SEARCH

• US 20100070483A1
• Filed: 07/13/2009
• Published: 03/18/2010
• Est. Priority Date: 07/11/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: MOSHE, Asael, BORENSTEIN, Eran, SHARON, Eitan, BRANDT, Achiezer, DELGO, Lior
• Granted Patent: US 8,364,660 B2
• Time in Patent Office: Days
• US Class Current: 707/706
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/329 on a disc [VTOC]

Abstract

A method analyzes the visual content of media such as videos for collecting together visually-similar appearances in their constituent images (e.g. same scenes, same objects, faces of the same people.) As a result, the most relevant and salient (of clearest and largest presence) visual appearances depicted in the videos are presented to the user, both for the sake of summarizing the video content for the users to “see before they watch” (that is, judge by the depicted video content in a filmstrip-like summary whether they want to mouse-click on the video and actually spend time watching it), as well as for allowing to users to further refine their video search result set according to the most relevant and salient video content returned (e.g. largest screen-time faces).

Claims

1. A method comprising the steps of:

receiving a search query specifying search criteria;

searching for videos satisfying the search criteria;

identifying visual objects in the videos;

grouping the videos based on the visual objects; and

displaying images of the visual objects in association with respective ones of the groups of videos.

2. The method according to claim 1 further comprising:

selecting one of the groups of videos; and

displaying a result of the searching step in an order responsive to the selecting step.

3. The method according to claim 1 wherein the objects comprise images of faces.

4. The method according to claim 1 wherein the step of identifying includes determining which of the objects appear most frequently in the videos.

5. The method according to claim 1 wherein the step of identifying includes determining a ranking of the objects.

6. The method according to claim 5 wherein the step of ranking the objects is based on prominency of appearance and duration of each of the objects within the videos.

7. The method according to claim 1 further comprising ranking the objects and increasing a ranking of videos containing the ranked objects based on the rankings of the objects.

8. The method according to claim 7 further comprising displaying links to the videos in an order determined by the rankings of the videos.

9. An apparatus comprising:

an interface operating to receive a search query specifying search criteria;

a search engine operating to search for videos satisfying the search criteria;

a recognition engine operating to identify visual objects in the videos;

a formatting engine operating to group the videos based on the visual objects; and

a display engine operating to display images of the visual objects in association with respective ones of the groups of videos.

10. The apparatus according to claim 9 wherein said formatting engine further operates to select one of the groups of videos and said display engine operates to display video satisfying the search criteria in an order giving preference to videos within the selected group of videos.

11. The apparatus according to claim 9 wherein the objects comprise images of faces.

12. The apparatus according to claim 9 wherein the recognition engine further operates to determine which of the objects appear most frequently in the videos.

13. The apparatus according to claim 9 wherein the formatting engine further operates to determine a ranking of the objects.

14. The apparatus according to claim 13 wherein the ranking of the objects is based on prominency of appearance and duration of each of the objects within the videos.

15. The apparatus according to claim 9 wherein the formatting engine further operates to rank the objects and increase a ranking of videos containing the ranked objects based on the rankings of the objects.

16. The apparatus according to claim 15 further wherein the display engine further operates to display links to the videos in an order determined by the rankings of the videos.

17. A computer program comprising:

a computer usable medium having computer readable program code embodied therein, the computer readable program code including;

computer readable program code for causing the computer to receive a search query specifying search criteria;

computer readable program code for causing the computer to search for videos satisfying the search criteria;

computer readable program code for causing the computer to identify visual objects in the videos;

computer readable program code for causing the computer to group the videos based on the visual objects; and

computer readable program code for causing the computer to display images of the visual objects in association with respective ones of the groups of videos.