APPARATUS AND SOFTWARE SYSTEM FOR AND METHOD OF PERFORMING A VISUAL-RELEVANCE-RANK SUBSEQUENT SEARCH

• US 20130014016A1
• Filed: 09/14/2012
• Published: 01/10/2013
• Est. Priority Date: 07/11/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: DELGO, Lior, SHARON, Eitan, BRANDT, Achiezer, BORENSTEIN, Eran, MOSHE, Asael
• Granted Patent: US 9,031,974 B2
• Time in Patent Office: Days
• US Class Current: 715/723
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/743 a collection of video files or sequences
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

A method analyzes the visual content of media such as videos for collecting together visually-similar appearances in their constituent images (e.g. same scenes, same objects, faces of the same people.) As a result, the most relevant and salient (of clearest and largest presence) visual appearances depicted in the videos are presented to the user, both for the sake of summarizing the video content for the users to “see before they watch” (that is, judge by the depicted video content in a filmstrip-like summary whether they want to mouse-click on the video and actually spend time watching it), as well as for allowing to users to further refine their video search result set according to the most relevant and salient video content returned (e.g. largest screen-time faces).

Claims

1. A method of identifying a video comprising the steps of:

identifying sequences of frames of the video as comprising respective scenes;

determining visual relevance rank of each of the scenes;

selecting a number of the scenes based on the visual relevance rank associated with each of the scenes;

identifying, within each of the selected scenes, a representative thumbnail frame; and

displaying (i) a first thumbnail corresponding to one of the representative thumbnail frames based on the visual relevance rank of the associated scene and (ii) a filmstrip including an ordered sequence of the representative thumbnail frames.

2. The method according to claim 1 further comprising linking each of the thumbnails to a corresponding one of the scenes.

3. The method according to claim 1 further comprising recognizing a selection of one of the thumbnails and playing the video starting at the scene corresponding to the selected thumbnail.

4. The method according to claim 1 wherein the visual relevance rank is based on a visual importance of the associated scene.

5. The method according to claim 1 wherein the visual relevance rank is based on a contextual importance of the associated scene.

6. The method according to claim 1 wherein the step of identifying includes designating a type of object to be included in each of the representative thumbnail frames.

7. The method according to claim 1 wherein the type of object is selected from the group consisting of faces, people, cares and moving objects.

8. The method according to claim 1 wherein the visual relevance rank of a scene is downgraded for those scenes having specified characteristics.

9. The method according to claim 8 wherein the specified characteristics are selected from the group consisting of scenes having low contrast images, scenes having a significant textual content and scenes having relatively little or no motion.

10. The method according to claim 1 wherein the step of identifying sequences of frames of the video as comprising respective scenes includes identifying one or more regions of interest appearing in the frames and segmenting sequences of the frames into scenes based on continuity of objects appearing in frames of the sequences of frames.

11. The method according to claim 1 wherein the step of identifying sequences of frames of the video as comprising respective scenes includes identifying objects appearing in the frames and segmenting sequences of the frames into scenes based on continuity of objects appearing in frames of the sequences of frames.

12. The method according to claim 1 wherein the step of identifying sequences of frames of the video as comprising respective scenes includes identifying an object appearing in the frames and segmenting sequences of the frames into scenes based the object appearing in frames of the sequences of frames.

13. The method according to claim 12 wherein the frames of the sequence of frames are discontiguous.

14. The method according to claim 1 wherein the step of determining visual relevance rank of each of the scenes is determined by identifying an object appearing in the scene that correspond to a target object specified by search criteria.

15. The method according to claim 14 wherein the target object is a person.

16. The method according to claim 1 wherein the step of determining the visual relevance rank of each of the scenes includes identifying a tube length corresponding to a duration of each of the scenes and, in response, calculating a visual relevance rank score for each of the scenes.

17. The method according to claim 1 wherein the step of identifying, within each of the selected scenes, a representative thumbnail frame includes reviewing frames of each of the selected scenes and selecting a frame as a representative thumbnail including an object visual relevance rank to an associated scene.

18. The method according to claim 1 wherein the step of selecting a frame as a representative thumbnail includes identifying an object appearing in a scene that corresponds to a target object specified by search criteria.

19. The method according to claim 1 wherein the step of displaying a filmstrip includes ordering the representative thumbnail frames in a temporal sequence corresponding to an order of appearance of the selected scenes in the video.

20. The method according to claim 1 further comprising the steps of:

identifying a selection of a thumbnail associated with the video; and

playing the video.