Interactive framework for understanding user's perception of multimedia data

• US 6,408,293 B1
• Filed: 06/09/1999
• Issued: 06/18/2002
• Est. Priority Date: 06/09/1999
• Status: Expired due to Term

• Current Assignee: Uniloc 2017 LLC (SoftBank Group Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Ravi, Tumkur Venkatanarayana Rao, Kulshreshtha, Ashutosh, Ghosal, Sugata, Dubey, Pradeep Kumar, Aggarwal, Gaurav
• Primary Examiner(s): Mizrahi, Diane D.
• Application Number:
  US09/328,968
• Time in Patent Office: 1,105 Days
• Field of Search: 707/104, 707/6, 707/3, 707/5, 707/2, 382/170, 382/284, 382/217
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A methodology of highly interactive intra-object relevance feedback is used to retrieve multimedia data from a database. The query object could consist of one or more images, images derived from video, a video sequence, or an audio clip. The query is adjusted using the information fed-back by the user about the relevance of previously extracted part(s) from the object itself, such that the adjusted query is a better approximation to the user'"'"'s perception. The information fed-back by the user during intra-query modification is used for intra-object learning of the user'"'"'s perception. The refined query is subsequently used for inter-object relevance feedback where data is retrieved from the database based on parameters learnt by intra-query object feedback mechanism, and the user provides feedback by ranking the retrieved objects in order of their relevance to him or her. In the system according to the invention, inter-object learning of user'"'"'s perception is expedited by utilizing the learnt parameters in the intra-object relevance feedback. Furthermore, the methodology of the invention allows for building refined queries based on part(s) or sub-sequence(s) of the query object rather than the entire object itself, thereby reducing the number of irrelevant objects, retrieved from the database. The methodology allows synthesis and modification of the input query object itself in the event a query object is not directly available, and, also to learn the user'"'"'s perception.

Claims

1. A computer implemented method embodying an interactive framework for understanding a user'"'"'s perception of multimedia data for the retrieval of data from a database comprising the steps of:

(a) receiving a user'"'"'s specification of area(s) or portion(s) of interest as part of the user'"'"'s query object,

(b) tentatively over-segmenting at least a part of the query object,

(c) merging or splitting of segments specified by the user,

(d) learning the user'"'"'s perception of area(s) or portion(s) of interest selected from the group consisting of color, texture, shape, time sequence, audio mixture, and topological arrangement of segments, and

(e) extracting from the database semantic object(s) using the segments learned in said learning step.

2. The method according to claim 1, wherein the query object is defined by a set of segments characterized by at least one of a feature vector, a shape description, or a similarity measure.

3. The method according to claim 1, wherein the user'"'"'s specification includes at least one of color, texture, shape, audio frequency, motion or region of interest of the query object.

4. The method according to claim 1, wherein the query object is over-segmented by at least one of color, texture, shape, audio frequency, or motion.

5. The computer implemented method of claim 1, wherein the step of inter-object processing comprises the steps of:

(a) searching the database for similar query object(s) using learned parameters from intra-query object processing;

(b) receiving the user'"'"'s ranking of the retrieved objects as relevant or irrelevant data; and

(c) understanding the user'"'"'s perception.

6. A computer program product comprising a computer usable medium having computer readable program code embodied in the medium for embodying an interactive framework for understanding a user'"'"'s perception of multimedia data for the retrieval of data from a database, the computer program product including:

first computer program code for receiving a user'"'"'s specification of area(s) or portion(s) of interest as part of the user'"'"'s query object;

second computer program code for tentatively over-segmenting at least a part of the query object;

third computer program code for merging or splitting of segments specified by the user;

fourth computer program code for learning the user'"'"'s perception of area(s) or portion(s) of interest selected from the group consisting of color, texture, shape, time sequence, audio mixture, and topological arrangement of segments; and

fifth computer program code for extracting from the database semantic object(s) using the segments learned by said fourth computer program.

7. A computer program product comprising according to claim 6, wherein the fifth computer program code comprises:

sixth computer program code for searching the database for similar query object(s) using learned parameters from intra-query object processing;

seventh computer program code for receiving the user'"'"'s ranking of the retrieved objects as relevant or irrelevant data; and

eighth computer program code for understanding the user'"'"'s perception.

8. A system for embodying an interactive framework for understanding a user'"'"'s perception of multimedia data for the retrieval of data from a database comprising:

means for receiving a user'"'"'s specification of area(s) or portion(s) of interest as part of the user'"'"'s query object;

means for tentatively over-segmenting at least a part of the query object;

means for merging or splitting of segments specified by the user;

means for learning the user'"'"'s perception of area(s) or portion(s) of interest selected from the group consisting of color, texture, shape, time sequence, audio mixture, and topological arrangement of segments; and

means for extracting from the database semantic object(s) using the segments learned by said learning means.

9. A system for embodying an interactive framework for understanding a user'"'"'s perception of multimedia data for the retrieval of data from a database comprising according to claim 8, wherein the means for extracting comprises:

(a) means for searching the database for similar query object(s) using learned parameters from intra-query object processing;

(b) means for receiving the user'"'"'s ranking of the retrieved objects as relevant or irrelevant data; and

(c) means for understanding the user'"'"'s perception.