Efficiently identifying images, videos, songs or documents most relevant to the user based on attribute feedback

• US 9,292,517 B2
• Filed: 08/13/2013
• Issued: 03/22/2016
• Est. Priority Date: 01/03/2013
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Original Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Inventors: Grauman, Kristen, Kovashka, Adriana, Parikh, Devi
• Primary Examiner(s): Shmatov, Alexey
• Application Number:
  US13/965,594
• Publication Number:
  US 20140188863A1
• Time in Patent Office: 952 Days
• Field of Search: 707/728, 707/766
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A method, system and computer program product for efficiently identifying images, videos, audio files or documents relevant to a user. Using either manual annotations or learned functions, the method predicts the relative strength of an attribute in an image, video, audio file or document from a pool of images, videos, audio files or documents. At query time, the system presents an initial set of reference images, videos, audio files or documents, and the user selects among them to provide relative attribute feedback. Using the resulting constraints in the multi-dimensional attribute space, the relevance function for the pool of images, videos, audio files or documents is updated and the relevance of the pool of images, videos, audio files or documents is re-computed. This procedure iterates using the accumulated constraints until the top-ranked images, videos, audio files or documents are acceptably close to the user'"'"'s envisioned image, video, audio file or document.

Claims

1. A method for efficiently identifying images, videos, audio files or documents relevant to a user, the method comprising:

determining a set of attribute ranking functions, each of which predicts or assigns a relative strength of an attribute in an image, video, audio file or document from a pool of ranked database images, videos, audio files or documents;

presenting a set of reference images, videos, audio files or documents from said pool of database images, videos, audio files or documents;

receiving a selection of one or more images, videos, audio files or documents from said set of reference images, videos, audio files or documents along with relative attribute feedback pertaining to one or more attributes of said selected one or more images, videos, audio files or documents, wherein said relative attribute feedback comprises feedback regarding a desired degree of a characteristic of an attribute; and

revising, by a processor, a system'"'"'s model of what images, videos, audio files or documents are relevant to said user by updating one or more relevance ranking functions of a set of relevance ranking functions using said relative attribute feedback and said set of attribute ranking functions, wherein said set of relevance ranking functions are used to rank said database images, videos, audio files or documents based on how relevant said database images, videos, audio files or documents are to a user'"'"'s search.

2. The method as recited in claim 1 further comprising:

training said set of relevance ranking functions using said relative attribute feedback.

3. The method as recited in claim 1 further comprising:

updating a relevance ranking function for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents;

re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance ranking function; and

displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

4. The method as recited in claim 3 further comprising:

identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

5. The method as recited in claim 3 further comprising:

receiving a selection of one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents along with relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

updating said relevance ranking function a subsequent time for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

re-ranking said pool of database images, videos, audio files or documents a subsequent time in response to updating said relevance ranking function said subsequent time; and

displaying a subsequent top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

6. The method as recited in claim 1 further comprising:

receiving one or more keywords, images, audio files, documents or videos to initialize a search for an image, video, audio file or document; and

presenting said set of reference images, videos, audio files or documents based on said one or more keywords, images, audio files, documents or videos.

7. The method as recited in claim 1, wherein said set of reference images, videos, audio files or documents is a random or otherwise automatically selected set of top-ranked images, videos, audio files or documents from said pool of database images, videos, audio files or documents.

8. A computer program product embodied in a non-transitory computer readable storage medium for efficiently identifying images, videos, audio files or documents relevant to a user, the computer program product comprising the programming instructions for:

determining a set of attribute ranking functions, each of which predicts or assigns a relative strength of an attribute in an image, video, audio file or document from a pool of ranked database images, videos, audio files or documents;

presenting a set of reference images, videos, audio files or documents from said pool of database images, videos, audio files or documents;

receiving a selection of one or more images, videos, audio files or documents from said set of reference images, videos, audio files or documents along with relative attribute feedback pertaining to one or more attributes of said selected one or more images, videos, audio files or documents, wherein said relative attribute feedback comprises feedback regarding a desired degree of a characteristic of an attribute; and

revising a system'"'"'s model of what images, videos, audio files or documents are relevant to said user by updating one or more relevance ranking functions of a set of relevance ranking functions using said relative attribute feedback and said set of attribute ranking functions, wherein said set of relevance ranking functions are used to rank said database images, videos, audio files or documents based on how relevant said database images, videos, audio files or documents are to a user'"'"'s search.

9. The computer program product as recited in claim 8 further comprising the programming instructions for:

training said set of relevance ranking functions using said relative attribute feedback.

10. The computer program product as recited in claim 8 further comprising the programming instructions for:

updating a relevance ranking function for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents;

re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance ranking function; and

displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

11. The computer program product as recited in claim 10 further comprising the programming instructions for:

identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

12. The computer program product as recited in claim 10 further comprising the programming instructions for:

receiving a selection of one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents along with relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

updating said relevance ranking function a subsequent time for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

re-ranking said pool of database images, videos, audio files or documents a subsequent time in response to updating said relevance ranking function said subsequent time; and

displaying a subsequent top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

13. The computer program product as recited in claim 8 further comprising the programming instructions for:

receiving one or more keywords, images, audio files, documents or videos to initialize a search for an image, video, audio file or document; and

presenting said set of reference images, videos, audio files or documents based on said one or more keywords, images, audio files, documents or videos.

14. The computer program product as recited in claim 8, wherein said set of reference images, videos, audio files or documents is a random or otherwise automatically selected set of top-ranked images, videos, audio files or documents from said pool of database images, videos, audio files or documents.

15. A system, comprising:

a memory unit for storing a computer program for efficiently identifying images, videos, audio files or documents relevant to a user; and

a processor coupled to said memory unit, wherein said processor, responsive to said computer program, comprises;

circuitry for determining a set of attribute ranking functions, each of which predicts or assigns a relative strength of an attribute in an image, video, audio file or document from a pool of ranked database images, videos, audio files or documents;

circuitry for presenting a set of reference images, videos, audio files or documents from said pool of database images, videos, audio files or documents;

circuitry for receiving a selection of one or more images, videos, audio files or documents from said set of reference images, videos, audio files or documents along with relative attribute feedback pertaining to one or more attributes of said selected one or more images, videos, audio files or documents, wherein said relative attribute feedback comprises feedback regarding a desired degree of a characteristic of an attribute; and

circuitry for revising a system'"'"'s model of what images, videos, audio files or documents are relevant to said user by updating one or more relevance ranking functions of a set of relevance ranking functions using said relative attribute feedback and said set of attribute ranking functions, wherein said set of relevance ranking functions are used to rank said database images, videos, audio files or documents based on how relevant said database images, videos, audio files or documents are to a user'"'"'s search.

16. The system as recited in claim 15, wherein said processor further comprises:

circuitry for training said set of relevance ranking functions using said relative attribute feedback.

17. The system as recited in claim 15, wherein said processor further comprises:

circuitry for updating a relevance ranking function for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents;

circuitry for re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance ranking function; and

circuitry for displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

18. The system as recited in claim 17, wherein said processor further comprises:

circuitry for identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

19. The system as recited in claim 17, wherein said processor further comprises:

circuitry for receiving a selection of one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents along with relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

circuitry for updating said relevance ranking function a subsequent time for said set of database images, videos, audio files or documents in response to receiving said relative attribute feedback pertaining to said one or more attributes of said selected one or more images, videos, audio files or documents from said displayed set of said re-ranked pool of database images, videos, audio files or documents;

circuitry for re-ranking said pool of database images, videos, audio files or documents a subsequent time in response to updating said relevance ranking function said subsequent time; and

circuitry for displaying a subsequent top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

20. The system as recited in claim 15, wherein said processor further comprises:

circuitry for receiving one or more keywords, images, audio files, documents or videos to initialize a search for an image, video, audio file or document; and

circuitry for presenting said set of reference images, videos, audio files or documents based on said one or more keywords, images, audio files, documents or videos.

21. The system as recited in claim 15, wherein said set of reference images, videos, audio files or documents is a random or otherwise automatically selected set of top-ranked images, videos, audio files or documents from said pool of database images, videos, audio files or documents.