Efficiently identifying images, videos, songs or documents most relevant to the user using binary search trees on attributes for guiding relevance feedback

• US 9,176,993 B2
• Filed: 08/13/2013
• Issued: 11/03/2015
• Est. Priority Date: 01/03/2013
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Original Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Inventors: Grauman, Kristen, Kovashka, Adriana
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US13/965,677
• Publication Number:
  US 20140188901A1
• Time in Patent Office: 812 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor

Abstract

A method, system and computer program product for efficiently identifying images, videos, audio files or documents relevant to a user using binary search trees in attribute space for guiding relevance feedback. A binary tree is constructed for each relative attribute of interest. A “pivot exemplar” (at a node of the binary tree) is set for each relative attribute'"'"'s binary tree as corresponding to the database image, video, audio file or document with a median relative attribute value among that subtree'"'"'s child examples. A pivot exemplar out of the available current pivot exemplars that has the highest expected information gain is selected to be provided to the user. Comparative attribute feedback is then received from the user regarding whether a degree of the attribute in the user'"'"'s target image, video, audio file or document is more, less or equal with the attribute displayed in the selected pivot exemplar.

Claims

1. A method for efficiently identifying images, videos, audio files or documents relevant to a user using attribute binary trees for guiding relevance feedback, the method comprising:

constructing a binary search tree for each relative attribute of interest in a pool of database images, videos, audio files or documents;

setting a current pivot exemplar for each binary tree as corresponding to a database image, video, audio file or document with a median relative attribute value among all remaining database items in a subtree;

predicting an information gain that would result from asking a user how a user'"'"'s target image, video, audio file or document compares to each pivot exemplar of a plurality of pivot exemplars, wherein said comparison is made in terms of an attribute associated with that pivot'"'"'s binary tree;

providing a pivot exemplar deemed most informative from among said plurality of pivot exemplars to a user to provide a comparison between said attribute in said user'"'"'s target image, video, audio file or document and in said current pivot exemplar; and

receiving, by a processor, comparative attribute feedback from said user regarding whether a degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said provided pivot exemplar.

2. The method as recited in claim 1 further comprising:

updating relevance predictions on said pool of database images, videos, audio files or documents in response to receiving said comparative attribute feedback.

3. The method as recited in claim 2 further comprising:

re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance predictions; and

displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

4. The method as recited in claim 3 further comprising:

identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

5. The method as recited in claim 1 further comprising:

removing said current pivot exemplar from consideration in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is equal with said attribute displayed in said current pivot exemplar.

6. The method as recited in claim 1 further comprising:

moving said current pivot exemplar down one level in a binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

7. The method as recited in claim 6 further comprising:

replacing said current pivot exemplar with its child in said binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

8. The method as recited in claim 7 further comprising:

receiving additional comparative attribute feedback from said user regarding whether said degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said current pivot exemplar'"'"'s child.

9. A computer program product embodied in a computer readable storage medium for efficiently identifying images, videos, audio files or documents relevant to a user using attribute binary trees for guiding relevance feedback, the computer program product comprising the programming instructions for:

constructing a binary search tree for each relative attribute of interest in a pool of database images, videos, audio files or documents;

setting a current pivot exemplar for each binary tree as corresponding to a database image, video, audio file or document with a median relative attribute value among all remaining database items in a subtree;

predicting an information gain that would result from asking a user how a user'"'"'s target image, video, audio file or document compares to each pivot exemplar of a plurality of pivot exemplars, wherein said comparison is made in terms of an attribute associated with that pivot'"'"'s binary tree;

providing a pivot exemplar deemed most informative from among said plurality of pivot exemplars to a user to provide a comparison between said attribute in said user'"'"'s target image, video, audio file or document and in said current pivot exemplar; and

receiving comparative attribute feedback from said user regarding whether a degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said provided pivot exemplar.

10. The computer program product as recited in claim 9 further comprising the programming instructions for:

updating relevance predictions on said pool of database images, videos, audio files or documents in response to receiving said comparative attribute feedback.

11. The computer program product as recited in claim 10 further comprising the programming instructions for:

re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance predictions; and

displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

12. The computer program product as recited in claim 11 further comprising the programming instructions for:

identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

13. The computer program product as recited in claim 9 further comprising the programming instructions for:

removing said current pivot exemplar from consideration in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is equal with said attribute displayed in said current pivot exemplar.

14. The computer program product as recited in claim 9 further comprising the programming instructions for:

moving said current pivot exemplar down one level in a binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

15. The computer program product as recited in claim 14 further comprising the programming instructions for:

replacing said current pivot exemplar with its child in said binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

16. The computer program product as recited in claim 15 further comprising the programming instructions for:

receiving additional comparative attribute feedback from said user regarding whether said degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said current pivot exemplar'"'"'s child.

17. A system, comprising:

a memory unit for storing a computer program for efficiently identifying images, videos, audio files or documents relevant to a user using attribute binary trees for guiding relevance feedback; and

a processor coupled to said memory unit, wherein said processor, responsive to said computer program, comprises;

circuitry for constructing a binary search tree for each relative attribute of interest in a pool of database images, videos, audio files or documents;

circuitry for setting a current pivot exemplar for each binary tree as corresponding to a database image, video, audio file or document with a median relative attribute value among all remaining database items in a subtree;

circuitry for predicting an information gain that would result from asking a user how a user'"'"'s target image, video, audio file or document compares to each pivot exemplar of a plurality of pivot exemplars, wherein said comparison is made in terms of an attribute associated with that pivot'"'"'s binary tree;

circuitry for providing a pivot exemplar deemed most informative from among said plurality of pivot exemplars to a user to provide a comparison between said attribute in said user'"'"'s target image, video, audio file or document and in said current pivot exemplar; and

circuitry for receiving comparative attribute feedback from said user regarding whether a degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said provided pivot exemplar.

18. The system as recited in claim 17, wherein said processor further comprises:

circuitry for updating relevance predictions on said pool of database images, videos, audio files or documents in response to receiving said comparative attribute feedback.

19. The system as recited in claim 18, wherein said processor further comprises:

circuitry for re-ranking said pool of database images, videos, audio files or documents in response to updating said relevance predictions; and

circuitry for displaying a top-ranked set of said re-ranked pool of database images, videos, audio files or documents.

20. The system as recited in claim 19, wherein said processor further comprises:

circuitry for identifying an image, video, audio file or document relevant to said user in response to an image, video, audio file or document of said displayed top-ranked set of said re-ranked pool of database images, videos, audio files or documents being acceptable to said user.

21. The system as recited in claim 17, wherein said processor further comprises:

circuitry for removing said current pivot exemplar from consideration in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is equal with said attribute displayed in said current pivot exemplar.

22. The system as recited in claim 17, wherein said processor further comprises:

circuitry for moving said current pivot exemplar down one level in a binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

23. The system as recited in claim 22, wherein said processor further comprises:

circuitry for replacing said current pivot exemplar with its child in said binary tree for said attribute in response to said comparative attribute feedback indicating said degree of said attribute in said user'"'"'s target image, video, audio file or document is more or less with said attribute displayed in said current pivot exemplar.

24. The system as recited in claim 23, wherein said processor further comprises:

circuitry for receiving additional comparative attribute feedback from said user regarding whether said degree of said attribute in said user'"'"'s target image, video, audio file or document is more, less or equal with said attribute displayed in said current pivot exemplar'"'"'s child.