Method, system, and medium for retrieving photo using multimodal information

• US 20080010275A1
• Filed: 07/02/2007
• Published: 01/10/2008
• Est. Priority Date: 07/04/2006
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jae Won, Jung, Yong Ju, Han, Ick Sang
• Granted Patent: US 7,739,276 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A photo retrieval method, system, and medium using multimodal information which prevents a photo of an irrelevant category to be retrieved having a high ranking due to an unfavorable side effect from a query expansion, by reflecting a level of consistency between an object category of a query and a category of a photo retrieved by expansion retrieval using ontology, to a photo retrieval result ranking is provided. The photo retrieval method using the multimodal information includes: assigning an object category with respect to a query; retrieving photos associated with an expanded query term extracted from the query; determining a ranking of the retrieved photo by reflecting the assigned object category; and arranging and displaying the photos in response to the determined ranking.

Claims

1. A photo retrieval method implemented by a computing device, comprising:

assigning an object category based on a query;

retrieving photos associated with an expanded query term extracted from the query;

determining a ranking of the retrieved photos based on the assigned object category; and

arranging and displaying the photos in response to the determined ranking.

2. The method of claim 1, wherein the query comprises a plurality of query terms, and wherein assigning the object category comprises:

determining a keyword from the query terms forming the query; and

assigning the object category to the determined keyword.

3. The method of claim 2, wherein, in determining the keyword, the keyword is determined to be query term shown as an object in the photo.

4. The method of claim 2, wherein assigning the object category to the determined keyword comprises:

calculating a distance between the keyword and each object category previously defined by ontology; and

assigning, from a result of the calculation, object category with the least distance to be the determined keyword.

5. The method of claim 1, wherein the expanded query term includes a synonym or a hyponym existing in the ontology associated with the query.

6. The method of claim 1, wherein the retrieving photos associated with the expanded query term comprises:

maintaining a memory storing photos and memo information corresponding to each photo;

identifying memo information including the expanded query term from the memory; and

extracting each photo corresponding to the identified memo information from the memory.

7. The method of claim 6, wherein the determining the ranking of each retrieved photo comprises:

identifying a category with respect to for each retrieved photo by referring to content of each retrieved photo;

measuring a level of consistency between the assigned object category and the identified category for each retrieved photo and determining an Image_Score corresponding to each measured level of consistency; and

assigning a score to each retrieved photo by applying the determined Image_Score and a previously provided weight.

8. The method of claim 7, wherein, in assigning the score to each retrieved photo, each score s_i given to the photo is calculated by satisfying an equation as

s_i=w×

TF_IDF_Score+(1-w)×

Image_Score

in which w is a weight for TF_IDF_Score, wherein;

each TF_IDF_Score is a value of modeling a probability that the expanded query term is included in the memo information of the corresponding photo, and

Image_Score is a value probability that the category exists.

9. The method of claim 8, wherein the TF_IDF_Score that is the value of modeling the probability that the expanded query term is included in the memo information of the corresponding photo is calculated by satisfying an equation as

$\mathrm{TF\_IDF}$

_Score = Pr 



( D i 

t 1 



…





t n ) ≅

∏

j = 1 n 

Pr 



( t j 

D i ) / Pr 

( DF tj )

in which D indicates an annotation part, t indicates a query term, and DF indicates a number of photos including the expanded query term.

10. The method of claim 8, wherein the Image_Score that is the probability that the category exists in the corresponding photo is calculated by satisfying an equation as

$\mathrm{Image\_Score}=\mathrm{Pr}$



( I i 

c 1 



…





c n ) ≅

∏

j = 1 n 

Pr 



( c j 

I i ) / Pr 

( I i ) ≅

∏

j = 1 n 

Pr 

( g i )

in which I indicates an image part of the photo, c indicates a category with respect to the keyword, g_i indicates a reliability factor of a predetermined pattern classifier, and Pr (g_i) indicates a posterior probability value acquired by recalculating the reliability factor as a probability.

22. The method of claim 1, wherein the computing device is one of mobile phone, personal digital assistant, personal computer, a notebook computer, or mobile communication query terminal.

11. At least one computer readable medium storing instructions that control at least one processor for executing a photo retrieval method, the method comprising:

assigning an object category based on a query;

retrieving photos associated with an expanded query term extracted from the query;

determining a ranking of the retrieved photos based on the assigned object category; and

arranging and displaying the photos in response to the determined ranking.

12. A photo retrieval system using multimodal information, the system comprising:

a category assignor to assign an object category based on a query;

an expansion retriever to retrieve photos associated with an expanded query term extracted from the query;

a ranking determiner to determine a ranking of the retrieved photos based on the assigned object category; and

a photo arranger to arrange and display the photos in response to the determined ranking.

13. The system of claim 12, wherein the category assignor determines a keyword from query terms forming the query;

and assigns the object category to the determined keyword.

14. The system of claim 13, wherein the category assignor determines a query term shown as an object in the photo to be the keyword.

15. The system of claim 13, wherein the category assignor calculates a distance between the keyword and each object category previously defined by ontology;

and assigns, from a result of the calculation, object category with the least distance to be the determined keyword.

16. The system of claim 12, wherein the expanded query term includes a synonym or a hyponym existing in the ontology associated with the query.

17. The system of claim 12, further comprising a memory to store photos and memo information corresponding to each photo, wherein the expansion retriever identifies memo information including the expanded query term from the memory;

and extracts each photo corresponding to the identified memo information from the memory.

18. The system of claim 17, wherein the ranking determiner identifies a category with respect to each retrieved photo by referring to content of each retrieved photo;

measurers a level of consistency between the assigned object category and the identified category for each retrieved photo and determines an Image_Score corresponding to the measured level of consistency; and

assigns a score to each retrieved photo by applying the determined Image_Score and a previously provided weight.

19. The system of claim 18, wherein the ranking determiner calculates a score s_i given to each photo, by satisfying an equation as

s_i=w×

TF_IDF_Score+(1-w)×

Image_Score

in which w is a weight for TF_IDF_Score, wherein;

each TF_IDF_Score is a value of modeling a probability that the expanded query term is included in the memo information of the corresponding photo, and

Image_Score is a value probability that the category exists.

20. The system of claim 19, wherein the ranking determiner calculates the TF_IDF_Score that is the value of modeling the probability that the expanded query term is included in the memo information of the corresponding photo, by satisfying an equation as

$\mathrm{TF\_IDF}$

_Score = Pr 

( D i 

t 1 



…





t n ) ≅

∏

j = 1 n 

Pr 



( t j 

D i ) / Pr 

( DF tj )

in which D indicates an annotation part, t indicates a query term, and DF indicates a number of photos including the expanded query term.

21. The system of claim 19, wherein the ranking determiner calculates the Image_Score that is a probability that the category exists in the corresponding photo, by satisfying an equation as

$\mathrm{Image\_Score}=\mathrm{Pr}$



( I i 

c 1 



…





c n ) ≅

∏

j = 1 n 

Pr 

( c j 

I i ) / Pr 

( I i ) ≅

∏

j = 1 n 

Pr 



( g i )

in which I indicates an image part of the photo, c indicates a category with respect to the keyword, g_i indicates a reliability factor of a predetermined pattern classifier, and Pr (g_i) indicates a posterior probability value acquired by recalculating the reliability factor as a probability.

23. A photo retrieval method implemented by a computing device, comprising:

extracting a query term from a query;

assigning an object category based on the query;

expanding the query term to form an expanded query term;

retrieving photos associated with the expanded query term extracted from the query; and

determining a ranking of the retrieved photos based on the assigned object category.

24. The photo retrieval method of claim 23, further comprising arranging and displaying the photos in response to the determined ranking.

25. The method of claim 23, wherein the computing device is one of mobile phone, personal digital assistant, personal computer, a notebook computer, or mobile communication query terminal.

26. At least one computer readable medium storing instructions that control at least one processor for executing a photo retrieval method, the method comprising:

extracting a query term from a query;

assigning an object category based on the query;

expanding the query term to form an expanded query term;

retrieving photos associated with the expanded query term extracted from the query; and

determining a ranking of the retrieved photos based on the assigned object category.