Method, system, and medium for retrieving photo using multimodal information

• US 7,739,276 B2
• Filed: 07/02/2007
• Issued: 06/15/2010
• Est. Priority Date: 07/04/2006
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jung, Yong Ju, Han, Ick Sang, Lee, Jae Won
• Primary Examiner(s): Rones; Charles
• Assistant Examiner(s): Cao; Phuong-Thao
• Application Number:
  US11/822,156
• Publication Number:
  US 20080010275A1
• Time in Patent Office: 1,079 Days
• Field of Search: 707/5, 707/3, 707/7, 707/E17.019, 707/E17.02
• US Class Current: 707/723
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A photo retrieval method, system, and medium using multimodal information which prevents a photo of an irrelevant category to be retrieved having a high ranking due to an unfavorable side effect from a query expansion, by reflecting a level of consistency between an object category of a query and a category of a photo retrieved by expansion retrieval using ontology, to a photo retrieval result ranking is provided. The photo retrieval method using the multimodal information includes: assigning an object category with respect to a query; retrieving photos associated with an expanded query term extracted from the query; determining a ranking of the retrieved photo by reflecting the assigned object category; and arranging and displaying the photos in response to the determined ranking.

Claims

1. A photo retrieval method implemented by a computing device which comprises a category assignor, an expansion retriever, a ranking determiner and a photo arranger, the method comprising:

using the category assignor, assigning an object category based on a query;

using the expansion retriever, retrieving photos associated with an expanded query term extracted from the query;

using the ranking determiner, identifying a category with respect to each retrieved photo by referring to content of each retrieved photo;

using the ranking determiner, measuring a level of consistency between the assigned object category and the identified category for each retrieved photo and determining an Image_Score corresponding to each measured level of consistency;

using the ranking determiner, assigning a score to each retrieved photo by applying the determined Image_Score and a previously provided weight;

using the ranking determiner, determining a ranking of the retrieved photos based on the assigned score; and

using the photo arranger, arranging and displaying the photos in response to the determined ranking,

wherein, in assigning the score to each retrieved photo, each score s_i given to the photo is calculated by satisfying an equation as

s_i=w×

TF_IDF_Score+(1−

w)×

Image_Score

in which w is a weight for TF_IDF_Score, wherein;

each TF_IDF_Score is a value of modeling a probability that the expanded query term is included in the memo information of the corresponding photo, and

Image_Score is a probability that the category exists.

2. The method of claim 1, wherein the query comprises a plurality of query terms, and wherein assigning the object category comprises:

determining a keyword from the query terms forming the query; and

assigning the object category to the determined keyword.

3. The method of claim 2, wherein, in determining the keyword, the keyword is determined to be query term shown as an object in the photo.

4. The method of claim 2, wherein assigning the object category to the determined keyword comprises:

calculating a distance between the keyword and each object category previously defined by ontology; and

assigning, from a result of the calculation, object category with the least distance to be the determined keyword.

5. The method of claim 1, wherein the expanded query term includes a synonym or a hyponym existing in the ontology associated with the query.

6. The method of claim 1, wherein the retrieving photos associated with the expanded query term comprises:

maintaining a memory storing photos and memo information corresponding to each photo;

identifying memo information including the expanded query term from the memory; and

extracting each photo corresponding to the identified memo information from the memory.

7. The method of claim 1, wherein the TF_IDF_Score that is the value of modeling the probability that the expanded query term is included in the memo information of the corresponding photo is calculated by satisfying an equation as

$\mathrm{TF\_IDF}$

_Score = Pr ⁢

⁢

( D i ❘

t 1 ⁢

⁢

…

⁢

⁢

t n ) ≅

∏

j = 1 n ⁢

Pr ⁢

⁢

( t j ❘

D i ) / Pr ⁡

( DF tj )

in which D indicates an annotation part, t indicates a query term, and DF indicates a number of photos including the expanded query term.

8. The method of claim 1, wherein the Image_Score that is the probability that the category exists in the corresponding photo is calculated by satisfying an equation as

$\mathrm{Image\_Score}=\mathrm{Pr}$

⁢

( I i ❘

c 1 ⁢

⁢

…

⁢

⁢

c n ) ≅

∏

j = 1 n ⁢

Pr ⁢

⁢

( c j ❘

I i ) / Pr ⁡

( I i ) ≅

∏

j = 1 n ⁢

Pr ⁡

( g i )

in which I indicates an image part of the photo, c indicates a category with respect to the keyword, g_i indicates a reliability factor of a predetermined pattern classifier, and Pr (g_i) indicates a posterior probability value acquired by recalculating the reliability factor as a probability.

9. The method of claim 1, wherein the computing device is one of mobile phone, personal digital assistant, personal computer, a notebook computer, or mobile communication query terminal.

10. At least one computer readable medium storing instructions that control at least one processor for executing a photo retrieval method implemented by a computing device which comprises of a category assignor, an expansion retriever, a ranking determiner and a photo arranger, the method comprising:

using the category assignor, assigning an object category based on a query;

using the expansion retriever, retrieving photos associated with an expanded query term extracted from the query;

using the ranking determiner, identifying a category with respect to each retrieved photo by referring to content of each retrieved photo;

using the ranking determiner, measuring a level of consistency between the assigned object category and the identified category for each retrieved photo and determining an Image_Score corresponding to each measured level of consistency;

using the ranking determiner, assigning a score to each retrieved photo by applying the determined Image_Score and a previously provided weight;

using the ranking determiner, determining a ranking of the retrieved photos based on the assigned score; and

using the photo arranger, arranging and displaying the photos in response to the determined ranking,

wherein, in assigning the score to each retrieved photo, each score s_i given to the photo is calculated by satisfying an equation as

s_i=w×

TF_IDF_Score+(1−

w)×

Image_Score

in which w is a weight for TF_IDF_Score, wherein;

each TF_IDF_Score is a value of modeling a probability that the expanded query term is included in the memo information of the corresponding photo, and

Image Score is a probability that the category exists.

11. A photo retrieval system to retrieve photos using computing device which comprises a category assignor, an expansion retriever, a ranking determiner and a photo arranger, the system comprising:

a memory coupled to the computing device;

the category assignor to assign an object category based on a query;

the expansion retriever to retrieve photos associated with an expanded query term extracted from the query;

the ranking determiner to identify a category with respect to each retrieved photo by referring to content of each retrieved photo, to measure a level of consistency between the assigned object category and the identified category for each retrieved photo and to determine an Image_Score corresponding to the measured level of consistency, to assign a score to each retrieved photo by applying the determined Image—

Score and a previously provided weight, and to determine a ranking of the retrieved photos based on the assigned score; and

the photo arranger to arrange and display the photos in response to the determined ranking,

wherein the ranking determiner calculates a score s_i given to each photo, by satisfying an equation as

s_i=w×

TF_IDF_Score+(1−

w)×

Image_Score

in which w is a weight for TF_IDF_Score, wherein;

each TF_IDF_Score is a value of modeling a probability that the expanded query term is included in the memo information of the corresponding photo, and

Image_Score is a probability that the category exists.

12. The system of claim 11, wherein the category assignor determines a keyword from query terms forming the query;

and assigns the object category to the determined keyword.

13. The system of claim 12, wherein the category assignor determines a query term shown as an object in the photo to be the keyword.

14. The system of claim 12, wherein the category assignor calculates a distance between the keyword and each object category previously defined by ontology;

and

assigns, from a result of the calculation, object category with the least distance to be the determined keyword.

15. The system of claim 11, wherein the expanded query term includes a synonym or a hyponym existing in the ontology associated with the query.

16. The system of claim 11, wherein the memory stores photos and memo information corresponding to each photo;

and wherein the expansion retriever identifies memo information including the expanded query term from the memory, and extracts each photo corresponding to the identified memo information from the memory.

17. The system of claim 11, wherein the ranking determiner calculates the TF_IDF_Score that is the value of modeling the probability that the expanded query term is included in the memo information of the corresponding photo, by satisfying an equation as

$\mathrm{TF\_IDF}$

_Score = Pr ⁡

( D i ❘

t 1 ⁢

⁢

…

⁢

⁢

t n ) ≅

∏

j = 1 n ⁢

Pr ⁢

⁢

( t j ❘

D i ) / Pr ⁡

( DF tj )

in which D indicates an annotation part, t indicates a query term, and DF indicates a number of photos including the expanded query term.

18. The system of claim 11, wherein the ranking determiner calculates the Image_Score that is a probability that the category exists in the corresponding photo, by satisfying an equation as

$\mathrm{Image\_Score}=\mathrm{Pr}$

⁢

( I i ❘

c 1 ⁢

⁢

…

⁢

⁢

c n ) ≅

∏

j = 1 n ⁢

Pr ⁡

( c j ❘

I i ) / Pr ⁡

( I i ) ≅

∏

j = 1 n ⁢

Pr ⁢

⁢

( g i )

in which I indicates an image part of the photo, c indicates a category with respect to the keyword, g_i indicates a reliability factor of a predetermined pattern classifier, and Pr (g_i) indicates a posterior probability value acquired by recalculating the reliability factor as a probability.