Apparatus and a method for retrieving image objects based on correlation with natural language sentence parameters

• US 5,684,999 A
• Filed: 12/02/1994
• Issued: 11/04/1997
• Est. Priority Date: 12/06/1993
• Status: Expired due to Term

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Okamoto, Shusaku
• Primary Examiner(s): Hayes, Gail O.
• Assistant Examiner(s): Stevens, George N.
• Application Number:
  US08/348,736
• Time in Patent Office: 1,068 Days
• Field of Search: 364/419.13, 364/419.01-419.02, 364/419.07-419.08, 394/147, 382/305, 382/306, 382/175
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An apparatus for retrieving image objects in accordance with a natural language sentence according to the present invention includes: an input section for receiving a natural language sentence; a language processing section for parsing the natural language sentence by referring to a dictionary for language analysis so as to obtain a syntactic structure of the natural language sentence; a situation element division section for converting the syntactic structure of the natural language sentence into a semantic structure of the natural language sentence and for dividing a situation represented by the semantic structure of the natural language sentence into at least one situation element by referring to a situation element division knowledge base; an image database for storing at least one image object corresponding to the at least one situation element; and a retrieval section for retrieving at least one image object from the image database by using the situation element as an image retrieval key.

Claims

1. An apparatus for retrieving image objects in accordance with a natural language sentence, comprising:

an image database for storing a plurality of image objects, each of the plurality of the image objects comprising image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image;

input means for receiving a natural language sentence;

a knowledge base for defining a semantic structure of the natural language sentence;

extracting means for extracting at least one predicate and at least one case element from the natural language sentence;

judging means for judging whether the extracted predicate and the extracted case element are correlated with each other by referring to the knowledge base;

correlating means for correlating the extracted predicate and the extracted case element with each other in accordance with the judging result, so as to define an image retrieval key representing a partial meaning of the natural language sentence; and

retrieval means for retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, the retrieval means being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

2. An apparatus according to claim 1, wherein the image attribute data includes region data defining one of a single region and a combination of a plurality of regions included in the image and information representing the meaning of one of the single region and the combination of the plurality of regions.

3. An apparatus according to claim 2, wherein the information is represented by the natural language sentence.

4. An apparatus according to claim 1, wherein the retrieval means comprises means for providing region availability information defining whether the image attribute data is available or not available in accordance with the semantic similarity between the image retrieval key and the image attribute data.

5. An apparatus according to claim 4, wherein the apparatus further comprises display means for displaying at least a portion of the image object retrieved by the retrieval means in accordance with the region availability information.

6. An apparatus according to claim 1, wherein the knowledge base comprises a plurality of case frames for defining a meaning of the natural language sentence, and wherein each of the plurality of the case frames comprises:

a word defining a predicate in the natural language sentence;

a slot describing a name of a case element related to the word and a constraint for the case element to satisfy; and

situation element data describing knowledge for correlating a predicate with at least one case element.

7. An apparatus according to claim 1, wherein the correlating means classifies the image retrieval key into a group related to a background and a group related to a foreground based on the situation element division data of the case frames.

8. An apparatus for retrieving image objects in accordance with a natural language sentence, comprising:

an image database for storing a plurality of image objects, each of the plurality of image objects comprising image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image;

input means for receiving a natural language sentence;

parsing means for parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element;

correlating means for correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and

retrieval means for retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, the retrieval means being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

9. An apparatus according to claim 8, wherein the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

10. An apparatus according to claim 8, wherein the correlating means defines either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.

11. A method for retrieving image objects from an image database in accordance with a natural language sentence, the image database storing a plurality of image objects, each of the plurality of image objects including image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image, wherein the method comprises the steps of:

a) receiving a natural language sentence;

b) parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element;

c) correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and

d) retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, such retrieving being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

12. A method according to claim 11, wherein the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

13. A method according to claim 11, wherein the step c) comprises a step for defining either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.