Semantic and cognition based image retrieval

• US 5,930,783 A
• Filed: 08/29/1997
• Issued: 07/27/1999
• Est. Priority Date: 02/21/1997
• Status: Expired due to Term

• Current Assignee: NEC Corporation
• Original Assignee: NEC USA (NEC Corporation)
• Inventors: Candan, Kasim S., Li, Wen-Syan
• Primary Examiner(s): Lintz, Paul R.
• Assistant Examiner(s): Colbert, Ella
• Application Number:
  US08/920,780
• Time in Patent Office: 697 Days
• Field of Search: 707/1, 707/104, 707/3, 382/230, 382/124, 382/181, 382/220
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A computer implemented method for searching and retrieving images contained within a database of images in which both semantic and cognitive methodologies are utilized. The method accepts a semantic and cognitive description of an image to be searched from a user, and successively refines the search utilizing semantic and cognitive methodologies and then ranking the results for presentation to the user.

Claims

1. A method of searching a database of images and retrieving at least one image contained in said database of images, comprising the steps of:

i) generating a first set of images which match a semantic criteria of the at least one image to be retrieved;

ii) generating a second set of images from the first set of images, which match a cognition criteria of the at least one image to be retrieved;

iii) generating a third set of images from the second set of images, which match a scene criteria of the at least one image to be retrieved;

iv) ranking the third set of images based on similarity to the semantic criteria and the cognition criteria of the at least one image to be retrieved into a set of results; and

v) presenting the user with the set of results.

2. The method of claim 1, wherein said second set of images contains fewer images than said first set of images.

3. The method of claim 2, wherein said third set of images contains fewer images than said second set of images.

4. A method of querying a database of images, comprising the steps of:

i) specifying a query having a semantics-based criteria and a cognition-based criteria;

ii) processing said query based on said semantics-based criteria and said cognition-based criteria of said query to obtain a set of resulting images; and

iii) providing said set of resulting images to a user.

5. The method of claim 4, wherein said step of specifying a query comprises specifying a query having said semantics-based criteria, said cognition-based criteria, and a scene-based criteria.

6. The method of claim 4, further comprising the step of ranking said set of resulting images based on similarity with said semantics-based criteria and said cognition-based criteria of said query prior to providing said set of resulting images to said user.

7. The method of claim 4, further comprising, subsequent to specifying said query, the step of verifying said query for term reformulation.

8. The method of claim 7, wherein the step of verifying said query further comprises determining alternative terms for inclusion in said query.

9. The method of claim 8, wherein the step of verifying said query further comprises one of reformulating said query to form a reformulated query and keeping said query unchanged.

10. The method of claim 9, wherein the step of processing said query comprises processing the one of said reformulated query and said query.

11. The method of claim 4, wherein the step of providing said resulting images to said user further comprises displaying said resulting images on a visual display unit for said user.

12. The method of claim 5, wherein the step of processing said query further comprises:

forwarding said semantics-based criteria to a semantics-based query processor; and

selecting a first set of images from said database of images, said first set of images having image object semantics corresponding to semantics specified in said semantics-based criteria.

13. The method of claim 12, wherein the step of processing said query further comprises, after the step of selecting said first set of images:

forwarding said cognition-based criteria to an image retrieval system;

identifying image regions in said cognition-based criteria; and

selecting a second set of images from said first set of images, said second set of images having image objects which match said image regions of said cognition-based criteria.

14. The method of claim 13, wherein the step of processing said query further comprises after the step of selecting said second set of images:

forwarding said scene-based criteria to said semantics-based query processor;

selecting said set of resulting images from said second set of images, said resulting images having image objects in a spatial relationship specified in said scene-based criteria.

15. The method of claim 5, wherein the step of processing said query further comprises:

forwarding said cognition-based criteria to an image retrieval system;

identifying image regions in said cognition-based criteria; and

selecting a first set of images from said database of images having image objects which match said image regions of said cognition-based criteria.

16. The method of claim 15, wherein the step of processing said query further comprises, after the step of selecting said first set of images:

forwarding said semantics-based criteria to a semantics-based query processor; and

selecting a second set of images from said first set of images, said second set of images having image object semantics corresponding to semantics specified in said semantics-based criteria.

17. The method of claim 16, wherein the step of processing said query further comprises, after the step of selecting said second set of images:

forwarding said scene-based criteria to said semantics-based query processor;

selecting said set of resulting images from said second set of images, said resulting images having image objects in a spatial relationship specified in said scene-based criteria.

18. The method of claim 4, wherein said cognition-based criteria of said query comprises a color constituent and a shape constituent.

19. The method of claim 4, further comprising, prior to providing said set of resulting images to said user, the step of verifying said query for term relaxation.

20. The method of claim 19, wherein the step of verifying said query further comprises determining alternative terms and predicates for inclusion in said query.

21. The method of claim 20, wherein the step of verifying said query further comprises one of relaxing said query to form a relaxed query and keeping said query unchanged;

and, if said relaxed query is formed, processing said relaxed query.

22. A method of searching a database of images and retrieving at least one image contained in said database of images, comprising the steps of:

i) generating a first set of images which match a semantics criteria of the at least one image to be retrieved;

ii) generating a second set of images from said first set of images which match a cognition criteria of the at least one image to be retrieved and;

iii) providing said at least one image contained in said database of images to a user.

23. The method of claim 22, further comprising the step of retrieving a third set of images from said second set of images which match a scene criteria of the at least one image to be retrieved, prior to providing said at least one image to said user.

24. The method of claim 22, further comprising the step of ranking said second set of images based on similarity to said semantic criteria and said cognition criteria of the at least one image to be retrieved, prior to providing said at least one image to said user.

25. The method of claim 22, wherein the step of providing said at least one image to said user further comprises displaying said at least one image on a visual display unit for said user.

26. The method of claim 22, wherein said cognition-based criteria comprises a color constituent and a shape constituent.

27. A method of searching a database of images and retrieving at least one image contained in said database of images, comprising the steps of:

i) generating a set of resulting images matching a semantics-based criteria and a cognition-based criteria of said at least one image to be retrieved from said database of images; and

ii) presenting said set of resulting images to a user.

28. The method of claim 27, wherein said step of generating said set of resulting images further comprises matching a scene-based criteria of said at least one image to be retrieved from said database of images.

29. The method of claim 27, further comprising the step of ranking said set of resulting images based on similarity to said semantics-based criteria and said cognition-based criteria of said at least one image to be retrieved from said database of images prior to presenting said set of resulting images to said user.

30. The method of claim 27, wherein the step of presenting said set of resulting images to said user further comprises displaying said set of resulting images on a visual display unit for said user.

31. The method of claim 28, wherein the step of generating said set of resulting images further comprises:

forwarding said semantics-based criteria to a semantics-based query processor; and

generating a first set of images from said database of images, said first set of images having image object semantics corresponding to semantics specified in said semantics-based criteria.

32. The method of claim 31, wherein the step of generating said set of resulting images further comprises, after the step of generating said first set of images:

forwarding said cognition-based criteria to an image retrieval system;

identifying image regions in said cognition-based criteria; and

generating a second set of images from said first set of images, said second set of images having image objects which match said image regions of said cognition-based criteria.

33. The method of claim 32, wherein said step of generating said set of resulting images further comprises, after the step of generating said second set of images:

forwarding said scene-based criteria to said semantics-based query processor;

generating said set of resulting images from said second set of images, said set of resulting images having image objects in a spatial relationship specified in said scene-based criteria.

34. The method of claim 28, wherein the step of generating said set of resulting images further comprises:

forwarding said cognition-based criteria to an image retrieval system;

identifying image regions in said cognition-based criteria; and

generating a first set of images from said database of images, said first set of images having image objects which match said image regions of said cognition-based criteria.

35. The method of claim 34, wherein the step of generating said set of resulting images further comprises, after the step of generating said first set of images:

forwarding said semantics-based criteria to a semantics-based query processor; and

generating a second set of images from said first set of images, said second set of images having image object semantics corresponding to semantics specified in said semantics-based criteria.

36. The method of claim 35, wherein said step of generating said set of resulting images further comprises, after the step of generating said second set of images:

forwarding said scene-based criteria to said semantics-based query processor;

generating said set of resulting images from said second set of images, said set of resulting images having image objects in a spatial relationship specified in said scene-based criteria.

37. The method of claim 27, wherein said cognition-based criteria comprises a color constituent and a shape constituent.