Image description system and method

• US 7,254,285 B1
• Filed: 11/05/1999
• Issued: 08/07/2007
• Est. Priority Date: 11/06/1998
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Ana Benitez, Atul Puri, Chung-Sheng Li, John R. Smith, Lawrence D. Bergman, Qian Huang, Seungup Paek, Shih-Fu Chang
• Inventors: Chang, Shih-Fu, Smith, John R., Huang, Qian, Li, Chung-Sheng, Puri, Atul, Bergman, Lawrence D., Benitez, Ana, Paek, Seungup
• Primary Examiner(s): Ahmed; Samir
• Assistant Examiner(s): Kassa; Yosef
• Application Number:
  US09/831,215
• Time in Patent Office: 2,832 Days
• Field of Search: 382/282, 382/291, 382/295, 382/305, 358/538, 358/452, 358/453
• US Class Current: 382/305
• CPC Class Codes: G06V 10/424 Syntactic representation, e.g. by using alphabets or grammars

Abstract

Systems and methods for describing image content establish image description records which include an object set (24), an object hierarchy (26) and entity relation graphs (28). For image content, image objects can include global objects (O0 8) and local objects (O1 2 and O2 6). The image objects are further defined by a number of features of different classes (36, 38 and 40), which in turn are further defined by a number of feature descriptors. The relationships between and among the objects in the object set are defined by the object hierarchy (26) and entity relation graphs (28). The image description records provide a standard vehicle for describing the content and context of image information for subsequent access and processing by computer applications such as search engines, filters, and archive systems.

Claims

1. A system for generating a description record from image information, comprising:

at least one image input interface for receiving said image information;

a computer processor coupled to said at least one image input interface for receiving said image information therefrom, processing said image information by performing image object extraction processing to generate image object descriptions from said image information, processing said generated image object descriptions by object hierarchy construction and extraction processing to generate image object hierarchy descriptions, and processing said generated image object descriptions by entity relation graph generation processing to generate entity relation graph descriptions, wherein at least one description record including said image object descriptions, said image object hierarchy descriptions and said entity relation graph descriptions is generated to represent content embedded within said image information; and

a data storage system, operatively coupled to said processor, for storing said at least one description record.

2. The system of claim 1, wherein said image object extraction processing and said object hierarchy construction and extraction processing are performed in parallel.

3. The system of claim 1, wherein said image object extraction processing comprises:

image segmentation processing to segment each image in said image information into regions within said image; and

feature extraction and annotation processing to generate one or more feature descriptions for one or more said regions;

whereby said generated image object descriptions comprise said one or more feature descriptions for one or more said regions.

4. The system of claim 3, wherein said one or more feature descriptions are selected from the group consisting of media features, visual features, and semantic features.

5. The system of claim 4, wherein said semantic features are further defined by at least one feature description selected from the group consisting of who, what object, what action, where, when, why, code downloading, and text annotation.

6. The system of claim 4, wherein said visual features are further defined by at least one feature description selected from the group consisting of color, texture, position, size, shape, motion, code downloading, and orientation.

7. The system of claim 4, wherein said media features are further defined by at least one feature description selected from the group consisting of file format, file size, color representation, resolution, data file location, author, creation, scalable layer, code downloading, and modality transcoding.

8. The system of either claim 3 or 4, wherein said one or more feature descriptions include pointers to extraction and matching code in order to facilitate code downloading.

9. The system of claim 1, wherein said object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on visual feature relationships of image objects represented by said image object descriptions.

10. The system of claim 1, wherein said object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on semantic feature relationships of image objects represented by said image object descriptions.

11. The system of claim 1, wherein said object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on media feature relationships of image objects represented by said image object descriptions.

12. The system of claim 1, wherein said object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

13. The system of claim 1, wherein said object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said image object hierarchy descriptions have a plurality of hierarchical levels.

14. The system of claim 13, wherein said image object hierarchy descriptions having a plurality of hierarchical levels comprise clustering hierarchies.

15. The system of claim 14, wherein said clustering hierarchies are based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

16. The system of claim 13, wherein said image object hierarchy descriptions having a plurality of hierarchical levels are configured to comprise multiple levels of abstraction hierarchies.

17. The system of claim 16, wherein said multiple levels of abstraction hierarchies are configured to be based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships, and media feature relationships.

18. The system of claim 1, wherein said entity relation graph generation processing generates entity relation graph descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

19. The system of claim 1, further comprising an encoder for receiving and encoding said image object descriptions into encoded description information, wherein said data storage system is operative to store said encoded description information as said at least one description record.

20. The system of claim 1, wherein said image object descriptions, said image object hierarchy descriptions, and said entity relation graph descriptions are combined together to form image descriptions, and further comprising an encoder for receiving and encoding said image descriptions into encoded description information, wherein said data storage system is operative to store said encoded description information as said at least one description record.

21. The system of claim 20, wherein said encoder comprises a binary encoder.

22. The system of claim 20, wherein said encoder comprises an XML encoder.

23. A method for generating a description record from image information, comprising the steps of:

receiving said image information;

processing said image information by performing image object extraction processing to generate image object descriptions from said image information;

processing said generated image object descriptions by object hierarchy construction and extraction processing to generate image object hierarchy descriptions;

processing said generated image object descriptions by entity relation graph generation processing to generate entity relation graph descriptions, wherein at least one description record including said image object descriptions, said image object hierarchy descriptions and said entity relation graph descriptions is generated to represent content embedded within said image information; and

storing said at least one description record.

24. The method of claim 23, wherein said steps of image object extraction processing and object hierarchy construction and extraction processing are performed in parallel.

25. The method of claim 23, wherein said step of image object extraction processing comprises the further steps of:

image segmentation processing to segment each image in said image information into regions within said image; and

feature extraction and annotation processing to generate one or more feature descriptions for one or more said regions;

whereby said generated image object descriptions comprise said one or more feature descriptions for one or more said regions.

26. The method of claim 25, further comprising the step of selecting said one or more feature descriptions from the group consisting of media features, visual features, and semantic features.

27. The method of claim 26, wherein said semantic features are further defined by at least one feature description selected from the group consisting of who, what object, what action, where, when, why, code downloading, and text annotation.

28. The method of claim 26, wherein said visual features are further defined by at least one feature description selected from the group consisting of color, texture, position, size, shape, motion, code downloading, and orientation.

29. The method of claim 26, wherein said media features are further defined by at least one feature description selected from the group consisting of file format, file size, color representation, resolution, data file location, author, creation, scalable layer, code downloading, and modality transcoding.

30. The method of either claim 25 or 26, wherein said one or more feature descriptions include pointers to extraction and matching code in order to facilitate code downloading.

31. The method of claim 23, wherein said step of object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on visual feature relationships of image objects represented by said image object descriptions.

32. The method of claim 23, wherein said step of object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on semantic feature relationships of image objects represented by said image object descriptions.

33. The method of claim 23, wherein said step of object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on media feature relationships of image objects represented by said image object descriptions.

34. The method of claim 23, wherein said step of object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

35. The method of claim 23, wherein said step of object hierarchy construction and extraction processing generates image object hierarchy descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said image object hierarchy descriptions are configured to have a plurality of hierarchical levels.

36. The method of claim 35, wherein said image object hierarchy descriptions having a plurality of hierarchical levels are configured to comprise clustering hierarchies.

37. The method of claim 36, wherein said clustering hierarchies are configured to be based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

38. The method of claim 35, wherein said image object hierarchy descriptions having a plurality of hierarchical levels are configured to comprise multiple levels of abstraction hierarchies.

39. The method of claim 38, wherein said multiple levels of abstraction hierarchies are configured to be based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships, and media feature relationships.

40. The method of claim 23, wherein said step of entity relation graph generation processing generates entity relation graph descriptions of said image object descriptions based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

41. The method of claim 23, further comprising the steps of receiving and encoding said image object descriptions into encoded description information, and storing said encoded description information as said at least one description record.

42. The method of claim 23, further comprising the steps of combining said image object descriptions, said image object hierarchy descriptions, and said entity relation graph descriptions to form image descriptions, and receiving and encoding said image descriptions into encoded description information, and storing said encoded description information as said at least one description record.

43. The method of claim 42, wherein said step of encoding comprises the step of binary encoding.

44. The method of claim 42, wherein said step of encoding comprises the step of XML encoding.

45. A computer readable media containing digital information with at least one description record representing image content embedded within corresponding image information, the at least one description record comprising:

one or more image object descriptions generated from said image information using image object extraction processing;

one or more image object hierarchy descriptions generated from said generated image object descriptions using object hierarchy construction and extraction processing; and

one or more entity relation graph descriptions generated from said generated image object descriptions using entity relation graph generation processing.

46. The computer readable media of claim 45, wherein said image object descriptions, said image object hierarchy descriptions, and said entity relation graph descriptions further comprise one or more feature descriptions.

47. The computer readable media of claim 46, wherein said one or more feature descriptions are selected from the group consisting of media features, visual features, and semantic features.

48. The computer readable media of claim 47, wherein said semantic features are further defined by at least one feature description selected from the group consisting of who, what object, what action, where, when, why, code downloading, and text annotation.

49. The computer readable media of claim 47, wherein said visual features are further defined by at least one feature description selected from the group consisting of color, texture, position, size, shape, motion, code downloading, and orientation.

50. The computer readable media of claim 47, wherein said media features are further defined by at least one feature description selected from the group consisting of file format, file size, color representation, resolution, data file location, author, creation, scalable layer, code downloading, and modality transcoding.

51. The computer readable media of either claim 46 or 47, wherein said one or more feature descriptions include pointers to extraction and matching code in order to facilitate code downloading.

52. The computer readable media of claim 45, wherein said object hierarchy descriptions are based on visual feature relationships of image objects represented by said image object descriptions.

53. The computer readable media of claim 45, wherein said image object hierarchy descriptions are based on semantic feature relationships of image objects represented by said image object descriptions.

54. The computer readable media of claim 45, wherein said image object hierarchy descriptions are based on media feature relationships of image objects represented by said image object descriptions.

55. The computer readable media of claim 45, wherein said image object hierarchy descriptions are based on relationships of image objects represented by said image object descriptions, wherein said image object hierarchy descriptions have a plurality of hierarchical levels.

56. The computer readable media of claim 55, wherein said image object hierarchy descriptions having a plurality of hierarchical levels are configured to comprise multiple levels of abstraction hierarchies.

57. The computer readable media of claim 56, wherein said multiple levels of abstraction hierarchies are configured to be based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships, and media feature relationships.

58. The computer readable media of claim 55, wherein said image object hierarchy descriptions having a plurality of hierarchical levels comprise clustering hierarchies.

59. The computer readable media of claim 58, wherein said clustering hierarchies are based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

60. The computer readable media of claim 45, wherein said entity relation graph descriptions are based on relationships of image objects represented by said image object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships and media feature relationships.

61. The computer readable media of claim 45, wherein said image object descriptions are in the form of encoded description information.

62. The computer readable media of claim 45, wherein said image object descriptions, said image object hierarchy descriptions, and said entity relation graph descriptions are combined together in the form of encoded description information.

63. The computer readable media of claim 62, wherein said encoded description information is in the form of binary encoded information.

64. The computer readable media of claim 62, wherein said encoded description information is in the form of XML encoded information.