System and method for measuring image similarity based on semantic meaning

• US 20030195883A1
• Filed: 04/15/2002
• Published: 10/16/2003
• Est. Priority Date: 04/15/2002
• Status: Active Grant

• Current Assignee: Wrp IP Management, LLC
• Original Assignee: International Business Machines Corporation
• Inventors: Mojsilovic, Aleksandra, Gomes, Jose, Rogowitz, Bernice
• Granted Patent: US 7,043,474 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and system for determining image similarity is based on the semantic meaning of images. The method includes deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features; comparing an image to said semantic categories; and classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image. A database contains image information, where the image information includes at least one of already classified images, network locations of already classified images and documents containing already classified images. The database is searched for images matching an input query, that can comprise an image, text, or both. The images stored in or referenced by the database may be obtained at least in part through the Internet, such as by the activity of an automated web crawler. In the presently preferred embodiment the images are medical images, and the medical database may be searched for at least one image that satisfies thresholds established by a search query. Also disclosed is a method and system for searching the Internet. The method includes (a) providing at least one server accessible by a plurality of clients; (b) downloading a program to a client, where the program includes an Internet searching program operable for locating at least one type of data; and (c) executing the downloaded program on the client, and returning to the server URLs corresponding to located data.

Claims

1. A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising:

deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories;

for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features;

comparing an image to said semantic categories; and

classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image.

2. A method as in claim 1, where a database stores image information, where the image information comprises at least one of already classified images, network locations of already classified images and documents containing already classified images, and where said database is searched for images matching an input query.

3. A method as in claim 2, where said input query comprises an input image, further comprising extracting features from said input image and applying a metric to determine the semantic category for said input image.

4. A method as in claim 2, where said input query comprises an input image, further comprising processing said input image and comparing said input image to each semantic category using a metric that computes a similarity measure between the features used to describe the semantic category and corresponding features extracted from said input image;

and assigning said input image to the semantic category that corresponds to a highest value of the similarity measure.

5. A method as in claim 2, where said image information is obtained at least in part through the Internet.

6. A method as in claim 1, where said images are medical images.

7. A method as in claim 5, where said images are medical images.

8. A method as in claim 2, further comprising searching said database for at least one image or image-related document that satisfies at least one threshold established by said search query.

9. A method as in claim 2, where the database is located at a remote location and is searched by issuing a query to a database server through a data communications network.

10. A method as in claim 9, and further comprising retrieving an image or image-related document from the remote database.

11. A method as in claim 2, where the database is located at a remote location and is searched by issuing a query to a database server, through a data communications network, in conjunction with a text-based search algorithm executed by said server to retrieve a multi-media object from the database.

12. A method as in claim 2, where the database comprises image information obtained from a plurality of autonomous searchers that execute in client computers and that return URLs corresponding to located image data and image-related documents.

13. A method as in claim 1, further comprising segmenting an image by employing texture segmentation, color segmentation and foreground/background segmentation.

14. A method as in claim 13, where texture segmentation includes forming a texture map.

15. A method as in claim 14, where color segmentation includes forming a region map.

16. A method as in claim 15, where foreground/background segmentation includes using the texture map and the region map to determine if there is at least one dominant object in the image, and forming an object map.

17. A method as in claim 16, where the image, texture map, region map and object map are further processed to compute, for each region in the region map and for each object in the object map, a set of local features comprising information descriptive of size, boundary, neighbors, boundary curvature, texture, mean color, color name, and shape properties for that region/object.

18. A method as in claim 13, where local features are analyzed to compute regional features for indicating a presence of semantic cues.

19. A method as in claim 18, where said semantic cues comprise at least one of skin, sky, grass, snow, nature, water, man-made objects, man-made blobs, flowers and texture.

20. A method as in claim 13, where local features are combined to compute global features that describe a global appearance of the image.

21. A method as in claim 20, where said global features comprise at least one of number of details, histogram of color names, description of color composition, number of regions, number of blobs, number of objects, number of straight lines, number of regular curves, energy, spatial activity, symmetry and measures of contrast.

22. A data processing system comprising a data processor, a graphical user interface and a memory that stores a database containing image information, where the image information comprises at least one of already classified images, network locations of already classified images and documents containing already classified images, said data processor operating in accordance with a stored program for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories;

said data processor, for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features;

comparing an image to said semantic categories; and

classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image, said data processor storing image-related classification data in said database.

23. A data processing system as in claim 22, where said database is searched for images matching an input query.

24. A data processing system as in claim 23, where said input query comprises an input image, and where data processor extracts features from said input image and applies a metric to determine the semantic category for said input image.

25. A data processing system as in claim 23, where said input query comprises an input image, and where said data processor processes said input image and compares said input image to each semantic category using a metric that computes a similarity measure between the features used to describe the semantic category and corresponding features extracted from said input image;

and assigns said input image to the semantic category that corresponds to a highest value of the similarity measure.

26. A data processing system as in claim 22, where said image information is obtained at least in part through the Internet.

27. A data processing system as in claim 22, where said image information is comprised of medical image information.

28. A data processing system as in claim 22, where said image information comprises data obtained from a plurality of autonomous searchers that execute in client computers and that return URLs corresponding to located images.

29. A data processing system as in claim 23, where said data processor searches said database for locating at least one image that satisfies thresholds established by said search query.

30. A data processing system as in claim 23, where the database is located at a remote location and is searched by issuing a query to a database server through a data communications network.

31. A data processing system as in claim 30, and further comprising retrieving an image through said data communications network.

32. A data processing system as in claim 23, where said database is located at a remote location and is searched by issuing a query to a database server, through a data communications network, in conjunction with a text-based search algorithm executed by said server to retrieve a multi-media object.

33. A data processing system as in claim 23, where said image information comprises medical image information and is searched by issuing a query to a database server through the Internet.

34. A data processing system as in claim 23, where said data processor segments an input image by employing texture segmentation, color segmentation and foreground/background segmentation.

35. A data processing system as in claim 34, where texture segmentation includes forming a texture map.

36. A data processing system as in claim 35, where color segmentation includes forming a region map.

37. A data processing system as in claim 36, where foreground/background segmentation includes using the texture map and the region map to determine if there is at least one dominant object in the image, and to form an object map.

38. A data processing system as in claim 37, where the input image, texture map, region map and object map are further processed to compute for each region in the region map and for each object in the object map, a set of local features comprising information concerning size, boundary, neighbors, boundary curvature, texture, mean color, color name, and shape properties for that region/object.

39. A data processing system as in claim 34, where local features are analyzed to compute regional features, indicating a presence of semantic cues.

40. A data processing system as in claim 39, where said semantic cues comprise at least one of skin, sky, grass, snow, nature, water, man-made objects, man-made blobs, flowers and texture.

41. A data processing system as in claim 39, where local features are combined to compute global features that describe a global appearance of the image.

42. A data processing system as in claim 41, where said global features comprise at least one of number of details, histogram of color names, description of color composition, number of regions, number of blobs, number of objects, number of straight lines, number of regular curves, energy, spatial activity, symmetry and measures of contrast.

43. A computer program embodied on a computer readable media for directing a computer to execute a method for processing digitally represented images, comprising program instructions for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories, where for each semantic category said program instructions form a set of the perceptual features comprising required features and frequently occurring features, compares an image to said semantic categories and classifies said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image.

44. A computer program as in claim 43, where classified images are stored in a database, and where classified images or network locations of classified images are added to said database after being assigned to one of said semantic categories.

45. A computer program as in claim 43, were said images are obtained at least in part through the Internet.

46. A computer program as in claim 44, further comprising searching said database for at least one image or image-containing document that satisfies thresholds established by a search query.

47. An Internet searching tool, comprising:

an Internet searching robot comprising at least one autonomous searcher that locates images and returns at least one of image location data and image-related document location data, said searcher executing in a client computer;

a processor responsive to returned image location data and image-related document location data for accessing a located image for segmenting the located image and for extracting features from the located image for obtaining a set of image metadata;

storage for storing the set of image metadata; and

a user interface for entering a user query for use in searching the storage and for returning an image having image metadata that is relevant to the user query.

48. An Internet searching tool as in claim 47, where said user query comprises an image.

49. An Internet searching tool as in claim 47, where said user query comprises text.

50. An Internet searching tool as in claim 47, where there are a plurality of client computers and a corresponding plurality of searching robots individual ones of which execute in individual ones of said client computers.

51. An Internet searching tool as in claim 47, where said processor is responsive to the user query for segmenting an input query image and for extracting features from the input query image for obtaining a set of input image metadata, said processor using the input image metadata when searching the storage.

52. An Internet searching tool as in claim 51, where said processor derives a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories.

53. A method for searching the Internet, comprising:

providing at least one server accessible by a plurality of clients;

downloading a program to a client, said program comprising an Internet searching program operable for locating at least one type of data; and

executing said downloaded program on the client, and returning to said server URLs corresponding to located data.

54. A method as in claim 53, where the at least one type of data comprises image data.

55. A method as in claim 53, where said URLs comprise URLs of located digital images, and URLs of image related documents.

56. A system for obtaining information from the Internet, comprising at least one server accessible by a plurality of clients, said server operable for downloading a program to a client, said program comprising an Internet searching program operable for locating at least one type of data, said downloaded program being executable by said client and returning to said server URLs corresponding to located data, said server further comprising a database, searchable by the clients, for storing information corresponding to information returned by said downloaded program.

57. A system as in claim 56, where the at least one type of data comprises image data.

58. A system as in claim 56, where said URLs comprise URLs of located digital images, and URLs of image related documents.