Representation and retrieval of images using context vectors derived from image information elements

• US 20040249774A1
• Filed: 06/14/2004
• Published: 12/09/2004
• Est. Priority Date: 09/20/1993
• Status: Active Grant

• Current Assignee: Robert Hecht-Nielsen, William R. Caid
• Original Assignee: Robert Hecht-Nielsen, William R. Caid
• Inventors: Caid, William R., Hecht-Nielsen, Robert
• Granted Patent: US 7,072,872 B2
• Time in Patent Office: Days
• US Class Current: 706/14
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells

Abstract

Image features are generated by performing wavelet transformations at sample points on images stored in electronic form. Multiple wavelet transformations at a point are combined to form an image feature vector. A prototypical set of feature vectors, or atoms, is derived from the set of feature vectors to form an “atomic vocabulary.” The prototypical feature vectors are derived using a vector quantization method, e.g., using neural network self-organization techniques, in which a vector quantization network is also generated. The atomic vocabulary is used to define new images. Meaning is established between atoms in the atomic vocabulary. High-dimensional context vectors are assigned to each atom. The context vectors are then trained as a function of the proximity and co-occurrence of each atom to other atoms in the image. After training, the context vectors associated with the atoms that comprise an image are combined to form a summary vector for the image. Images are retrieved using a number of query methods, e.g., images, image portions, vocabulary atoms, index terms. The user'"'"'s query is converted into a query context vector. A dot product is calculated between the query vector and the summary vectors to locate images having the closest meaning. The invention is also applicable to video or temporally related images, and can also be used in conjunction with other context vector data domains such as text or audio, thereby linking images to such data domains.

Claims

1. A computer-implemented method for retrieving objects, comprising the steps of:

receiving a text query;

generating a query context vector derived from a context vector of at least one word included in the text query;

comparing the query context vector with a plurality of summary vectors, each summary vector associated with object features derived from object data of one of the plurality of objects; and

retrieving at least one object having a summary vector similar to the query context vector.

2. The method of claim 1, further the step of:

retrieving at least one summary vector with an orientation in a vector space similar to an orientation of the query context vector in the vector space.

3. The method of claim 1, wherein said object is any of:

text, image, image data, and one or more image features.

4. The method of claim 1, wherein the orientation of a vector in the vector space is indicative of the meaning of the object with which the vector is associated, and wherein objects having similar meaning are associated with vectors having similar orientations in the vector space.

5. The method of claim 1, further comprising the step of:

determining the orientation of a vector in the vector space by the frequency of proximal co-occurrences of objects in a corpus of records, wherein objects that frequently proximally co-occur are associated with vectors having similar orientations in the vector space.

6. The computer-implemented method of claim 3, wherein the image data are pixel data.

7. The computer-implemented method of claim 3, wherein image features are derived from the images by wavelet transformations.

8. The computer-implemented method of claim 3, wherein image features are coefficients of wavelet transformations on the images.

9. The computer-implemented method of claim 1, wherein the summary vectors are oriented in a vector space, and wherein axes of the vector space not associated with selected objects.

10. A computer-implemented method for retrieving objects, comprising the steps of:

receiving a query image;

deriving at least one feature vector from image data of the query image;

generating a query context vector from one or more context vectors associated with the at least one feature vector;

comparing the query context vector with a plurality of summary vectors, each summary vector derived from context vectors associated with feature vectors derived from object data of one of the plurality of objects; and

retrieving at least one image having a summary vector similar to the query context vector.

11. The method of claim 10, futher comprising the step of:

retrieving at least one summary vector with an orientation in a vector space similar to an orientation of the query context vector in the vector space.

12. The method of claim 10, wherein said object is any of:

text, image, image data, and one or more image features.

13. The method of claim 10, wherein the orientation of a vector space in a vector space is indicative of the meaning of the object with which the vector is associated, and wherein objects having similar meaning are associated with vectors having similar orientations in the vector space.

14. The method of claim 10, further comprising the step of:

determining the orientation of a vector in the vector space by the frequency of proximal co-occurrences of objects in a corpus of records, wherein objects that frequently proximally co-occur are associated with vectors having similar orientations in the vector space.

15. The method of claim 12, wherein the image data are pixel data.

16. The method of claim 12, wherein the image features are derived from the images by wavelet transformations.

17. The method of claim 12, wherein the image features are coefficients of wavelet transformations on the images.

18. The method of claim 12, wherein the summary vectors are oriented in a vector space, and wherein axes of the vector space not associated with selected image features.

19. A computer-implemented method for training context vectors for objects within documents, comprising the steps of:

for each of a plurality of objects, generating a plurality of feature vectors from object data of the object;

for each object, associating each of the object'"'"'s feature vectors with a context vector;

for each object, aligning each of the context vectors of the object using a context vector of at least one word included in a document containing the object; and

aligning each of the context vectors of the object by adjusting the object context vector to be more similar to the summary vector of the at least one word included in the document containing the image.

20. The method of claim 19, wherein the object is any of:

text, image, image data, and one or more image features.

21. The method of claim 19, wherein the step of associating each of the object'"'"'s feature vectors with a context vector further comprises the step of:

initializing the context vectors to be substantially orthogonal to each other.

22. The method of claim 19, wherein the context vectors are oriented in a vector space, and wherein axes of the vector space not associated with selected objects or object features.

23. A computer-implemented method for training context vectors for objects within documents, comprising the steps of:

providing a plurality of word context vectors, each context vector having an orientation in a vector space, wherein words having similar meaning have context vectors with similar orientations in the vector space;

providing a plurality of object context vectors, each object context vector associated with a feature vector, each feature vector derived from object data of at least one object, each object context vector having an orientation in the vector space;

for each document containing an object, aligning the object context vectors associated with the feature vectors derived from the object, with a summary vector derived from context vectors of selected words contained in the document; and

wherein aligning the object context vectors with a summary vector derived from context vectors of selected words contained in the document by adjusting the object context vector to be more similar to the summary vector of the selected words included in the document.

24. The method of claim 23, wherein the object is any of:

text, image, image data, and one or more image features.

25. The method of claim 23, wherein the context vectors are oriented in the vector space, and wherein axes of the vector space not associated with selected terms or image features.

26. A computer-implemented method for retrieving records having different object types, the method comprising:

providing a plurality of first records, each first record having a first object type;

for each of the first record having the first object type, deriving from elements of the first record a context vector, the context vector having an orientation in a vector space;

providing a plurality of second records, each second record having a second object type;

for each of the second records having the second object type, deriving from elements of the second record a context vector, the context vector having an orientation in the vector space;

receiving a query, comprising at least one element of the first object type;

deriving a query context vector from the query; and

retrieving at least one second record having a context vector similar to the query context vector.

27. The computer-implemented method of claim 26, wherein the object is any of a first media type and a second media type.