Image interpretation method and apparatus

• US 6,922,489 B2
• Filed: 10/29/1998
• Issued: 07/26/2005
• Est. Priority Date: 10/29/1997
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Information Systems Research Australia Pty Limited (Canon Inc.), Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Le, Delphine Anh Dao, Lennon, Alison Joan
• Primary Examiner(s): Dastouri, Mehrdad
• Application Number:
  US09/182,006
• Publication Number:
  US 20020034330A1
• Time in Patent Office: 2,462 Days
• Field of Search: 382/228, 382/229, 382/181, 382/190, 382/224, 382/225, 382/232, 704/256, 707/104, 707/500
• US Class Current: 382/228
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries

Abstract

A method of interpreting an image using a statistical or probabilistic interpretation model is disclosed. The image has associated therewith contextual information. The method comprises the following steps: providing the contextual information associated with the image for analysis; analyzing the additional contextual information to identify predetermined features relating to the image; and biasing the statistical or probabilistic interpretation model in accordance with the identified features.

Claims

1. A method of selecting a semantic label to describe a portion of an image, using a statistical or probabilistic interpretation model, said method comprising the steps of:

providing one or more distinct semantic labels;

determining an initial probability value for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing contextual information simultaneously with image data representing the image, the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

analyzing the contextual information captured with the image data to identify predetermined features relating to the image;

adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features; and

analysing at least the portion of the image using the interpretation model in order to select at least one of the distinct semantic labels to describe the portion of the image, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

2. The method as recited in claim 1, further including the step of:

interpreting the image in accordance with said at least one semantic label.

3. The method as recited in claim 1, further comprising the step of providing metadata including the selected at least one label describing at least a portion of the content of the image.

4. The method as recited in claim 3, wherein the metadata is an encoded representation providing information about at least a portion of the content of the image.

5. The method as recited in claim 3, wherein the metadata includes information regarding adjacency of regions of the image.

6. The method as recited in any one of claims 3 to 5, wherein said metadata is associated with image data.

7. The method as recited in claim 6, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

8. The method as recited in claim 1, further comprising the steps of

segmenting the image into substantially homogeneous regions;

analyzing the regions to provide a Region Adjacency Graph (RAG); and

associating at least one of the one or more distinct semantic labels with the Region Adjacency Graph to provide a labeled Region Adjacency Graph.

9. The method as recited in claim 1 or 5, wherein the contextual information originated from any one or more of the following types of information sources:

audio information;

electromagnetic spectral information;

environmental information;

time lapse or sequential image information; and

textual annotation, phrases or keywords associated with the image.

10. The method as recited in claim 9, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

11. The method as recited in claim 1, wherein the adjusting step includes selecting a predetermined application domain from a plurality of application domains based on the features identified, wherein said predetermined application domain has an associated set of semantic labels and associated default probabilities.

13. The method as recited in any one of claims 1, 2, 3-8, 11, or 12, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

12. A method of selecting a semantic label to describe an object in an image, using a statistical or probabilistic interpretation model, said method comprising the steps of:

providing one or more distinct semantic labels;

determining an initial probability value for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing contextual information simultaneously with image data representing the image, the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

analyzing the contextual information captured with the image data to identify predetermined features relating to the image;

adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features;

analyzing at least a portion of the image comprising the object using the statistical or probabilistic interpretation model; and

selecting at least one of the distinct semantic labels to describe the object in the image based on the analysis of the image, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

14. An image interpretation system for selecting a semantic label to describe an object of an image, the image comprising a plurality of objects, the system comprising:

initial probability determining means for determining an initial probability values for substantially each of one or more predetermined semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing means for capturing image data representing the image simultaneously with contextual information the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

analyzing means for analyzing the captured contextual information captured with the image data to identify predetermined features of the information;

initial probability adjusting means for adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features; and

image data analyzing means for analysing at least a portion of the image comprising the object using a statistical or probabilistic interpretation model in order to select at least one of the distinct semantic labels to describe the object in the image, wherein said at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

15. The image interpretation system recited in claim 14, further comprising labelling means for providing semantic labels for each object identified.

16. A computer readable medium including instruction modules arranged to select a semantic label to describe an object in an image, using a statistical or probabilistic interpretation model, said modules comprising:

an initial probability determining module for determining an initial probability value for substantially each of one or more predetermined semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

a capturing module for capturing image data representing the image simultaneously with contextual information, the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

an analyzing module for identifying, from the contextual information corresponding to the captured image data, predetermined features relating to the image;

initial probability adjusting module for adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features; and

an image data analyzing module for analysing at least a portion of the image comprising the object using the statistical or probabilistic interpretation model, in order to select at least one of the distinct semantic labels to describe the object, wherein said at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

17. The computer readable medium recited in claim 16, further including a labelling module for providing metadata describing the identified objects of the image.

18. An apparatus for selecting a semantic label to describe a portion of an image using a statistical or probabilistic interpretation model, said apparatus comprising:

means for providing one or more distinct semantic labels;

means for determining an initial probability value for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing means for capturing contextual information simultaneously with image data representing the image, the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

means for analyzing the contextual information captured with the image data to identify predetermined features relating to the image;

means for adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features; and

means for analysing at least the portion of the image using the interpretation model in order to select at least one of the distinct semantic labels to describe the portion of the image, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

19. The apparatus as recited in claim 18, further including:

means for interpreting the image in accordance with said at least one semantic label.

20. The apparatus as recited in claim 18, further comprising the step of providing metadata including said selected labels describing at least a portion of the content of the image.

21. The apparatus as recited in claim 20, wherein the metadata is an encoded representation providing information about at least a portion of the content of the image.

22. The apparatus as recited in claim 20, wherein the metadata includes information regarding adjacency of regions of the image.

23. The apparatus as recited in claim 18, further comprising:

means for segmenting the image into substantially homogeneous regions;

means for analyzing the regions to provide a Region Adjacency Graph (RAG); and

means for associating at least one of the one or more semantic labels with the Region Adjacency Graph to provide a labeled Region Adjacency Graph.

24. The apparatus as recited in any one of claims 20 to 22, wherein the metadata is associated with the image.

25. The apparatus as recited in claim 24, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

26. The apparatus as recited in claim 18 or 22, wherein the contextual information originated from any one or more of the following types of information sources:

audio information;

electromagnetic spectral information;

environmental information;

time lapse or sequential image information; and

textual annotation, phrases or keywords associated with the image.

27. The apparatus as recited in claim 26, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

28. The apparatus as recited in claim 18, wherein the means for adjusting includes a means for selecting a predetermined application domain from a plurality of application domains based on the features identified.

30. The apparatus as recited in any one of claims 18, 19, 20-23, 28, or 29, wherein the statistical or probabilistic interpretation model is a Markov Random Field Model.

29. An apparatus for selecting a semantic label to describe an object in an image, using a statistical or probabilistic interpretation model the apparatus comprising:

means for providing one or more distinct semantic labels;

means for determining an initial probability value for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

means for capturing contextual information simultaneously with image data representing the image, the contextual information being directly associated with the captured image data and originating from one or more different types of information sources, the information sources being distinct from the image and the captured image data representing the image;

means for analyzing the contextual information captured with the image data to identify predetermined features relating to the image;

means for adjusting the initial probability values for the one or more semantic labels based on the analysis of the captured contextual information and on the identified features;

means for analyzing at least a portion of the image data comprising the object using the statistical or probabilistic interpretation model; and

means for selecting at least one of the distinct semantic labels to describe the object in the image based on the image data analysis, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

31. A method of selecting a semantic label to describe a portion of an image, using a statistical or probabilistic interpretation model, said method comprising the steps of:

providing one or more distinct semantic labels;

determining an initial probability values for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing contextual information simultaneously with image data representing the image, the contextual information being directly associated with the captured image data and originating from one or more different types information sources, said information sources being distinct from the image and the captured image data representing the image;

analyzing the contextual information captured with the image data to identify predetermined features relating to the image;

selecting an application domain from a plurality of predetermined application domains based on the features identified, each of the predetermined application domains having a corresponding one or more of the semantic labels;

adjusting the initial probability values for the one or more semantic labels of the selected application domain based on the analysis of the contextual information and on the identified features; and

analysing at least the portion of the image using the interpretation model in order to select at least one of the distinct semantic labels to describe the portion of the image, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.

32. A method of selecting a semantic label to describe at least one portion of an image, using a statistical or probalistic interpretation model, said method comprising the steps of:

providing one or more distinct semantic labels;

determining an initial probability value for substantially each of the one or more semantic labels, each of said initial probability values being set to a predetermined default value for each of the one or more semantic labels;

capturing audio data simultaneously with image data representing the image;

analyzing the audio data captured with the image data to identify predetermined features relating to the image;

adjusting the initial probability values for the one or more semantic labels based on the analysis of the audio data and on the identified features; and

analysing at least the portion of the image using the interpretation model in order to select at least one of the distinct semantic labels to describe the portion of the image, wherein the at least one semantic label is selected by multiplying the adjusted initial probability value for each semantic label with a conditional probability density value determined for the portion of the image.