Stereo-based image processing

• US 20060193509A1
• Filed: 02/25/2005
• Published: 08/31/2006
• Est. Priority Date: 02/25/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Cross, Geoffrey, Blake, Andrew
• Granted Patent: US 7,512,262 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06V 40/164 using holistic features

Abstract

Images of the same scene from multiple cameras may be use to generate a stereo disparity map. At least a portion of the stereo disparity map may be compared to a kernel image to detect and/or determine the location of an object in the disparity map. The kernel image is an array of pixel values which represent the stereo disparity of an object to be located, more particularly, the kernel image indicates the 3-dimensional surface shape of the object to be located from a point of view. The disparity map containing the located object may be process to manipulate the display of the stereo-based image and/or an input image. For example, the display of the image may be cropped and/or zoomed, areas of the image that are not the located object may be modified, an object such as an emoticon or smart-emoticon may be virtually inserted into the three dimensions of the image and may interact with the object, the location of the object in the image may localize further searches, presence of the located object in the image may indicate selected storing of the image and/or image indexing, and/or the located object in the image may be used as a non-standard input device to a computing system.

Claims

1. A method comprising:

a) determining a disparity map based on a first image and a second image, the disparity map containing a first disparity value of a pixel representing a stationary object in the first image and the second image;

b) comparing a first predetermined kernel image to an area portion of the disparity map including the first disparity value, the first kernel image containing at least one first disparity value representing a first object; and

c) computing a score based on the comparison of the first kernel image and the area portion.

2. The method of claim 1, further comprising indicating an object in a stereo-based image based on the first image and the second image, based on the score.

3. The method of claim 1, wherein the disparity map contains a plurality of disparity values, the method further comprising comparing each of the plurality of disparity values to at least one disparity value of the first kernel image.

4. The method of claim 1, further comprising determining a first depth of the first disparity value.

5. The method of claim 4, further comprising modifying a size of the first kernel image based on the first depth.

6. The method of claim 5, further comprising selecting the area portion of the disparity map based on the size of the kernel image.

7. The method of claim 1, further comprising cropping a stereo-based image based on the first image and the second image based on the score.

8. The method of claim 7, further comprising digitally zooming the cropped stereo-based image based on a depth of at least a portion of the area portion.

9. The method of claim 1, further comprising inserting a smart-emoticon into a stereo-based image based on the first image and the second image, and displaying the smart-emoticon at least partially occluded by an object in the stereo-based image based on a second disparity value of the disparity map.

10. The method of claim 1, further comprising displaying a stereo-based image based on the first image and the second image, at least a portion of a background of the displayed stereo-based image being modified, the background excluding an object portion of the stereo-based image associated with the area portion of the disparity map.

11. The method of claim 1, further comprising localizing a search for features of the first image and the second image based on the score.

12. The method of claim 1, further comprising comparing at least a portion of the area portion with a predetermined second kernel image, the second kernel image containing at least one second disparity value representing a second object;

13. The method of claim 1, further comprising, based on the score, associating a first object identifier with at least one the first image, the second image, and a stereo-based image based on the first image and the second image.

14. The method of claim 1, further comprising determining a location of a top of an object represented in the are portion, and re-selecting a center point of the object based on the location of the top of the object and a depth of at least a portion of the object.

15. One or more computer readable media containing computer executable instructions that, when implemented, perform a method comprising:

a) determining a disparity map based on a first image and a second image, the disparity map containing a disparity value of a pixel representing a scene object in the first image and the second image;

b) comparing a predetermined kernel image to an area portion of the disparity map including the disparity value, the kernel image containing at least one disparity value representing a comparison object; and

c) computing a score based on the comparison of the kernel image and the area portion, the score being based on at least one of a previous position of the scene object in a prior disparity map, a distance of the pixel from a center of the first image and/or the second image, and the disparity value.

16. The one or more computer readable media of claim 15, wherein computing the score includes computing a score based on a digital signal processing cost function.

17. The one or more computer readable media of claim 16, wherein computing the score includes computing a linear function of the digital signal processing cost function and at least one of a previous position of the scene object in a prior disparity map, a distance of the pixel from a center of the first image and/or the second image, and the disparity value.

18. One or more computer readable media containing executable components comprising:

a) means for generating a disparity map based on a first image and a second image, the disparity map containing a disparity value for each matched pixel of the first image and the second image;

b) means for comparing a predetermined kernel image to an area portion of the disparity map, the kernel image containing at least one disparity value representing a comparison object; and

b) means for computing a score based on the comparison of the kernel image and the area portion.

19. The one or more computer readable media of claim 18, further comprising means for clamping a top of an object in the comparison area.

20. The one or more computer readable media of claim 18, further comprising means for performing an action based on a computed score of the area portion.