Stereo-based image processing

• US 7,512,262 B2
• Filed: 02/25/2005
• Issued: 03/31/2009
• Est. Priority Date: 02/25/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Cross, Geoffrey, Blake, Andrew
• Primary Examiner(s): Strege; John B
• Application Number:
  US11/066,946
• Publication Number:
  US 20060193509A1
• Time in Patent Office: 1,495 Days
• Field of Search: 382/154, 382/190, 382/106
• US Class Current: 382/154
• CPC Class Codes: G06V 40/164 using holistic features

Abstract

Images of the same scene from multiple cameras may be use to generate a stereo disparity map. At least a portion of the stereo disparity map may be compared to a kernel image to detect and/or determine the location of an object in the disparity map. The kernel image is an array of pixel values which represent the stereo disparity of an object to be located, more particularly, the kernel image indicates the 3-dimensional surface shape of the object to be located from a point of view. The disparity map containing the located object may be process to manipulate the display of the stereo-based image and/or an input image. For example, the display of the image may be cropped and/or zoomed, areas of the image that are not the located object may be modified, an object such as an emoticon or smart-emoticon may be virtually inserted into the three dimensions of the image and may interact with the object, the location of the object in the image may localize further searches, presence of the located object in the image may indicate selected storing of the image and/or image indexing, and/or the located object in the image may be used as a non-standard input device to a computing system.

Claims

1. A method:

determining a disparity map based on a first image and a second image, the disparity map containing a first disparity value of a pixel representing a stationary object in the first image and the second image;

comparing a first predetermined kernel image to an area portion of the disparity map including the first disparity value, wherein the first predetermined kernel image contains at least one first disparity value representing a first object and wherein the first predetermined kernel image represents a modified image of a model person'"'"'s head and shoulders and the modified image has been asymmetrically expanded horizontally;

computing a score based on the comparison of the first predetermined kernel image and the area portion; and

indicating an object in a stereo-based image based on the first image and the second image, based on the score;

wherein the determining, comparing, computing and indicating are performed by one or more processors.

2. The method of claim 1, further comprising:

determining a first depth of the first disparity value.

3. The method of claim 1, wherein the disparity map contains a plurality of disparity values, the method further comprising comparing each of the plurality of disparity values to at least one disparity value of the first kernel image.

4. The method of claim 2, further comprising modifying a size of the first kernel image based on the first depth.

5. The method of claim 4, further comprising selecting the area portion of the disparity map based on the size of the kernel image.

6. The method of claim 1, further comprising cropping a stereo-based image that is based on the first image and the second image, based on the score.

7. The method of claim 6, further comprising digitally zooming the cropped stereo-based image based on a depth of at least a portion of the area portion.

8. The method of claim 1, further comprising displaying a stereo-based image based on the first image and the second image, at least a portion of a background of the displayed stereo-based image being modified, the background excluding an object portion of the stereo-based image associated with the area portion of the disparity map.

9. The method of claim 1, further comprising localizing a search for features of the first image and the second image based on the score.

10. The method of claim 1, further comprising comparing at least a portion of the area portion with a predetermined second kernel image, the second kernel image containing at least one second disparity value representing a second object.

11. The method of claim 1, further comprising, based on the score, associating a first object identifier with at least one of the first image, the second image, or a stereo-based image that is based on the first image and the second image.

12. The method of claim 1, further comprising determining a location of a top of an object represented in the area portion, and re-selecting a center point of the object based on the location of the top of the object and a depth of at least a portion of the object.

13. The method of claim 1, wherein in the modified image of the model person'"'"'s head and shoulders, the head but not the shoulders has been asymmetrically expanded horizontally.

14. The method of claim 13, comprising locating, based on the score, an object in a stereo-based image that is based on the first image and the second image, wherein the located object includes an image of a person'"'"'s head and an image of an object proximate to the person'"'"'s head.

15. The method of claim 1, comprising generating the first kernel image.

16. A method comprising:

determining a disparity map based on a first image and a second image, the disparity map containing a first disparity value of a pixel representing a stationary object in the first image and the second image;

comparing a first predetermined kernel image to an area portion of the disparity map including the first disparity value, the first predetermined kernel image containing at least one first disparity value representing a first object; and

computing a score based on the comparison of the first kernel image and the area portion; and

inserting a smart-emoticon into a stereo-based image based on the first image and the second image, and displaying the smart-emoticon at least partially occluded by an object in the stereo-based image based on a second disparity value of the disparity map.

17. One or more computer readable storage media comprising computer executable instructions that, when executed, cause a computer to perform a method comprising:

determining a disparity map based on a first image and a second image, the disparity map containing a disparity value of a pixel representing a scene object in the first image and the second image;

comparing a predetermined kernel image to an area portion of the disparity map including the disparity value, wherein the predetermined kernel image contains at least one disparity value representing a comparison object and wherein the predetermined kernel image represents a modified image of a model person'"'"'s head and shoulders wherein the head but not the shoulders has been asymmetrically expanded horizontally; and

computing a score based on the comparison of the predetermined kernel image and the area portion, the score being additionally based on at least one of a) a previous position of the scene object in a prior disparity map, b) at least one of a distance of the pixel from a center of the first image or a distance of the pixel from a center of the second image, or c) the disparity value; and

indicating an object in a stereo-based image based on the first image and the second image, based on the score.

18. The one or more computer readable storage media of claim 17, wherein computing the score includes computing a linear function of the digital signal processing cost function and at least one of a previous position of the scene object in a prior disparity map, a distance of the pixel from a center of the first image and/or the second image, and the disparity value.

19. The computer readable storage media of claim 17, wherein the media comprise instructions for causing the computer to generate the first kernel image.