Determining connectedness and offset of 3D objects relative to an interactive surface

• US 7,379,562 B2
• Filed: 03/31/2004
• Issued: 05/27/2008
• Est. Priority Date: 03/31/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D.
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): Hung; Yubin
• Application Number:
  US10/814,761
• Publication Number:
  US 20050226505A1
• Time in Patent Office: 1,518 Days
• Field of Search: 382/103, 382/106, 382/154, 382171-173, 382/180, 356/4.07
• US Class Current: 382/103
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A position of a three-dimensional (3D) object relative to a display surface of an interactive display system is detected based upon the intensity of infrared (IR) light reflected from the object and received by an IR video camera disposed under the display surface. As the object approaches the display surface, a “hover” connected component is defined by pixels in the image produced by the IR video camera that have an intensity greater than a predefined hover threshold and are immediately adjacent to another pixel also having an intensity greater than the hover threshold. When the object contacts the display surface, a “touch” connected component is defined by pixels in the image having an intensity greater than a touch threshold, which is greater than the hover threshold. Connected components determined for an object at different heights above the surface are associated with a common label if their bounding areas overlap.

Claims

1. A method of detecting a three-dimensional object adjacent to a user-interactive side of a surface that optically diffuses light, as a function of an infrared light transmitted toward the three-dimensional object from an opposite side of the surface and reflected back through the surface from the three-dimensional object to be received by a light sensor disposed on the opposite side of the surface comprising the steps of:

(a) creating a first pixilated image representing the intensity of the infrared light reflected from the three-dimensional object and received by the light sensor;

(b) creating a first binarized image from the first pixilated image by filtering out pixels of the first pixilated image that do not have an intensity exceeding a first threshold value, the first binarized image representing a first planar distance of the three-dimensional object from the interactive side of the surface;

(c) creating a second binarized image from the first pixilated image by filtering out pixels of the first pixilated image that do not have an intensity exceeding a second threshold value, the second binarized image having substantially equal area and coordinate locations as the first binarized image, the second binarized image representing a second planar distance of the three-dimensional obiect from the interactive side of the surface;

(d) detecting a first connected component in the first binarized image, the first connected component representing a first set of pixels that have an intensity exceeding the first threshold value, the first connected component having a first bounding area defined by a first outer boundary and within which the first set of pixels reside and within which no pixels of the first binarized image have an intensity that fails to exceed the first threshold value;

(e) determining the first bounding area and a first coordinate location of the first connected component in the first binarized image;

(f) detecting a second connected component in the second binarized image, the second connected component representing a second set of pixels that have an intensity exceeding the second threshold value, the second connected component having a second bounding area defined by a second outer boundary and within which the second set of pixels reside and within which no pixels of the second binarized image have an intensity that fails to exceed the second threshold value;

(g) determining the second bounding area and a second coordinate location of the second connected component in the second binarized image that has substantially equal area and coordinate locations as the first binarized image; and

(h) determining that both the first connected component and the second connected component correspond to the three-dimensional object adjacent to the interactive side of the surface, if one of the first connected component and the second connected component is disposed substantially within the bounding area of the other of the first connected component and the second connected component.

2. The method of claim 1, further comprising the steps of:

(a) associating a first label with the first connected component;

(b) associating a second label with the second connected component; and

(c) associating the first label with the second label to indicate that both the first connected component and the second connected component correspond to the three-dimensional object that is adjacent to the interactive side of the surface.

3. The method of claim 1, further comprising the steps of:

(a) placing the three-dimensional object on the surface prior to creating the first pixilated image, such that the first planar distance is substantially zero; and

(b) after creating the first pixilated image, determining a touch threshold value for pixels that should be included in the first pixilated image, corresponding to an intensity of the infrared light reflected from a portion of the three-dimensional object and received by the light sensor when said portion of the three-dimensional object is contacting the surface.

4. The method of claim 3, further comprising the step of setting the first threshold value to the touch threshold value prior to creating the first binarized image, thereby determining whether any portion of the three-dimensional object is contacting the surface.

5. The method of claim 3, further comprising the steps of:

(a) determining a hover threshold that is different from the touch threshold, the hover threshold corresponding to an intensity of the infrared light reflected by the three-dimensional object and received by the light sensor when the three-dimensional object is a determinable distance from the surface on the interactive side of the surface;

(b) prior to creating the second binarized image, setting the second threshold value to the hover threshold; and

(c) determining that the three-dimensional object is at least within a hover region away from the surface, the hover region comprising a volume between the surface and a plane that is parallel to the surface and is separated from the surface on the interactive side of the surface by the determinable distance.

6. The method of claim 5, wherein the step of determining the hover threshold is performed prior to creating the first binarized image and the second binarized image, the step of determining the hover threshold comprising the steps of:

(a) moving the three-dimensional object toward the surface on the interactive side of the surface from a distance sufficiently far from the surface so that an intensity of the infrared light reflected from the three-dimensional object and received by the light sensor initially is not above the first threshold value;

(b) creating a series of pixilated images over time as the three-dimensional object is moved toward the surface;

(c) detecting a plurality of calibration connected components, including one calibration connected component corresponding to the three-dimensional object for each of the series of pixilated images, each calibration connected component comprising a set of immediately adjacent pixels with a corresponding minimum pixel intensity for the infrared light received by the light sensor that is greater than zero for each of the sequence of pixilated images;

(d) determining which one of the plurality of calibration connected components has a maximum bounding area; and

(e) storing as the hover threshold the corresponding minimum pixel intensity that was determined for said one of the plurality of calibration connected components that has the maximum bounding area.

7. The method of claim 5, wherein the determinable distance is a function of an infrared light reflectivity property of the three-dimensional object.

8. The method of claim 5, further comprising the steps of:

(a) creating a second pixilated image an interval of time after creating the first pixilated image;

(b) creating a third binarized image from the second pixilated image by filtering out pixels of the second binarized image that do not have an intensity exceeding the hover threshold, so that the second binarized image and the third binarized image comprise a sequence of binarized images at the hover threshold;

(c) detecting a third connected component in the third binarized image;

(d) determining at least one of a third bounding area and a third location of the third connected component in the third binarized image; and

(e) computing at least one of a current distance from the surface, a change in distance, and a velocity of the three-dimensional object as a function of at least two of the second bounding area, the third bounding area, the second location, and the third location.

9. The method of claim 8, further comprising the steps of:

(a) predicting whether the three-dimensional object will contact the surface within a predefined period; and

(b) producing an indication that the three-dimensional object has already contacted the surface if the prediction indicates that the three-dimensional object will contact the surface within the predefined period, thereby reducing latency in providing the indication that the three-dimensional object has contacted the surface.

10. A computer-readable medium encoded with computer-executable instructions for carrying out the steps of claim 1.

11. A system for detecting a relative position of a three-dimensional object, the system comprising:

(a) a surface that diffuses light and has;

(i) an interactive side adjacent to which the three-dimensional object can be manipulated; and

(ii) an opposite side that is opposite the interactive side;

(b) a light source spaced away the opposite side of the surface, the light source emitting an infrared light that is transmitted through the surface to the interactive side of the surface;

(c) a light sensor disposed on the opposite side of the surface so as to sense infrared light reflected back from the three-dimensional object, through the surface;

(d) a processor in communication with the light sensor; and

(e) a memory in communication with the processor, the memory storing data and machine instructions that when executed by the processor cause a plurality of functions to be carried out, including;

(i) creating a first pixilated image representing the intensity of the infrared light reflected from the three-dimensional object and received by the light sensor;

(ii) creating a first binarized image from the first pixilated image by filtering out pixels of the first pixilated image that do not have an intensity exceeding a first threshold value, the first binarized image representing a first planar distance of the three-dimensional object from the interactive side of the surface;

(iii) creating a second binarized image from the first pixilated image by filtering out pixels of the first pixilated image that do not have an intensity exceeding a second threshold value, the second binarized image having substantially equal area and coordinate locations as the first binarized image, the second binarized image representing a second planar distance of the three-dimensional obiect from the interactive side of the surface;

(iv) detecting a first connected component in the first binarized image, the first connected component representing a first set of pixels that have an intensity exceeding the first threshold value, the first connected component having a first bounding area defined by a first outer boundary and within which the first set of pixels reside and within which no pixels of the first binarized image have an intensity that fails to exceeding the first threshold value;

(v) determining the first bounding area and a first coordinate location of the first connected component in the first binarized image;

(vi) detecting a second connected component in the second binarized image, the second connected component representing a second set of pixels that have an intensity exceeding the second threshold value, the second connected component having a second bounding area defined by a second outer boundary and within which the second set of pixels reside and within which no pixels of the second binarized image have an intensity that fails to exceed the second threshold value;

(vii) determining the second bounding area and a second coordinate location of the second connected component in the second binarized image that has substantially equal area and coordinate locations as the first binarized image; and

(viii) determining that both the first connected component and the second connected component correspond to the three-dimensional object adjacent to the interactive side of the surface, if one of the first connected component and the second connected components is disposed substantially within the bounding area of the other of the first connected component and the second connected component.

12. The system of claim 11, wherein the machine language instructions further cause the processor to:

(a) associate a first label with the first connected component;

(b) associate a second label with the second connected component; and

(c) associate the first label with the second label to indicate that both the first connected component and the second connected component correspond to the three-dimensional object that is adjacent to the interactive side of the surface.

13. The system of claim 11, wherein after placing the three-dimensional object on the surface prior to creating the first pixilated image, so that the first planar distance is substantially zero, and after creating the first pixilated image, the machine language instructions further cause the processor to determine a touch threshold value for pixels that should be included in the first pixilated image, corresponding to an intensity of the infrared light reflected from a portion of the three-dimensional object and received by the light sensor when said portion of the three-dimensional object is contacting the surface.

14. The system of claim 13, wherein the machine language instructions further cause the processor to set the first threshold value to the touch threshold value prior to creating the first binarized image, thereby determining whether any portion of the three-dimensional object is contacting the surface.

15. The system of claim 13, wherein the machine language instructions further cause the processor to:

(a) determine a hover threshold that is different from the touch threshold, the hover threshold corresponding to an intensity of the infrared light reflected by the three-dimensional object and received by the light sensor when the three-dimensional object is a determinable distance from the surface on the interactive side of the surface;

(b) prior to creating the second binarized image, set the second threshold value to the hover threshold; and

(c) determine that the three-dimensional object is at least within a hover region away from the surface, the hover region comprising a volume between the surface and a plane that is parallel to the surface and is separated from the surface on the interactive side of the surface by the determinable distance.

16. The system of claim 15, wherein the hover threshold is determined prior to creating the first binarized image and the second binarized image, and wherein the hover threshold is determined by:

(a) moving the three-dimensional object toward the surface on the interactive side of the surface from a distance sufficiently far from the surface so that an intensity of the infrared light reflected from the three-dimensional object and received by the light sensor initially is not above the first threshold value;

(b) creating a series of pixilated images over time as the three-dimensional object is moved toward the surface;

(c) detecting a plurality of calibration connected components, including one calibration connected component corresponding to the three-dimensional object for each of the series of pixilated images, each calibration connected component comprising a set of immediately adjacent pixels with a corresponding minimum pixel intensity for the infrared light received by the light sensor that is greater than zero for each of the sequence of pixilated images;

(d) determining which one of the plurality of calibration connected components has a maximum bounding area; and

(e) storing as the hover threshold the corresponding minimum pixel intensity that was determined for said one of the plurality of calibration connected components that has the maximum bounding area.

17. The system of claim 15, wherein the determinable distance is a function of an infrared light reflectivity property of the three-dimensional object.

18. The system of claim 15, wherein the machine language instructions further cause the processor to:

(a) create a second pixilated image an interval of time after creating the first pixilated image;

(b) create a third binarized image from the second pixilated image by filtering out pixels of the second binarized image that do not have an intensity exceeding the hover threshold, so that the second binarized image and the third binarized image comprise a sequence of binarized images at the hover threshold;

(c) detect a third connected component in the third binarized image;

(d) determine at least one of a third bounding area and a third location of the third connected component in the third binarized image; and

(e) compute at least one of a current distance from the surface, a change in distance, and a velocity of the three-dimensional object as a function of at least two of the second bounding area, the third bounding area, the second location, and the third location.

19. The system of claim 18, wherein the machine language instructions further cause the processor to:

(a) predict whether the three-dimensional object will contact the surface within a predefined period; and

(b) produce an indication that the three-dimensional object has already contacted the surface if the prediction indicates that the three-dimensional object will contact the surface within the predefined period, thereby reducing latency in providing the indication that the three-dimensional object has contacted the surface.

20. The system of claim 15, wherein the machine language instructions further cause the processor to respond to a determination that the three-dimensional object is at least within the hover region over a specific predefined portion of the surface, by displaying a predetermined graphic image at the specific predefined portion of the surface, enabling a user to contact a selection within the predetermined graphic image with the three-dimensional object, to select an option displayed there on the surface.