Calibration of an interactive display system

• US 20060007170A1
• Filed: 06/16/2004
• Published: 01/12/2006
• Est. Priority Date: 06/16/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Yee, Dawson L.
• Granted Patent: US 7,432,917 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

In an interactive display system, a projected image on a display surface and a vision system used to detect objects touching the display surface are aligned, and optical distortion of the vision system is compensated. Also, calibration procedures correct for a non-uniformity of infrared (IR) illumination of the display surface by IR light sources and establish a touch threshold for one or more uses so that the interactive display system correctly responds to each user touching the display surface. A movable IR camera filter enables automation of the alignment of the projected image and the image of the display surface and help in detecting problems in either the projector or vision system.

Claims

1. A method for calibrating and compensating at least one error in an interactive display system that includes a projector for projecting images on a display surface as defined by a software program, a light source for illuminating an object on the display surface with light, and a vision sensing system having an imaging sensor for receiving light reflected back from the display surface by an object in contact with the display surface, said method including the steps of:

(a) receiving a signal from the imaging sensor, said signal varying in response to an intensity of the light reflected from an object in contact with the display surface;

(b) automatically compensating the signal received from the imaging sensor for the at least one error on the image produced by the imaging sensor, yielding a compensated signal;

(c) aligning at least one of the vision sensing system and the projector relative to the other and to the display surface; and

(d) employing the compensated signal as an input to the software program that defines the images projected on the display surface, so that the software program accurately detects a location of an object on the display surface relative to the images projected on the display surface by the projector.

2. The method of claim 1, wherein the at least one error includes a barrel distortion caused by a lens on the imaging sensor, for imaging the light reflected by an object contacting the display surface, so that the step of automatically compensating comprises the step of mapping points on the display surface to corresponding points in a barrel distortion image produced by the imaging sensor.

3. The method of claim 1, wherein the at least one error includes a trapezoidal distortion caused by the imaging sensor being disposed at an off-axis location relative to the display surface, for imaging the light reflected by any object contacting the display surface, so that the step of compensating comprises the step of mapping points on the display surface to corresponding points in a trapezoidal distortion image produced by the imaging sensor.

4. The method of claim 1, wherein the step of automatically compensating comprises the step of normalizing an intensity of the light reflected from an object in contact with the display surface to compensate for variations in the intensity of the light at the imaging sensor, including variations caused by the light reflected from an object following paths of differing length in reaching the imaging sensor from the light source.

5. The method of claim 4, wherein the step of normalizing comprises the steps of:

(a) moving a reflective sheet over substantially all of the display surface while automatically detecting a maximum intensity and a minimum intensity of the light reflected from the reflective sheet with the imaging sensor; and

(b) thereafter, applying a normalizing function to the signal from the imaging sensor in response to light reflected from any object in contact with the display surface, the normalizing function being determined based upon the maximum intensity and the minimum intensity that were detected.

6. The method of claim 1, wherein the light source produces infrared (IR) light and a movable IR filter is disposed between the display surface and the imaging sensor in a normal configuration, the IR filter preventing visible light from reaching the imaging sensor so that the imaging sensor responds substantially only to IR light reflected from an object on the display surface when detecting the object, the step of calibrating comprising the steps of:

(a) moving the IR filter out from between the display surface and the imaging sensor, placing the vision system into a calibration configuration in which the imaging sensor receives and responds to visible light from the display surface;

(b) projecting an image onto the display surface with the projector, with all color components in the image set to predetermined levels;

(c) automatically detecting whether the signal from the imaging sensor for an image of the display surface corresponds to all color components for the image being substantially equal to expected values that correspond to the predetermined levels; and

if not,

(d) indicating that an error exists in one of the projector and the vision system.

7. The method of claim 6, further comprising the steps of:

(a) projecting a template onto the display screen with the projector; and

(b) automatically detecting whether the signal from the imaging sensor indicates that an image of the template on the display surface is substantially equal to the template; and

if not,

(c) indicating that an error exists and that the error be reduced by making an alignment adjustment to at least one of the projector and the imaging sensor.

8. The method of claim 6, further comprising the step of positioning a different filter between the display surface and the imaging sensor, so that the different filter modifies the light from the display surface received by the imaging sensor while the template is projected onto the display surface.

9. The method of claim 8, wherein the different filter compensates for differences in a focal point of the image sensor when imaging using IR light and visible light.

10. The method of claim 1, wherein the step of automatically compensating comprises the steps of:

(a) illuminating a periphery region around the display surface;

(b) determining an extent of the display surface in an image of the display surface produced by the imaging sensor in response to light reflected from the periphery region;

(c) projecting a plurality of location targets on the display surface with the projector, at predefined locations relative to the display surface;

(d) imaging the location targets projected on the display surface with the imaging sensor; and

(e) automatically determining a calibration for the signal received from the imaging sensor based on an image of the location targets produced by the imaging sensor relative to the extents of the display surface in the image, so that after the calibration is applied, the location targets are disposed at expected positions within the extents of the display surface.

11. The method of claim 1, further comprising the step of determining a touch threshold for light reflected from an appendage of a user touching the display surface, to accommodate variations in light reflectivity for different users.

12. The method of claim 11, wherein the step of determining the touch threshold comprises the steps of:

(a) indicating a region on the display surface to be touched by the user;

(b) determining if an increase in an intensity of points within region detected by the imaging sensor is consistent with the user touching the region; and

if so,

(c) setting the touch threshold equal to an intensity of light detected after the increase; and

(d) refining the touch threshold by repetitively detecting the intensity of light reflected following a substantial increase in the intensity each time the user is likely to have touched the region, said touch threshold being used in software applications that respond to a user touching the display surface.

13. The method of claim 12, further comprising the step of determining if a shape represented by the points in the region for which the intensity increased substantially is consistent with a shape expected for the user touching the region, before enabling the step of setting the touch threshold to be carried out.

14. The method of claim 12, wherein a different touch threshold is determined for each of a plurality of different users of the interactive display system.

15. A memory medium storing machine executable instructions for carrying out the steps of claim 1.

16. An interactive display system, comprising:

(a) a display surface on which images are displayed and on which objects are detected;

(b) a light source for illuminating an object on the display surface with light;

(c) a vision sensing system having an imaging sensor for sensing a object in contact with the display surface by sensing the light reflected back from the object;

(d) a projector for projecting images on the display surface;

(e) a processor coupled to the projector and the vision sensing system; and

(f) a memory storing machine instructions including a software application and procedures for calibrating and compensating said at least one error in the interactive display system, by carrying out a plurality of functions, including;

(i) receiving a signal from the imaging sensor, said signal varying in response to an intensity of the light reflected from an object in contact with the display surface;

(ii) automatically compensating the signal received from the imaging sensor for the at least one error on the image produced by the imaging sensor, yielding a compensated signal;

(iii) facilitating alignment of at least one of the vision sensing system and the projector relative to the other and to the display surface; and

(iv) employing the compensated signal as an input to the software program that defines images projected on the display surface, so that the software program accurately detects a location of an object on the display surface relative to the images projected on the display surface by the projector.

17. The interactive display system of claim 16, wherein the at least one error includes a barrel distortion caused by a lens on the imaging sensor, for imaging the light reflected by an object contacting the display surface, so that the step of automatically compensating comprises the step of mapping points on the display surface to corresponding points in a barrel distortion image produced by the imaging sensor.

18. The interactive display system of claim 16, wherein the at least one error includes a trapezoidal distortion caused by the imaging sensor being disposed at an off-axis location relative to the display surface, for imaging the light reflected by any object contacting the display surface, so that the step of compensating comprises the step of mapping points on the display surface to corresponding points in a trapezoidal distortion image produced by the imaging sensor.

19. The interactive display system of claim 16, wherein the machine instructions further cause the processor to normalize an intensity of the light reflected from an object in contact with the display surface to compensate for variations in the intensity of the light at the imaging sensor, including variations caused by the light reflected from an object following paths of differing length in reaching the imaging sensor from the light source.

20. The interactive display system of claim 19, wherein to normalize the intensity, the machine instructions further cause the processor to automatically detect a maximum intensity and a minimum intensity of light reflected from a reflective sheet with the imaging sensor as the reflective sheet is moved over substantially all of the display surface.

21. The interactive display system of claim 16, wherein the light source produces infrared (IR) light, further comprising a movable IR filter that is disposed between the display surface and the imaging sensor in a normal configuration, the IR filter preventing visible light from reaching the imaging sensor so that the imaging sensor responds substantially only to IR light reflected from an object on the display surface when detecting the object, the IR filter being moved out from between the display surface and the imaging sensor, placing the vision system into a calibration configuration in which the imaging sensor receives and responds to visible light from the display surface, the machine instructions further causing the processor to:

(a) project an image onto the display surface with the projector, with all color components in the image set to predetermined levels;

(b) automatically detect whether the signal from the imaging sensor for an image of the display surface corresponds to all color components for the image being substantially equal to expected values that correspond to the predetermined levels; and

if not,

(c) indicate that an error exists in one of the projector and the vision system.

22. The interactive display system of claim 21, wherein the machine instructions further cause the processor to:

(a) project a template onto the display screen with the projector; and

(b) automatically detect whether the signal from the imaging sensor indicates that an image of the template on the display surface is substantially equal to the template; and

if not,

(c) indicate that an error exists that the error be reduced by making an alignment adjustment to at least one of the projector and the imaging sensor.

23. The interactive display system of claim 21, further comprising a different filter that is selectively positionable between the display surface and the imaging sensor, so that the different filter modifies the light from the display surface received by the imaging sensor while the template is projected onto the display surface.

24. The interactive display system of claim 23, wherein the different filter compensates for differences in a focal point of the image sensor when imaging using IR light and visible light.

25. The interactive display system of claim 16, wherein the machine instructions further cause the processor to:

(a) illuminate a periphery region around the display surface with one of the projector and the light source;

(b) determine a boundary of the display surface in an image of the display surface produced by the imaging sensor in response to light reflected from the periphery region;

(c) project a plurality of location targets on the display surface with the projector;

(d) image the location targets projected on the display surface with the imaging sensor; and

(e) automatically determining a calibration for the signal received from the imaging sensor based on an image of the location targets produced by the imaging sensor relative to the extents of the display surface in the image, so that after the calibration is applied, the location targets are disposed at expected positions within the extents of the display surface.

26. The interactive display system of claim 16, wherein the machine instructions further cause the processor to determine a touch threshold for light reflected from an appendage of a user touching the display surface, to accommodate variations in light reflectivity for different users.

27. The interactive display system of claim 26, wherein to determine the touch threshold, the machine instructions further cause the processor to:

(a) indicate a region on the display surface to be touched by a user;

(b) determine if an increase in an intensity of points within region detected by the imaging sensor is consistent with a user touching the region; and

if so,

(c) set the touch threshold equal to an intensity of light detected after the increase; and

(d) refine the touch threshold by using the imaging sensor for repetitively detecting the intensity of light reflected following a substantial increase in the intensity each time the user is likely to have touched the region, said touch threshold being used in software applications that respond to a user touching the display surface.

28. The interactive display system of claim 27, the machine instructions further cause the processor to determine if a shape represented by the points in the region for which the intensity increased substantially is consistent with a shape expected for the user touching the region, before enabling the step of setting the touch threshold to be carried out.

29. The interactive display system of claim 27, wherein a different touch threshold is determined for each of a plurality of different users of the interactive display system.

30. A method for calibrating a touch threshold for an interactive display system that responds to users touching a display surface on which images are projected by sensing light reflected from where the user touches the display surface, comprising the steps of:

(a) detecting a likely touch of a region on the display surface by a user, based upon criteria related to an image of the display surface that indicate the user has touched the region;

(b) detecting an intensity of reflected light for points in the image within the region where it is determined that the user likely touched the display surface; and

(c) setting the touch threshold equal to the intensity of the points within the region where it is determined that the user likely touched the display surface.

31. The method of claim 30, further comprising the step of refining the touch threshold based on statistical data determined by repeating steps (b) and (c) over time.

32. The method of claim 30, further comprising the step of projecting an image onto the display surface indicating the region where the user is to touch the display surface.

33. The method of claim 30, wherein the step of setting the threshold is only done if a change in the intensity of the points in the image within the region is consistent with the user touching the region, the change in intensity comprising one criterion.

34. The method of claim 30, wherein the step of setting the threshold is only done if a shape represented by the point in the image within the region is consistent with an expected shape, the expected shape comprising one criterion.

35. The method of claim 30, wherein -a different touch threshold is determined for each of a plurality of different users of the interactive display system.

36. The method of claim 35, wherein the light that is reflected from where the user is touching the display surface is infrared (IR) light.

37. The method of claim 36, wherein the plurality of different users can have different touch thresholds because of differences between a reflectivity of skin of each different user and because of differences in a size of a touch of the users on the display surface, further comprising the step of determining which user is likely to be touching the display surface so that the touch threshold for said user is employed when detecting whether said user is touching the display surface.

38. The method of claim 30, further comprising the step of requiring the user to touch the region to initiate a desired action on the interactive display system, to determine when it is likely that the user is touching the display surface within the region.

39. A memory medium storing machine executable instructions for carrying out the steps of claim 30.

40. An interactive display system in which provision is made for calibrating a touch threshold, comprising:

(a) a display surface on which images are displayed and on which objects are detected;

(b) a light source for illuminating an object on the display surface with light;

(c) a vision sensing system having an imaging sensor for sensing a object in contact with the display surface by sensing the light reflected back from the object;

(d) a projector for projecting images on the display surface;

(e) a processor coupled to the projector and the vision sensing system; and

(f) a memory storing machine instructions for carrying out a plurality of functions, including;

(i) detecting a likely touch of a region on the display surface by a user, based upon criteria related to an image of the display surface made by the imaging sensor that indicate the user has touched the region;

(ii) detecting an intensity of reflected light for points in the image within the region where it is determined that the user likely touched the display surface; and

(iii) setting the touch threshold equal to the intensity of the points within the region where it is determined that the user likely touched the display surface.

41. The interactive display system of claim 40, wherein the machine instructions further cause the processor to refine the touch threshold based on statistical data determined by repeating steps (ii) and (iii) over time.

42. The interactive display system of claim 40, wherein the machine instructions further cause the processor to use the projector to project an image onto the display surface indicating the region where the user is to touch the display surface.

43. The interactive display system of claim 40, wherein the machine instructions further cause the processor to set the threshold only if a change in the intensity of the points in the image within the region is consistent with the user touching the region, the change in intensity comprising one criterion.

44. The interactive display system of claim 40, wherein the machine instructions further cause the processor to set the threshold only if a shape represented by the point in the image within the region is consistent with an expected shape, the expected shape comprising one criterion.

45. The interactive display system of claim 40, wherein the machine instructions further cause the processor to determine a different touch threshold for each of a plurality of different users of the interactive display system.

46. The interactive display system of claim 45, wherein the light source produces infrared light so that IR light that is reflected from where the user is touching the display surface.

47. The interactive display system of claim 46, wherein the plurality of different users can have different touch thresholds because of differences between a reflectivity of skin of each different user and because of differences in a size of a touch of the users on the display surface, and wherein the machine instructions further cause the processor to determine which user is likely to be touching the display surface so that the touch threshold for said user is employed when detecting whether said user is touching the display surface.

48. The interactive display system of claim 40, wherein the machine instructions further cause the processor to require a user to touch the region to initiate a desired action on the interactive display system, to determine when it is likely that the user is touching the display surface within the region.