Multi-touch sensing display through frustrated total internal reflection

• US 8,441,467 B2
• Filed: 08/03/2007
• Issued: 05/14/2013
• Est. Priority Date: 08/03/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: Han, Jefferson Y.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Hegarty, Kelly B
• Application Number:
  US11/833,908
• Publication Number:
  US 20080029691A1
• Time in Patent Office: 2,111 Days
• Field of Search: 345/173, 345/175, 345/179, 345/176, 178/18.01, 178/18.09, 463/37, 250/221, 250/224, 250/227, 385/129, 385/131, 385/901
• US Class Current: 345/176
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

High-resolution, scalable multi-touch sensing display systems and processes based on frustrated total internal reflection employ an optical waveguide that receives light, such as infrared light, that undergoes total internal reflection and an imaging sensor that detects light that escapes the optical waveguide caused by frustration of the total internal reflection due to contact by a user. The optical waveguide may be fitted with a compliant surface overlay to greatly improve sensing performance, minimize the affect of contaminants on and damage to the contact surface, to generally extend system life and to provide other benefits. The systems and processes provide true multi-touch (multi-input) and high-spatial and temporal resolution capability due to the continuous imaging of the frustrated total internal reflection that escapes the entire optical waveguide. Among other features and benefits, the systems and processes are scalable to large installations and are well suited for use with rear-projection and other display devices.

Claims

1. A multi-touch sensing display, comprising:

a light source;

a display element having a viewer side and a rear side;

an optical waveguide, the optical waveguide having a contact surface, a non-contact surface, and one or more edge surfaces, the non-contact surface facing and covering at least a portion of the viewer side of the display element, at least one of the one or more edge surfaces of the optical waveguide being adapted to receive light emitted by the light source, the optical waveguide being adapted to cause some of the received light to undergo total internal reflection within the optical waveguide;

a compliant surface overlay having a touch surface and a display-side surface, the display-side surface facing the contact surface of the optical waveguide and being spaced apart from the contact surface by a gap, the optical waveguide, the compliant surface overlay and the gap therebetween being configured to promote total internal reflection of light in the optical waveguide, the compliant surface overlay being disposed sufficiently close to the optical waveguide to enable depression of the compliant surface overlay by an external force against the touch surface of the compliant surface overlay to cause the display-side surface of the compliant surface overlay to contact the contact surface of the optical waveguide, the optical waveguide and the compliant surface overlay being adapted to cause the total internal reflection within the optical waveguide to be frustrated upon contact of the contact surface of the optical waveguide by the display-side surface of the compliant surface overlay, the optical waveguide adapted to allow some of the light undergoing total internal reflection to escape through its non-contact surface when the total internal reflection is frustrated; and

at least an imaging sensor positioned opposing the non-contact surface of the optical waveguide and adapted to detect light escaping the optical waveguide toward the display element,

wherein the compliant surface overlay includes a component that rejects ambient light of the same wavelengths as the light source, a component that reduces friction at an interaction surface, and a component that reduces glare.

2. The multi-touch sensing display of claim 1, wherein the gap comprises a small gap disposed between a substantial portion of the compliant surface overlay and the optical waveguide so that the frustration of the total internal reflection is minimized during non-depression of the compliant surface overlay.

3. The multi-touch sensing display of claim 1, wherein the compliant surface overlay is adapted to be depressed by at least one finger of a user of the multi-touch sensing display, a stylus and another object applying force.

4. The multi-touch sensing display of claim 1, wherein the compliant surface overlay is adapted to contact the optical waveguide at a plurality of locations simultaneously in response to simultaneous depression of the compliant surface overlay at a plurality of corresponding locations, the optical waveguide adapted to cause the total internal reflection to be frustrated simultaneously within the optical waveguide at said plurality of locations.

5. The multi-touch sensing display of claim 4, wherein at least two of the plurality of corresponding locations at which the compliant surface overlay are simultaneously depressed are disposed apart from one another.

6. The multi-touch sensing display of claim 1, wherein the imaging sensor is adapted to discriminately detect light output from positions along a two-dimensional surface of the optical waveguide.

7. The multi-touch sensing display of claim 1, wherein the imaging sensor is adapted to discriminately detect light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

8. The multi-touch sensing display of claim 1, wherein the light source is adapted to emit light of a predetermined wavelength, and the imaging sensor is adapted to detect light only substantially at the predetermined wavelength.

9. The multi-touch sensing display of claim 1, wherein the light source is adapted to emit light of first and second predetermined wavelengths, and the imaging sensor is adapted to detect light only substantially at the first predetermined wavelength;

the multi-touch sensing display further comprising a second imaging sensor adapted to detect light only substantially at the second predetermined wavelength escaping the optical waveguide.

10. The multi-touch sensing display of claim 1, wherein the light source is adapted to emit pulses of light at a rate substantially synchronized to a shutter of the imaging sensor.

11. The multi-touch sensing display of claim 1, wherein the compliant surface overlay includes a non-planar contact surface, the compliant surface overlay being adapted to contact the optical waveguide upon depression by the external force of the non-planar contact surface.

12. The multi-touch sensing display of claim 1, further comprising a video projector adapted to project a two-dimensional video image.

13. The multi-touch sensing display of claim 12, wherein the display element comprises a diffuser onto which the video image is projected.

14. The multi-touch sensing display of claim 13, wherein the diffuser is an electro-switchable diffuser screen synchronized to a rate of projection of the video projector, the diffuser being adapted to be non-diffusive in accordance with shutter periods of the imaging sensor.

15. The multi-touch sensing display of claim 14, further comprising at least one additional imaging sensor adapted to image through the diffuser so as to image an object applying the external force to cause the compliant surface overlay to contact the optical waveguide.

16. The multi-touch sensing display of claim 13, wherein the diffuser is a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

17. The multi-touch sensing display of claim 1, wherein the compliant surface overlay includes a component that provides a comfortable cushion for a user to depress.

18. The multi-touch sensing display of claim 1, further comprising a video projector adapted to project a two-dimensional video image through the optical waveguide.

19. The multi-touch sensing display of claim 1, wherein the display element comprises an LCD display panel disposed between the imaging sensor and the optical waveguide for displaying a video image, the LCD display panel being transparent to light escaping the optical waveguide so as to not prevent the escaping light from being detected by the imaging sensor, and the compliant surface overlay being adapted not to diffuse.

20. A method of multi-touch sensing, comprising the steps of:

displaying one or more images using a display element having a viewer side and a rear side;

receiving, within an optical waveguide through at least one edge surface of the optical waveguide, light emitted by a light source, the optical waveguide having a contact surface, a non-contact surface, and one or more edge surfaces, the non-contact surface facing and covering at least a portion of the viewer side of the display element;

internally reflecting the received light within the optical waveguide;

depressing a compliant surface overlay disposed adjacent to the optical waveguide based on an external force against a touch surface of the compliant surface overlay, the compliant surface overlay having the touch surface and a display-side surface, the display-side surface facing the contact surface of the optical waveguide and being spaced apart from the contact surface by a gap, the optical waveguide, the compliant surface overlay and the gap therebetween being configured to promote total internal reflection of light in the optical waveguide, wherein the compliant surface overlay includes a component that rejects ambient light of the same wavelengths as the light source, a component that reduces friction at an interaction surface, and a component that reduces glare;

contacting the contact surface of the optical waveguide by the display-side surface of the compliant surface overlay upon depressing the compliant surface overlay;

frustrating the internally reflected light within the optical waveguide upon contacting the contact surface of the optical waveguide by the display-side surface of the compliant surface overlay to cause some of the internally reflected light to escape the optical waveguide; and

imaging, using at least an imaging sensor positioned opposing the non-contact surface of the optical waveguide, light escaping the optical waveguide toward the display element.

21. The method of claim 20, wherein the gap comprises a small gap between a substantial portion of the compliant surface overlay and the optical waveguide so that frustrating the internally reflected light is minimized when the compliant surface overlay is not depressed.

22. The method of claim 20, wherein depressing the compliant surface overlay comprises depressing the compliant surface overlay by a finger of a user.

23. The method of claim 20, wherein depressing the compliant surface overlay comprises depressing simultaneously the compliant surface overlay at a plurality of locations, contacting the optical waveguide comprises contacting simultaneously the optical waveguide by the compliant surface overlay at a plurality of corresponding locations, and frustrating the internally reflected light comprises frustrating simultaneously the internally reflected light within the optical waveguide at said plurality of corresponding locations.

24. The method of claim 23, wherein at least two of said plurality of corresponding locations are disposed apart from one another.

25. The method of claim 20, wherein imaging the escaped light comprises imaging discriminately light output from positions along a two-dimensional surface of the optical waveguide.

26. The method of claim 20, wherein imaging the escaped light comprises imaging discriminately light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

27. The method of claim 20, wherein receiving light within the optical waveguide comprises receiving light of a predetermined wavelength, and imaging the escaped light comprises imaging light only substantially at the predetermined wavelength.

28. The method of claim 20, wherein receiving light within the optical waveguide comprises receiving light of first and second predetermined wavelengths, and imaging the escaped light comprises imaging light only substantially at the first and second predetermined wavelengths.

29. The method of claim 28, wherein imaging the escaped light comprises imaging light only substantially at the first predetermined wavelength by a first imaging sensor and imaging light only substantially at the second predetermined wavelength by a second imaging sensor.

30. The method of claim 20, wherein receiving light comprises receiving pulses of light within the optical waveguide;

and imaging the escaped light comprises imaging the escaped light by the imaging sensor, which includes a shutter substantially synchronized to the rate of the pulses of light received within the optical waveguide.

31. The method of claim 20, comprising providing the compliant surface overlay with a non-planar contact surface, and depressing the compliant surface overlay comprises depressing the non-planar contact surface of the compliant surface overlay.

32. The method of claim 20, comprising projecting a two-dimensional video image.

33. The method of claim 32, comprising diffusing the projected video image by the display element.

34. The method of claim 32, comprising diffusing the projected video image by an electro-switchable diffuser screen synchronized to a rate of projection of the video image;

wherein imaging the escaped light is carried out by the imaging sensor, the diffuser being non-diffusive in accordance with shutter periods of the imaging sensor.

35. The method of claim 34, comprising imaging through the diffuser by a second imaging sensor an object causing the frustration of the internally reflected light.

36. The method of claim 32, comprising diffusing the projected video image by a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

37. The method of claim 20, wherein the compliant surface overlay includes a component that provides a comfortable cushion for a user to depress.

38. The method of claim 20, comprising projecting a two-dimensional video image through the optical waveguide.

39. A multi-touch sensing display, comprising:

a light source;

an optical waveguide configured to receive light emitted by the light source and to cause at least some of the received light to undergo total internal reflection within the optical waveguide, the optical waveguide having a contact surface, a non-contact surface, and one or more edge surfaces, at least one of the one or more edge surfaces of the optical waveguide being configured to receive light emitted by the light source;

a compliant surface overlay having a touch surface and a display-side surface, the display-side surface facing the contact surface of the optical waveguide so as to enable the display-side surface of the compliant surface overlay to contact the contact surface of the optical waveguide when the compliant surface overlay is physically depressed, the compliant surface overlay being spaced apart from the contact surface of the optical waveguide by a gap and being configured to;

cause frustration of the total internal reflection of the received light within the optical waveguide at a contact point between the display-side surface of the compliant surface overlay and the contact surface of the optical waveguide when the display-side surface of the compliant surface overlay is physically depressed to contact the contact surface of the optical waveguide such that some of the received light undergoing total internal reflection within the optical waveguide escapes from the optical waveguide at the contact point; and

operate as a diffuser for rear projection;

a projector positioned opposing the non-contact surface of the optical waveguide and configured to project images that are diffused by the compliant surface overlay and made visible to a user providing touch inputs to the touch surface of the compliant surface overlay, the non-contact surface of the optical waveguide facing the projector; and

an imaging sensor positioned opposing the non-contact surface of the optical waveguide and configured to detect light escaping the optical waveguide toward the projector to enable detection of touch input provided to the compliant surface overlay and discrimination between a relatively small point of contact with the compliant surface overlay and a relatively large point of contact with the compliant surface overlay, the imaging sensor being configured to detect pressure applied to the compliant surface overlay based on whether the touch input causes the relatively small point of contact with the compliant surface overlay or the relatively large point of contact with the compliant surface overlay,

wherein the compliant surface overlay is a composite of multiple layers affixed to one another using index-of-refraction matching optical adhesives, and

wherein the multiple layers of the compliant surface overlay comprise a layer that functions as an optical diffuser for rear projection, a layer that that provides wavelength selective shielding to mitigate interference from external ambient light, and an anti-glare layer to enhance visibility of the display.

40. A multi-touch sensing display, comprising:

an infrared light source;

an optical waveguide configured to receive infrared light emitted by the infrared light source and to cause at least some of the received infrared light to undergo total internal reflection within the optical waveguide, the optical waveguide having a contact surface, a non-contact surface, and one or more edge surfaces, at least one of the one or more edge surfaces of the optical waveguide being configured to receive infrared light emitted by the infrared light source;

a compliant surface overlay having a touch surface and a display-side surface, the display-side surface facing the contact surface of the optical waveguide so as to enable the display-side surface of the compliant surface overlay to contact the contact surface of the optical waveguide when the compliant surface overlay is physically depressed, the compliant surface overlay being spaced apart from the contact surface of the optical waveguide by a gap and being configured to;

cause frustration of the total internal reflection of the received infrared light within the optical waveguide at a contact point between the display-side surface of the compliant surface overlay and the contact surface of the optical waveguide when the display-side surface of the compliant surface overlay is physically depressed to contact the contact surface of the optical waveguide such that some of the received infrared light undergoing total internal reflection within the optical waveguide escapes from the optical waveguide at the contact point;

at least an imaging sensor positioned opposing the non-contact surface of the optical waveguide and configured to detect infrared light escaping the optical waveguide to enable detection of touch inputs being provided to the compliant surface overlay;

a liquid crystal display panel positioned between the optical waveguide and the imaging sensor, the liquid crystal display panel being configured to display images and being transparent to infrared light so that infrared light escaping the optical waveguide passes through the liquid crystal display panel and is imaged by the imaging sensor, where the compliant surface overlay has relatively low optical diffuser properties such that images displayed by the liquid crystal display panel are visible to a user providing touch inputs to the compliant surface overlay and the non-contact surface of the optical waveguide faces and covers at least a portion of the liquid crystal display panel,

wherein the compliant surface overlay comprises a layer that provides wavelength selective shielding to mitigate interference from external ambient light and an anti-glare layer to enhance display visibility.

41. The multi-touch sensing display of claim 40, further comprising:

multiple imaging sensors positioned opposing the non-contact surface of the optical waveguide and configured to detect infrared light escaping the optical waveguide; and

at least one computing machine configured to process images captured by the multiple imaging sensors to determine a pose of a user'"'"'s finger touching the touch surface of the compliant surface overlay.

42. The multi-touch sensing display of claim 40, further comprising:

multiple imaging sensors positioned opposing the non-contact surface of the optical waveguide and configured to detect infrared light escaping the optical waveguide; and

at least one computing machine configured to process images captured by the multiple imaging sensors to determine an identity of a user'"'"'s finger associated with each of multiple touch points on the touch surface of the compliant surface overlay.