Multi-touch sensing through frustrated total internal reflection

• US 8,144,271 B2
• Filed: 08/04/2008
• Issued: 03/27/2012
• Est. Priority Date: 08/03/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: HAN, Jefferson Y.
• Primary Examiner(s): Healy, Brian
• Application Number:
  US12/185,782
• Publication Number:
  US 20080284925A1
• Time in Patent Office: 1,331 Days
• Field of Search: 349/12, 349/23, 349/60, 349/64, 349/112, 349/138, 349/158, 385/12, 385/13, 385/901, 250/221, 313/504, 345/175
• US Class Current: 349/12
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: Y10S 385/901 Illuminating or display apparatus

Abstract

High-resolution, scalable multi-touch sensing display systems and processes based on frustrated total internal reflection employ an optical waveguide that receives light, such as infrared light, that undergoes total internal reflection and an imaging sensor that detects light that escapes the optical waveguide caused by frustration of the total internal reflection due to contact by a user. The optical waveguide when fitted with a compliant surface overlay provides superior sensing performance, as well as other benefits and features. The systems and processes described provide true multi-touch (multi-input) and high-spatial and temporal resolution capability due to the continuous imaging of the frustrated total internal reflection that escapes the entire optical waveguide. Among other features and benefits, the systems and processes are scalable to large installations.

Claims

1. A multi-touch sensing system, comprising:

an optical waveguide having an index of refraction sufficient to cause light of at least one wavelength received in a predetermined direction to undergo total internal reflection within the optical waveguide, the optical waveguide having first and second sides;

a compliant layer having a first surface disposed adjacent to and spaced from the first side of the optical waveguide, the compliant layer being sufficiently deformable upon depression of the compliant layer to allow portions of the first surface of the compliant layer to contact the optical waveguide, and the first surface of the compliant layer has an index of refraction sufficiently similar to the index of refraction of the optical waveguide to cause, upon the first surface of the compliant layer contacting the optical waveguide, some of the light undergoing total internal reflection to scatter and to escape the optical waveguide; and

an image sensor adapted to detect at least some of the light that escapes from the optical waveguide.

2. The multi-touch sensing system of claim 1, wherein the compliant layer has an associated light filter for filtering light of said at least one wavelength.

3. The multi-touch sensing system of claim 1, wherein the compliant layer is deformable by depressing a second surface of the compliant layer by a finger of a user of the multi-touch sensing system.

4. The multi-touch sensing system of claim 1, wherein the image sensor is optically disposed to receive the light that escapes from the second side of the optical waveguide.

5. The multi-touch sensing system of claim 4, further comprising a band pass filter optically disposed between the second side of the optical waveguide and the image sensor for allowing substantially only the light of said at least one wavelength to pass from the optical waveguide to the image sensor.

6. The multi-touch sensing system of claim 1, wherein the image sensor is a flat image sensor.

7. The multi-touch sensing system of claim 1, wherein the image sensor is a wedge-optic camera.

8. The multi-touch sensing system of claim 1, further comprising a light source disposed directly against an edge of the optical waveguide for emitting light of said at least one wavelength and received by the optical waveguide in at least said predetermined direction.

9. The multi-touch sensing system of claim 1, wherein the optical waveguide is a compound component comprised of a non-compliant optical waveguide layer and a compliant optical waveguide layer disposed on the non-compliant optical waveguide layer, the compliant optical waveguide layer being disposed on the first side of the optical waveguide so that the compliant layer is disposed adjacent to the compliant optical waveguide layer of the optical waveguide.

10. The multi-touch sensing system of claim 1, further comprising a rear projector for projecting a video image towards the second side of the optical waveguide, and wherein the compliant layer and the optical waveguide are substantially transparent to visible light so that the displayed video image is viewable through the compliant layer and the optical waveguide.

11. The multi-touch sensing system of claim 10, further comprising a diffuser disposed on the compliant layer for diffusing the video image projected towards the second side of the optical waveguide.

12. The multi-touch sensing system of claim 10, wherein the compliant layer is adapted to diffuse the video image projected towards the second side of the optical waveguide.

13. The multi-touch sensing system of claim 1, further comprising a film opaque to light of said at least one wavelength disposed on the compliant layer and a front projector for projecting a video image onto the film.

14. The multi-touch sensing system of claim 1, further comprising a liquid crystal display (LCD) panel disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, and wherein the compliant layer and the optical waveguide are substantially transparent to visible light so that the displayed video image is viewable through the compliant layer and the optical waveguide.

15. The multi-touch sensing system of claim 14, further comprising a plurality of emissive sources for providing backlight to the LCD panel and a backlight diffuser disposed between the LCD panel and the plurality of emissive sources.

16. The multi-touch sensing system of claim 14, further comprising a thin-profile wedge backlight unit for providing backlight to the LCD panel.

17. The multi-touch sensing system of claim 16, wherein the image sensor is a wedge-optic camera.

18. The multi-touch sensing system of claim 14, further comprising a plurality of light sources for providing backlight to the LCD panel and a backlight diffuser disposed between the LCD panel and the plurality of light sources, and wherein the image sensor is a wedge-optic camera.

19. The multi-touch sensing system of claim 14, wherein the LCD panel is disposed within an optical path of said at least some of the light escaping from the optical waveguide to the image sensor, and the LCD panel is transparent to light of said at least one wavelength so as to not prevent said light of said at least one wavelength escaping the optical waveguide from reaching the image sensor.

20. The multi-touch sensing system of claim 1, further comprising a liquid crystal display (LCD)/optical sensing panel disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, the image sensor corresponding to optical sensing elements of the LCD/optical sensing panel.

21. The multi-touch sensing system of claim 1, further comprising an active matrix OLED with integrated sensors disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, the image sensor corresponding to the integrated sensors of the active matrix OLED with integrated sensors.

22. The multi-touch sensing system of claim 1, further comprising a flexible OLED (FOLED) coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system, the FOLED being sufficiently flexible to deform the compliant layer, upon depression of the FOLED, so that the compliant layer contacts the optical waveguide.

23. The multi-touch sensing system of claim 1, further comprising a double-sided FOLED disposed on the compliant layer and having an emissive layer for displaying a video image in a direction towards a user of the multi-touch sensing system and a sensing layer for sensing some of the light that escapes from the optical waveguide in a direction towards the user, the double-sided FOLED being sufficiently flexible to deform the compliant layer, upon depression of the double-sided FOLED, so that the compliant layer contacts the optical waveguide, and the image sensor corresponds to the sensing layer of the FOLED.

24. The multi-touch sensing system of claim 1, further comprising a combination flexible LCD and a flexible backlight coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system, the combination of the flexible LCD and the flexible backlight being sufficiently flexible to deform the compliant layer, upon depression of the combination, so that the compliant layer contacts the optical waveguide.

25. The multi-touch sensing system of claim 1, further comprising a flexible LCD coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system, and a light source for directing visible light through the compliant layer toward the flexible LCD, the compliant layer being adapted to diffuse the visible light.

26. The multi-touch sensing system of claim 25, wherein the flexible LCD includes integrated sensors for sensing at least some of the light that escapes from the optical waveguide in a direction towards the user, and the image sensor corresponds to the sensors integrated in the flexible LCD.

27. The multi-touch sensing system of claim 1, wherein light of said at least one wavelength is infrared light and the image sensor is adapted to detect infrared light;

and the system further comprises a second image sensor for detecting visual information through the optical waveguide and the compliant layer, said second image sensor adapted to not detect light detected by the image sensor.

28. The multi-touch sensing system of claim 27, wherein the second image sensor is capable of sensing an area beyond the compliant layer.

29. A multi-touch sensing display, comprising:

a light source;

an optical waveguide configured to receive light emitted by the light source and to cause at least some of the received light to undergo total internal reflection within the optical waveguide;

a compliant layer that is spaced apart from the optical waveguide by an air gap and positioned such that the compliant layer contacts the optical waveguide when the compliant layer is physically depressed by a touch input, the compliant layer being configured to cause frustration of the total internal reflection of the received light within the optical waveguide at a contact point between the compliant layer and the optical waveguide when the compliant layer is physically depressed by a touch input to contact the optical waveguide such that some of the received light undergoing total internal reflection within the optical waveguide escapes from the optical waveguide at the contact point; and

an imaging sensor configured to detect light escaping the optical waveguide to enable detection of touch inputs that physically depress the compliant layer to contact the optical waveguide.

30. The multi-touch sensing display of claim 29, wherein the compliant layer is a composite of multiple layers affixed to one another using index-of-refraction matching optical adhesives.

31. The multi-touch sensing display of claim 30, wherein the multiple layers of the compliant layer comprise a layer that functions as an optical diffuser for rear projection, a layer that that provides wavelength selective shielding to mitigate interference from external ambient light, and an anti-glare layer to enhance visibility of the display.

32. The multi-touch sensing display of claim 29, wherein the imaging sensor is configured to discriminate between a relatively small point of contact between the compliant layer and the optical waveguide and a relatively large point of contact between the compliant layer and the optical waveguide.

33. The multi-touch sensing display of claim 29, wherein the imaging sensor is configured to detect pressure being applied to cause the compliant layer to contact the optical waveguide based on whether a point of contact between the compliant layer and the optical waveguide is relatively small or relatively.

34. The multi-touch sensing display of claim 29, wherein the compliant layer has an associated light filter for filtering light of at least one wavelength.

35. The multi-touch sensing display of claim 29, wherein the compliant layer is deformable by depressing a second surface of the compliant layer by a finger of a user of the multi-touch sensing display.

36. The multi-touch sensing display of claim 29, wherein the image sensor is optically disposed to receive the light that escapes from the optical waveguide.

37. The multi-touch sensing display of claim 36, further comprising a band pass filter optically disposed between the optical waveguide and the image sensor for allowing substantially only the light of the optical wavelength to pass from the optical waveguide to the image sensor.

38. The multi-touch sensing display of claim 29, wherein the image sensor is a flat image sensor.

39. The multi-touch sensing display of claim 29, wherein the image sensor is a wedge-optic camera.

40. The multi-touch sensing display of claim 29, wherein the light source is disposed directly against an edge of the optical waveguide for emitting light directly into the optical waveguide.

41. The multi-touch sensing display of claim 29, wherein the optical waveguide is a compound component comprised of a non-compliant optical waveguide layer and a compliant optical waveguide layer disposed on the non-compliant optical waveguide layer, the compliant optical waveguide layer being disposed on the first side of the optical waveguide so that the compliant layer is disposed adjacent to the compliant optical waveguide layer of the optical waveguide.

42. The multi-touch sensing display of claim 29, further comprising a rear projector for projecting an image towards the optical waveguide, and wherein the compliant layer and the optical waveguide are substantially transparent to visible light so that the image is viewable through the compliant layer and the optical waveguide.

43. The multi-touch sensing display of claim 42, further comprising a diffuser disposed on the compliant layer for diffusing the image projected towards the optical waveguide.

44. The multi-touch sensing display of claim 42, wherein the compliant layer is adapted to diffuse the image projected towards the optical waveguide.

45. The multi-touch sensing display of claim 29, further comprising a film opaque to light of at least one wavelength disposed on the compliant layer and a front projector for projecting an image onto the film.

46. A multi-touch sensing display, comprising:

an infrared light source;

an optical waveguide configured to receive infrared light emitted by the infrared light source and to cause at least some of the received infrared light to undergo total internal reflection within the optical waveguide;

a compliant layer that is spaced apart from the optical waveguide by an air gap and positioned such that the compliant layer contacts the optical waveguide when the compliant layer is physically depressed by a touch input on a touch surface of the compliant layer, the compliant layer being configured to cause frustration of the total internal reflection of the received infrared light within the optical waveguide at a contact point between the compliant layer and the optical waveguide when the compliant layer is physically depressed by a touch input on the touch surface to cause a non-touch surface of the compliant layer to contact the optical waveguide such that some of the received infrared light undergoing total internal reflection within the optical waveguide escapes from the optical waveguide at the contact point;

an imaging sensor positioned at a non-touch side of the compliant layer and configured to detect infrared light escaping the optical waveguide to enable detection of touch inputs that physically depress the compliant layer to contact the optical waveguide; and

a liquid crystal display panel positioned at the non-touch side of the compliant layer between the optical waveguide and the imaging sensor, the liquid crystal display panel being transparent to infrared light so that infrared light escaping the optical waveguide is imaged by the imaging sensor and being configured to display images, where the compliant layer has relatively low optical diffuser properties such that images displayed by the liquid crystal display panel are visible to a user providing touch inputs to the touch surface of the compliant layer.

47. The multi-touch sensing display of claim 46, further comprising:

multiple imaging sensors positioned at the non-contact side of the compliant layer and configured to detect infrared light escaping the optical waveguide; and

at least one computing machine configured to process images captured by the multiple imaging sensors to determine a pose of a user'"'"'s finger touching the compliant layer.

48. The multi-touch sensing display of claim 46, further comprising:

multiple imaging sensors positioned at the non-contact side of the compliant layer and configured to detect infrared light escaping the optical waveguide; and

at least one computing machine configured to process images captured by the multiple imaging sensors to determine an identify of a user'"'"'s finger associated with each of multiple touch points on the compliant layer.

49. The multi-touch sensing display of claim 46, wherein the compliant layer has an associated light filter for filtering light of at least one wavelength.

50. The multi-touch sensing display of claim 46, wherein the compliant layer is deformable by depressing a second surface of the compliant layer by a finger of a user of the multi-touch sensing display.

51. The multi-touch sensing display of claim 46, wherein the image sensor is optically disposed to receive the light that escapes from the optical waveguide.

52. The multi-touch sensing display of claim 51, further comprising a band pass filter optically disposed between the optical waveguide and the image sensor for allowing substantially only the light of the optical wavelength to pass from the optical waveguide to the image sensor.

53. The multi-touch sensing display of claim 46, wherein the image sensor is a flat image sensor.

54. The multi-touch sensing display of claim 46, wherein the image sensor is a wedge-optic camera.

55. The multi-touch sensing display of claim 46, wherein the light source is disposed directly against an edge of the optical waveguide for emitting light directly into the optical waveguide.

56. The multi-touch sensing display of claim 46, wherein the optical waveguide is a compound component comprised of a non-compliant optical waveguide layer and a compliant optical waveguide layer disposed on the non-compliant optical waveguide layer, the compliant optical waveguide layer being disposed on the first side of the optical waveguide so that the compliant layer is disposed adjacent to the compliant optical waveguide layer of the optical waveguide.

57. The multi-touch sensing display of claim 46, further comprising a plurality of emissive sources for providing backlight to the liquid crystal display panel and a backlight diffuser disposed between the liquid crystal display panel and the plurality of emissive sources.

58. The multi-touch sensing display of claim 46, further comprising a thin-profile wedge backlight unit for providing backlight to the liquid crystal display panel.

59. The multi-touch sensing display of claim 58, wherein the image sensor is a wedge-optic camera.

60. The multi-touch sensing display of claim 46, further comprising a plurality of light sources for providing backlight to the liquid crystal display panel and a backlight diffuser disposed between the liquid crystal display panel and the plurality of light sources, and wherein the image sensor is a wedge-optic camera.

61. The multi-touch sensing display of claim 46, wherein the liquid crystal display panel is disposed within an optical path of at least some of the light escaping from the optical waveguide to the image sensor, and the liquid crystal display panel is transparent to light of the at least one wavelength so as to not prevent light of the at least one wavelength escaping the optical waveguide from reaching the image sensor.