MULTI-TOUCH SENSING THROUGH FRUSTRATED TOTAL INTERNAL REFLECTION

• US 20080179507A2
• Filed: 08/03/2007
• Published: 07/31/2008
• Est. Priority Date: 08/03/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: HAN, Jefferson
• Granted Patent: US 8,441,467 B2
• Time in Patent Office: Days
• US Class Current: 250/224
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

High-resolution, scalable multi-touch sensing display systems and processes based on frustrated total internal reflection employ an optical waveguide that receives light, such as infrared light, that undergoes total internal reflection and an imaging sensor that detects light that escapes the optical waveguide caused by frustration of the total internal reflection due to contact by a user. The optical waveguide may be fitted with a compliant surface overlay to greatly improve sensing performance, minimize the affect of contaminants on and damage to the contact surface, to generally extend system life and to provide other benefits. The systems and processes provide true multi-touch (multi-input) and high-spatial and temporal resolution capability due to the continuous imaging of the frustrated total internal reflection that escapes the entire optical waveguide. Among other features and benefits, the systems and processes are scalable to large installations and are well suited for use with rear-projection and other display devices.

Claims

1. A multi-touch sensing display, comprising:

an optical waveguide adapted to receive light and adapted to cause at least some of the received light to undergo total internal reflection within the optical waveguide, the optical waveguide adapted to allow the total internal reflection to be frustrated upon occurrence of a physical phenomena, the optical waveguide adapted to allow some of the light undergoing total internal reflection to escape when the total internal reflection is frustrated; and

an imaging sensor adapted to detect light escaping the optical waveguide.

2. The multi-touch sensing display of claim 1, further comprising a plurality of infrared light emitting diodes disposed directly against a polished edge of the optical waveguide for emitting the light received by the optical waveguide.

3. The multi-touch sensing display of claim 1, wherein the optical waveguide is a sheet of acrylic with edges treated to admit light.

4. The multi-touch sensing display of claim 1, wherein the physical phenomena comprises contacting the optical waveguide with an object so that the total internal reflection is frustrated upon contacting the optical waveguide with the object.

5. The multi-touch sensing display of claim 4, wherein the object is a finger of a user of the multi-touch sensing display.

6. The multi-touch sensing display of claim 1, wherein the optical waveguide is adapted to allow the total internal reflection to be frustrated simultaneously at a plurality of positions within the optical waveguide upon occurrence of a plurality of physical phenomenon, each of the positions disposed apart from one another.

7. The multi-touch sensing display of claim 6, wherein the plurality of physical phenomenon include at least contacting a first location of a surface of the optical waveguide with a first object and contacting a second location of the surface of the optical waveguide with a second object, the first and second locations being disposed apart from one another, the optical waveguide being adapted to cause total internal reflection to be frustrated within the optical waveguide at first and second positions corresponding to the first and second locations upon contacting the optical waveguide with the first and second objects.

8. The multi-touch sensing display of claim 7, wherein the first and second objects are first and second fingers of a user of the multi-touch sensing display.

9. The multi-touch sensing display of claim 1, wherein the imaging sensor is adapted to discriminately detect light output from positions along a two-dimensional surface of the optical waveguide.

10. The multi-touch sensing display of claim 1, wherein the imaging sensor is adapted to discriminately detect light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

11. The multi-touch sensing display of claim 1, further comprising a light source adapted to emit light of predetermined wavelengths;

wherein the optical waveguide is adapted to receive the light emitted by the light source, and the imaging sensor is adapted to detect light only substantially at the predetermined wavelengths.

12. The multi-touch sensing display of claim 1, further comprising a light source adapted to emit light of first and second predetermined wavelengths;

wherein the optical waveguide is adapted to receive the light emitted by the light source, and the imaging sensor is adapted to detect light only substantially at the first predetermined wavelength;

the multi-touch sensing display further comprising a second imaging sensor adapted to detect light only substantially at the second predetermined wavelength escaping the optical waveguide.

13. The multi-touch sensing display of claim 1, further comprising a light source adapted to emit pulses of light at a rate substantially synchronized to a shutter of the imaging sensor;

wherein the optical waveguide is adapted to receive the light emitted by the light source.

14. The multi-touch sensing display of claim 1, wherein the optical waveguide has a non-planar surface, and the physical phenomena comprises contacting the non-planar surface of the optical waveguide so that the total internal reflection is frustrated upon contacting the non-planar surface of the optical waveguide.

15. The multi-touch sensing display of claim 1, further comprising a video projector adapted to project a two-dimensional video image onto the optical waveguide.

16. The multi-touch sensing display of claim 15, further comprising a diffuser onto which the video image is projected.

17. The multi-touch sensing display of claim 16, wherein the optical waveguide includes a non-contact surface and a contact surface;

the diffuser being disposed on the non-contact surface and the physical phenomena comprising contacting the contact surface of the optical waveguide so that the total internal reflection is frustrated upon contacting the contact surface of the optical waveguide.

18. The multi-touch sensing display of claim 17, wherein the video projector is disposed on a non-contact side of the optical waveguide corresponding to a side on which the non-contact surface is disposed, the video projector adapted to project the video image toward the optical waveguide for viewing by a user disposed on a contact side of the optical waveguide corresponding to a side on which the contact surface is disposed.

19. The multi-touch sensing display of claim 1, further comprising a video projector adapted to project a two-dimensional video image;

and a diffuser disposed adjacent to a surface of the optical waveguide, the video image being projected onto the diffuser.

20. The multi-touch sensing display of claim 19, wherein a small gap is disposed between a substantial portion of the diffuser and the optical waveguide so that the frustration of the total internal reflection by the diffuser is minimized.

21. The multi-touch sensing display of claim 19, wherein the diffuser is an electro-switchable diffuser screen synchronized to projection of the video projector and to the shutter of the imaging sensor.

22. The multi-touch sensing display of claim 19, wherein the diffuser is an electro-switchable diffuser screen synchronized to projection of the video projector and to the shutter of the imaging sensor.

23. The multi-touch sensing display of claim 1, wherein the imaging sensor includes a plurality of imaging sensors adapted to detect light escaping from respectively different portions of the optical waveguide.

24. The multi-touch sensing display of claim 23, further comprising a plurality of video projectors, each of the video projectors being adapted to project a respective two-dimensional video image onto a respective one of said different portions of the optical waveguide.

25. The multi-touch sensing display of claim 1, further comprising an LCD display panel disposed between the imaging sensor and the optical waveguide for displaying a video image, the LCD display panel being transparent to light escaping the optical waveguide so as to not prevent the escaping light from being detected by the imaging sensor.

85. The multi-touch sensing display of claim 16, wherein the diffuser is an electro-switchable diffuser screen synchronized to a rate of projection of the video projector, the diffuser being adapted to be non-diffusive in accordance with shutter periods of the imaging sensor.

86. The multi-touch sensing display of claim 85, further comprising at least one additional imaging sensor adapted to image through the diffuser so as to image the physical phenomena causing the frustration of the total internal reflection.

87. The multi-touch sensing display of claim 85, wherein the physical phenomena includes contacting the optical waveguide with an object, the multi-touch sensing display further comprising at least one additional imaging sensor adapted to image through the diffuser so as to image the object contacting the optical waveguide.

88. The multi-touch sensing display of claim 16, wherein the diffuser is a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

26. A multi-touch sensing display, comprising:

an optical waveguide adapted to receive light and adapted to cause some of the received light to undergo total internal reflection within the optical waveguide;

a compliant surface overlay disposed sufficiently close to the optical waveguide to enable depression of the compliant surface overlay by an external force to cause the compliant surface overlay to contact the optical waveguide, the optical waveguide and the compliant surface overlay adapted to cause the total internal reflection within the optical waveguide to be frustrated upon contact of the optical waveguide by the compliant surface overlay, the optical waveguide adapted to allow some of the light undergoing total internal reflection to escape when the total internal reflection is frustrated; and

an imaging sensor adapted to detect light escaping the optical waveguide.

27. The multi-touch sensing display of claim 26, wherein a small gap is disposed between a substantial portion of the compliant surface overlay and the optical waveguide so that the frustration of the total internal reflection is minimized during non-depression of the compliant surface overlay.

28. The multi-touch sensing display of claim 26, wherein the compliant surface overlay is adapted to be depressed by at least one finger of a user of the multi-touch sensing display, a stylus and another object applying force.

29. The multi-touch sensing display of claim 26, wherein the compliant surface overlay is adapted to contact the optical waveguide at a plurality of locations simultaneously in response to simultaneous depression of the compliant surface overlay at a plurality of corresponding locations, the optical waveguide adapted to cause the total internal reflection to be frustrated simultaneously within the optical waveguide at said plurality of locations.

30. The multi-touch sensing display of claim 29, wherein at least two of the plurality of corresponding locations at which the compliant surface overlay are simultaneously depressed are disposed apart from one another.

31. The multi-touch sensing display of claim 26, wherein the imaging sensor is adapted to discriminately detect light output from positions along a two-dimensional surface of the optical waveguide.

32. The multi-touch sensing display of claim 26, wherein the imaging sensor is adapted to discriminately detect light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

33. The multi-touch sensing display of claim 26, further comprising a light source adapted to emit light of a predetermined wavelength;

wherein the optical waveguide is adapted to receive the light emitted by the light source, and the imaging sensor is adapted to detect light only substantially at the predetermined wavelength.

34. The multi-touch sensing display of claim 26, further comprising a light source adapted to emit light of first and second predetermined wavelengths;

wherein the optical waveguide is adapted to receive the light emitted by the light source, and the imaging sensor is adapted to detect light only substantially at the first predetermined wavelength;

the multi-touch sensing display further comprising a second imaging sensor adapted to detect light only substantially at the second predetermined wavelength escaping the optical waveguide.

35. The multi-touch sensing display of claim 26, further comprising a light source adapted to emit pulses of light at a rate substantially synchronized to the shutter of the imaging sensor;

wherein the optical waveguide is adapted to receive the light emitted by the light source.

36. The multi-touch sensing display of claim 26, wherein the compliant surface overlay includes a non-planar contact surface, the compliant surface overlay being adapted to contact the optical waveguide upon depression by the external force of the non-planar contact surface.

37. The multi-touch sensing display of claim 26, further comprising a video projector adapted to project a two-dimensional video image onto the optical waveguide.

38. The multi-touch sensing display of claim 37, comprising a diffuser onto which the video image is projected.

39. The multi-touch sensing display of claim 37, wherein the compliant surface overlay is adapted to diffuse the video image projected by the video projector.

40. The multi-touch sensing display of claim 26, wherein the compliant surface overlay includes a component that rejects ambient light of the same wavelengths as the light source, a component that reduces friction at the interaction surface, a component that reduces glare, and a component that provides a comfortable cushion for the user to depress.

41. The multi-touch sensing display of claim 26, further comprising a video projector adapted to project a two-dimensional video image through the optical waveguide onto the compliant surface overlay, the compliant surface overlay being adapted to diffuse the video image projected by the video projector.

42. The multi-touch sensing display of claim 26, further comprising an LCD display panel disposed between the imaging sensor and the optical waveguide for displaying a video image, the LCD display panel being transparent to light escaping the optical waveguide so as to not prevent the escaping light from being detected by the imaging sensor, and the compliant surface adapted not to diffuse.

89. The multi-touch sensing display of claim 38, wherein the diffuser is an electro-switchable diffuser screen synchronized to a rate of projection of the video projector, the diffuser being adapted to be non-diffusive in accordance with shutter periods of the imaging sensor.

90. The multi-touch sensing display of claim 89, further comprising at least one additional imaging sensor adapted to image through the diffuser so as to image an object applying the external force to cause the compliant surface overlay to contact the optical waveguide.

91. The multi-touch sensing display of claim 38, wherein the diffuser is a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

92. The multi-touch sensing display of claim 26, wherein the compliant surface overlay includes a plurality of layers, at least one of the layers being a wavelength selective shield to mitigate interference by external ambient light of the detection of light escaping the optical waveguide by the imaging sensor.

93. The multi-touch sensing display of claim 92, wherein at least one of the layers of the compliant surface overlay is adapted to contribute to the frustration of the total internal reflection within the optical waveguide upon contact of the optical waveguide by the compliant surface overlay.

43. A method of multi-touch sensing, comprising the steps of:

receiving light within an optical waveguide;

internally reflecting the received light within the optical waveguide;

frustrating the internally reflected light within the optical waveguide to cause some of the reflected light to escape the optical waveguide; and

imaging the escaped light.

44. The method of claim 43, comprising emitting the light by a light source having a plurality of infrared light emitting diodes, the emitted light being received within the optical waveguide.

45. The method of claim 43, comprising providing a sheet of acrylic with edges treated to admit light as the optical waveguide.

46. The method of claim 43, wherein frustrating the internally reflected light comprises contacting the optical waveguide with an object.

47. The method of claim 43, wherein frustrating the internally reflected light comprises contacting the optical waveguide with a finger of a user.

48. The method of claim 43, wherein frustrating the internally reflected light includes frustrating the internally reflected light simultaneously at a plurality of positions within the optical waveguide, each of the positions disposed apart from one another.

49. The method of claim 43, wherein frustrating the internally reflected light comprises contacting simultaneously first and second locations of a surface of the optical waveguide to cause reflected light to escape simultaneously from first and second positions of the optical waveguide, the first and second positions being disposed apart from one another and corresponding respectively to the first and second locations.

50. The method of claim 49, wherein frustrating the internally reflected light comprises contacting simultaneously the first and second locations of the surface of the optical waveguide by first and second fingers of a user.

51. The method of claim 43, wherein imaging the escaped light comprises imaging discriminately light output from positions along a two-dimensional surface of the optical waveguide.

52. The method of claim 43, wherein imaging the escaped light comprises imaging discriminately light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

53. The method of claim 43, wherein receiving light within the optical waveguide comprises receiving light of a predetermined wavelength, and imaging the escaped light comprises imaging light only substantially at the predetermined wavelength.

54. The method of claim 43, wherein receiving light within the optical waveguide comprises receiving light of first and second predetermined wavelengths, and imaging the escaped light comprises imaging light only substantially at the first and second predetermined wavelengths.

55. The method of claim 54, wherein imaging the escaped light comprises imaging light only substantially at the first predetermined wavelength by a first imaging sensor and imaging light only substantially at the second predetermined wavelength by a second imaging sensor.

56. The method of claim 43, wherein receiving light comprises receiving pulses of light within the optical waveguide;

and imaging the escaped light comprises imaging the escaped light by an imaging sensor having a shutter substantially synchronized to the rate of the pulses of light received within the optical waveguide.

57. The method of claim 43, comprising providing an optical waveguide having a non-planar surface.

58. The method of claim 43, comprising projecting a two-dimensional video image onto the optical waveguide.

59. The method of claim 58, comprising diffusing the projected video image by the optical waveguide.

60. The method of claim 43, comprising projecting a two-dimensional video image, diffusing the projected video image on a non-contact surface of the optical waveguide;

and contacting a contact surface of the optical waveguide to frustrate the internally reflected light.

61. The method of claim 60, comprising projecting the two-dimensional video image from a non-contact side of the optical waveguide corresponding to a side on which the non-contact surface is disposed.

62. The method of claim 43, comprising projecting a two-dimensional video image and diffusing the projected video image adjacent to the optical waveguide.

63. The method of claim 62, comprising diffusing the projected video image by a diffuser disposed adjacent to the optical waveguide, and providing a small gap between a substantial portion of the diffuser and the optical waveguide so that frustration of the internally reflected light by the diffuser is minimized.

64. The method of claim 63, comprising providing as the diffuser an electro-switchable diffuser screen synchronized to a rate of projection of the video image.

65. The method of claim 43, wherein imaging the escaped light comprises imaging by a plurality of imaging sensors light escaping from respectively different portions of the optical waveguide.

66. The method of claim 65, comprising projecting a plurality of two-dimensional video images onto said respectively different portions of the optical waveguide.

67. The method of claim 43, comprising displaying through the optical waveguide a video image by an LCD display panel, and passing through the LCD display panel light escaping the optical waveguide, and wherein imaging the escaped light comprises imaging the escaped light passing through the LCD display panel.

94. The method of claim 58, comprising diffusing the projected video image by an electro-switchable diffuser screen synchronized to a rate of projection of the video image;

wherein imaging the escaped light is carried out by an imaging sensor, the diffuser being non-diffusive in accordance with shutter periods of the imaging sensor.

95. The method of claim 94, comprising imaging through the diffuser by a second imaging sensor an object causing the frustration of the internally reflected light.

96. The method of claim 58, comprising diffusing the projected video image by a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

68. A method of multi-touch sensing, comprising the steps of:

receiving light within an optical waveguide;

internally reflecting the received light within the optical waveguide;

depressing a compliant surface overlay disposed adjacent to the optical waveguide;

contacting the optical waveguide by the compliant surface overlay upon depressing the compliant surface overlay;

frustrating the internally reflected light within the optical waveguide upon contacting optical waveguide by the compliant surface overlay to cause some of the internally reflected light to escape the optical waveguide; and

imaging the escaped light.

69. The method of claim 68, comprising disposing a small gap between a substantial portion of the compliant surface overlay and the optical waveguide so that frustrating the internally reflected light is minimized when the compliant surface overlay is not depressed.

70. The method of claim 68, wherein depressing the compliant surface overlay comprises depressing the compliant surface overlay by a finger of a user.

71. The method of claim 68, wherein depressing the compliant surface overlay comprises depressing simultaneously the compliant surface overlay at a plurality of locations, contacting the optical waveguide comprises contacting simultaneously the optical waveguide by the compliant surface at a plurality of corresponding locations, and frustrating the internally reflected light comprises frustrating simultaneously the internally reflected light within the optical waveguide at said plurality of corresponding locations.

72. The method of claim 71, wherein at least two of said plurality of corresponding locations are disposed apart from one another.

73. The method of claim 68, wherein imaging the escaped light comprises imaging discriminately light output from positions along a two-dimensional surface of the optical waveguide.

74. The method of claim 68, wherein imaging the escaped light comprises imaging discriminately light simultaneously escaping from a plurality of positions along a two-dimensional surface of the optical waveguide.

75. The method of claim 68, wherein receiving light within the optical waveguide comprises receiving light of a predetermined wavelength, and imaging the escaped light comprises imaging light only substantially at the predetermined wavelength.

76. The method of claim 68, wherein receiving light within the optical waveguide comprises receiving light of first and second predetermined wavelengths, and imaging the escaped light comprises imaging light only substantially at the first and second predetermined wavelengths.

77. The method of claim 76, wherein imaging the escaped light comprises imaging light only substantially at the first predetermined wavelength by a first imaging sensor and imaging light only substantially at the second predetermined wavelength by a second imaging sensor.

78. The method of claim 68, wherein receiving light comprises receiving pulses of light within the optical waveguide;

and imaging the escaped light comprises imaging the escaped light by an imaging sensor having a shutter substantially synchronized to the rate of the pulses of light received within the optical waveguide.

79. The method of claim 68, comprising providing the compliant surface overlay with a non-planar contact surface, and depressing the compliant surface overlay comprises depressing the non-planar contact surface of the compliant surface overlay.

80. The method of claim 68, comprising projecting a two-dimensional video image onto the optical waveguide.

81. The method of claim 80, comprising diffusing the projected video image by the optical waveguide.

82. The method of claim 80, comprising diffusing the projected video image by the compliant surface overlay.

83. The method of claim 68, wherein the compliant surface overlay includes a component that rejects ambient light of the same wavelengths as the light source, a component that reduces friction at the interaction surface, a component that reduces glare, and a component that provides a comfortable cushion for the user to depress.

84. The method of claim 68, comprising projecting a two-dimensional video image through the optical waveguide onto the compliant surface overlay;

and diffusing the projected video image by the compliant surface overlay.

97. The method of claim 80, comprising diffusing the projected video image by an electro-switchable diffuser screen synchronized to a rate of projection of the video image;

wherein imaging the escaped light is carried out by an imaging sensor, the diffuser being non-diffusive in accordance with shutter periods of the imaging sensor.

98. The method of claim 97, comprising imaging through the diffuser by a second imaging sensor an object causing the frustration of the internally reflected light.

99. The method of claim 80, comprising diffusing the projected video image by a directional diffuser adapted to diffuse for predetermined angles of incidence, and adapted to not diffuse for other angles of incidence.

100. The method of claim 68, comprising providing the compliant surface overlay with a plurality of layers, at least one of the layers being a wavelength selective shield to mitigate interference by external ambient light of the imaging of light escaping the optical waveguide.

101. The multi-touch sensing display of claim 100, comprising providing the compliant surface overlay with at least one layer adapted to contribute to the frustration of the internally reflected light within the optical waveguide upon contact of the optical waveguide by the compliant surface overlay.