Human-machine interface

• US 20030137494A1
• Filed: 05/01/2000
• Published: 07/24/2003
• Est. Priority Date: 05/01/2000
• Status: Active Grant

• Current Assignee: David J. Tulbert
• Original Assignee: David J. Tulbert
• Inventors: Tulbert, David J.
• Granted Patent: US 7,859,519 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/0423 using sweeping light beams, e.g. using rotating or vibrating mirror
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A human-machine interface includes a panel formed of energy transmissive material having a contact surface on which one or more contacts may simultaneously be made. An energy source directs energy to the panel. The panel transmits energy received from the energy source to the contact surface. At least one parameter of the energy transmitted by the panel is altered at regions where contacts are made with the contact surface. A detector is coupled to the panel and detects the at least one parameter of the energy generally over the area of the contact surface and outputs values corresponding thereto. A processor is in communication with the detector. The processor processes the output values to determine the locations of the contact regions on the contact surface and at least one other attributed associated with each of the contacts.

Claims

1. A human-machine interface comprising:

a panel formed of energy transmissive material and having a contact surface on which one or more contacts may simultaneously be made;

an energy source directing energy to said panel, said panel transmitting energy received from said energy source to said contact surface, at least one parameter of energy transmitted by said panel being altered at regions where contacts are made with said contact surface;

a detector detecting the altered at least one parameter of said energy corresponding to contact regions on said contact surface and outputting values corresponding thereto; and

a processor in communication with said detector, said processor processing said output values to determine the locations of said contact regions on said contact surface and at least one other attribute associated with each of said contacts.

2. A human-machine interface as defined in claim 1 wherein said at least one other attribute includes the normal force applied to said panel at each of said contact regions.

3. A human-machine interface as defined in claim 2 wherein said at least one other attribute further includes shear forces applied to said panel at each of said contact regions.

4. A human-machine interface as defined in claim 1 wherein said locations are determined by calculating coordinates of said contact regions.

5. A human-machine interface as defined in claim 4 wherein said locations are defined by calculating the coordinates of the centers of said contact regions.

6. A human-machine interface as defined in claim 2 wherein said at least one other attribute further includes the instants contacts are made with said panel and the instants said contacts are ended.

7. A human-machine interface as defined in claim 6 wherein said at least one other attribute further includes the duration of each of said contacts.

8. A human-machine interface as defined in claim 2 wherein said at least one other attribute further includes oscillations in the normal forces applied to said panel during said contacts.

9. A human-machine interface as defined in claim 1 wherein contacts are made on the contact surface of said panel with a user'"'"'s fingers, contact between the panel and said fingers altering the direction of energy transmitted by said panel, said contact regions being generally elliptical in shape.

10. A human-machine interface as defined in claim 9 wherein a region above said panel is illuminated to illuminate fingers or portions thereof proximal to but spaced from said contact surface.

11. A human-machine interface as defined in claim 10 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact regions;

the instants contacts are made with said panel and the instants said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the durations of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

12. A human-machine interface as defined in claim 11 wherein said normal forces are calculated by determining the areas of said elliptical contact regions.

13. A human-machine interface as defined in claim 12 wherein said processor determines the positions of the tips of said user'"'"'s fingers contacting said panel and calculates the shear forces by determining the coordinates of the displaced finger tips with respect to the centers of said contact regions.

14. A human-machine interface as defined in claim 11 wherein said detector acquires images of said panel at selected intervals, each of said images being constituted by a plurality of pixels, each of said pixels attaining a pixel value proportional to its exposure to the at least one parameter of said energy during said interval, said processor generating histograms of said pixel values for pixels corresponding to contact regions in said images.

15. A human-machine interface as defined in claim 1 wherein energy directed to said panel is transmitted within said panel as a result of total internal reflection.

16. A human-machine interface as defined in claim 15 wherein said energy source is a light source and wherein said at least one parameter is the intensity of light, light transmitted within said panel escaping from said panel at regions where contacts are made with said contact surface thereby to alter said energy.

17. A human-machine interface as defined in claim 16 wherein said detector acquires images of said panel at selected intervals, each image being constituted by a plurality of pixels that attain pixel values proportional to their exposure to light intensity, said pixel values being output by said detector.

18. A human-machine interface as defined in claim 17 wherein said light source radiates visible light and wherein said detector is positioned relative to said panel at a side thereof opposite said contact surface, said light escaping from said panel at said contact regions and appearing as bright regions.

19. A human-machine interface as defined in claim 18 wherein a region above said panel is illuminated to illuminate fingers or portions thereof proximal to but spaced from said contact surface.

20. A human-machine interface as defined in claim 19 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact regions;

the instants contacts are made with said panel and the instants said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the duractions of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

21. A human-machine interface as defined in claim 20 wherein the normal forces are determined by examining the areas of the bright regions in said images.

22. A human-machine interface as defined in claim 21 further including a secondary light source to illuminate the region above said contact surface.

23. A human-machine interface as defined in claim 22 wherein said secondary light source produces a sheet of light above said contact surface, said sheet of light illuminating tips of fingers whose pads contact said contact surface sufficiently such that lunate highlights are included in images acquired by said detector, said processor using pixel values corresponding to the lunate highlights in said images to determine the central longitudinal axes of said fingers.

24. A human-machine interface as defined in claim 23 wherein shear forces are determined by comparing the positions of said bright regions with respect to the geometrical tips of the lunate highlights.

25. A human-machine interface as defined in claim 22 wherein said secondary light source illuminates said panel from the side opposite said contact surface, said secondary light source illuminating fingers whose pads contact said contact surface sufficiently such that outlines of said fingers are included in images acquired by detector, said processor using pixel values corresponding to the outlines of said fingers in said images to determine the central regions of said fingers.

26. A human-machine interface as defined in claim 25 wherein said shear forces are determined by comparing the positions of said bright regions with respect to the geometrical tips of the finger outlines.

27. A human-machine interface as defined in claim 1 wherein said energy source directs said energy towards a surface of said panel opposite said contact surface, said energy passing through said panel and escaping via said contact surface, energy being reflected off of objects in contact with said contact surface and propagating within said panel, said detector detecting energy propagating in said panel and acquiring images of said panel at selected intervals.

28. A human-machine interface as defined in claim 26 wherein said detector is optically coupled to said panel via a prism, and wherein each image is constituted by a plurality of pixels that attain pixel values proportional to their exposure to the at least one parameter of said energy, said pixel values being output by said detector.

29. A human-machine interface as defined in claim 28 further including a light valve associated with said prism, said light valve being actuable to couple selectively said detector and panel.

30. A human-machine interface as defined in claim 29 wherein said detector is oriented at an acute angle to said panel.

31. A human-machine interface as defined in claim 29 wherein said light valve is a shutter actuable to select different angles of view for said detector.

32. A human-machine interface as defined in claim 27 wherein said energy source directs light towards said surface at an acute angle.

33. A human-machine interface as defined in claim 32 wherein said surface and said contact surface are parallel and closely spaced to define a self-imaging waveguide.

34. A human-machine interface as defined in claim 33 further including a generally transparent sheet disposed on said contact surface, said sheet having a roughened surface and an index of refraction lower than that of said panel.

35. A human-machine interface as defined in claim 27 wherein contacts are made on said panel with a user'"'"'s fingers, said contact regions being generally elliptical in shape.

36. A human-machine interface as defined in claim 35 wherein a region above said panel is illuminated to illuminate fingers or portions thereof proximal to but spaced from said contact surface.

37. A human-machine interface as defined in claim 36 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact locations;

the instants contacts are made with said panel and the instants said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the durations of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

38. A human-machine interface as defined in claim 37 wherein said normal forces are calculated by determining the areas of said elliptical contact locations.

39. A human-machine interface as defined in claim 1 further including means to project an image onto said panel, said image being visible when looking at said contact surface.

40. A human-machine interface as defined in claim 39 wherein said energy source is a cathode ray tube (CRT) positioned relative to said panel opposite said contact surface, said CRT projecting said image on said panel by scanning said surface with an energy beam, objects in contact with said contact surface reflecting energy of said beam thereby to alter the same, said detector detecting reflected energy and outputting pixel values corresponding thereto.

41. A human-machine interface as defined in claim 40 wherein said detector includes red, blue, green (RBG) photodiodes to detect said reflected energy.

42. A human-machine interface as defined in claim 41 wherein said detector further includes signal dividers receiving red, green and blue intensity signals driving said CRT and the output of said photodiodes, said signal dividers calculating the reflectance of reflected energy thereby to generate said pixel values.

43. A human-machine interface as defined in claim 42 wherein said CRT is spaced from said panel, said human-machine interface further including at least one lens disposed between said CRT and said panel and at least one lens disposed between said RBG photodiodes and said panel.

44. A human-machine interface as defined in claim 41 wherein said CRT is proximal to said panel, said human-machine interface further including a sheet of transmissive material having at least one lens moulded therein, disposed on said panel and positioned between said panel and said CRT.

45. A human-machine interface as defined in claim 44 wherein said contact surface has a roughened surface.

46. A human-machine interface as defined in claim 1 wherein said energy source includes a scanner directing a beam of light into said panel and scanning said beam across said contact surface in a raster pattern.

47. A human-machine interface as defined in claim 46 wherein said scanner includes a light source to generate said beam of light and a moveable compound optical element disposed in the path of said beam, said compound optical element scanning said beam across said contact surface in said raster pattern as said compound optical element moves.

48. A human-machine interface as defined in claim 47 wherein said compound optical element includes at least two spaced, rotatable holographic optical elements (HOEs), each HOE including at least one ring of circumferentially spaced diffraction gratings, each of said diffraction gratings scanning said beam across a respective row of said raster pattern when said beam impinges thereon as said compound optical element is rotated.

49. A human-machine interface as defined in claim 48 wherein said scanner is off-axis and wherein said diffraction gratings have grating functions to compensate for keystoning.

50. A human-machine interface as defined in claim 48 wherein each HOE includes concentric rings of diffraction gratings, the diffraction gratings of one of said rings scanning said beam across said panel in said raster pattern, another of said rings projecting an image on said panel that is visible when looking at said contact surface.

51. A human-machine interface as defined in claim 48 further comprising an array of rows of diffraction gratings disposed on a surface of said panel opposite said contact surface, each row of diffraction gratings in said array being associated with a respective diffraction grating on said HOE and assisting to focus said beam on said contact surface during scanning of said beam across said respective row.

52. A human-machine interface as defined in claim 1 wherein said energy source includes a pair of scanners directing a beam of light into said panel and scanning said beam across said contact surface in a raster pattern.

53. A human-machine interface as defined in claim 52 wherein each of said scanners includes a pair of diffraction gratings movable relative to one another, movement of the diffraction gratings of one of said scanners scanning said beam across said panel in an X-direction and movement of the diffraction gratings of the other scanner moving said beam in a Y-direction.

54. A human-machine interface as defined in claim 53 wherein said panel includes mirrors at opposite edges thereof, said mirrors reflecting the beam entering said panel back and forth a plurality of times prior to impingement of said beam on said contact surface to lengthen the optical path of said beam within said panel.

55. A method for detecting contacts on a panel having a contact surface and being formed of energy transmissive material comprising steps of:

directing energy to the panel that is transmitted by the panel, at least one parameter of energy transmitted by the panel being altered at regions where contacts are made with the contact surface;

detecting the altered at least one parameter of said energy corresponding to contact regions on said contact surface and outputting values corresponding thereto; and

processing the output values to determine the locations of the contact regions on the contact surface and at least one other attribute associated with each of the contacts.

56. The method of claim 55 wherein said output values are processed to determine at least one of:

the normal forces applied to the panel at each of the contact regions;

shear forces applied to the panel at each of the contact regions;

the instants contacts are made with the panel and the instants the contacts are ended;

the velocities at which the contacts are made with the panel and the velocities at which the contacts are ended;

the duration of the contacts; and

oscillations in the normal forces applied to the panel during the contacts.

57. The method of claim 56 further comprising the step of projecting an image on the panel that is visible when looking at the contact surface.

58. An interface for use with a panel formed of energy transmissive material and having a contact surface on which one or more contacts may simultaneously be made, the panel transmitting energy received from an energy source to the contact surface wherein at least one parameter of the energy transmitted by the panel is altered at regions where contacts are made with the contact surface, said interface comprising:

a detector detecting the altered at least one parameter of said energy corresponding to contact regions on said contact surface and outputting values corresponding thereto; and

a processor in communication with said detector, said processor processing said output values to determine the locations of said contact regions on said contact surface and at least one other attribute associated with each of said contacts.

59. An interface as defined in claim 58 wherein said at least one other attribute includes the normal force applied to said panel at each of said contact regions.

60. An interface as defined in claim 59 wherein said at least one other attribute further includes shear forces applied to said panel at each of said contact regions.

61. An interface as defined in claim 58 wherein said locations are determined by calculating coordinates of said contact regions.

62. An interface as defined in claim 61 wherein said locations are defined by calculating the coordinates of the centers of said contact regions.

63. An interface as defined in claim 59 wherein said at least one other attribute further includes the instants contacts are made with said panel and the instants said contacts are ended.

64. An interface as defined in claim 63 wherein said at least one other attribute further includes the duration of each of said contacts.

65. An interface as defined in claim 59 wherein said at least one other attribute further includes oscillations in the normal forces applied to said panel during said contacts.

66. A display device to project an image comprising:

a panel formed of energy transmissive material and having a display surface on which said image is visible; and

a scanner to direct a beam of light into said panel and scanning said beam across said display surface in a raster pattern to project said image, said scanner including a light source to generate said beam of light and a movable compound optical element disposed in the path of said beam, said compound optical element scanning said beam across said display surface in said raster pattern as said compound optical element moves.

67. A display device as defined in claim 66 wherein said compound optical element includes at least two spaced, rotatable holographic optical elements (HOEs), each HOE including a ring of circumferentially spaced diffraction gratings, each of said diffraction gratings scanning said beam across a respective row of said raster pattern when said beam impinges thereon as said compound optical element rotates.

68. A display device as defined in claim 67 further comprising an array of diffraction gratings disposed on a surface of said panel opposite said display surface, each row of diffraction gratings in said array being associated with a respective diffraction grating on said HOE and assisting to focus said beam on said display surface during scanning of said beam across said respective row.

69. A display device as defined in claim 67 wherein said scanner is off-axis and wherein said diffraction gratings have grating functions to compensate for keystoning.