Human-machine interface

• US 7,859,519 B2
• Filed: 05/01/2000
• Issued: 12/28/2010
• Est. Priority Date: 05/01/2000
• Status: Expired due to Fees

• Current Assignee: David J. Tulbert
• Original Assignee: David J. Tulbert
• Inventors: Tulbert, David J.
• Primary Examiner(s): Nguyen; Jimmy H
• Application Number:
  US09/562,987
• Publication Number:
  US 20030137494A1
• Time in Patent Office: 3,893 Days
• Field of Search: 345/173, 345/174, 345/175, 345/176, 345/179, 345/180, 178/18.09, 178/19.11
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/0423 using sweeping light beams, e.g. using rotating or vibrating mirror
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A human-machine interface includes a panel formed of energy transmissive material having a contact surface on which one or more contacts may simultaneously be made. An energy source directs energy to the panel. The panel transmits energy received from the energy source to the contact surface. At least one parameter of the energy transmitted by the panel is altered at regions where contacts are made with the contact surface. A detector is coupled to the panel and detects the at least one parameter of the energy generally over the area of the contact surface and outputs values corresponding thereto. A processor is in communication with the detector. The processor processes the output values to determine the locations of the contact regions on the contact surface and at least one other attributed associated with each of the contacts.

Claims

1. A human-machine interface comprising:

a panel formed of energy transmissive material and having a substantially non-deformable major contact surface on which one or more object contacts is made and a second major surface generally parallel with said major contact surface;

an energy source directing energy to said panel, said panel transmitting energy received from said energy source to said major contact surface, at least one parameter of energy transmitted by said panel being altered at regions where object contacts are made with said major contact surface;

an image acquisition device acquiring, through said panel via said second major surface, an image generally over the area of the major contact surface including at least the alterations of the at least one parameter of the energy representative of said object; and

a processor in communication with said image acquisition device, said processor processing said image to determine the location of each contact region on said major contact surface and at least one other attribute of each contact.

2. A human-machine interface as defined in claim 1 wherein said at least one other attribute includes the normal force applied to said panel at each of said contact regions.

3. A human-machine interface as defined in claim 2 wherein said at least one other attribute further includes oscillations in the normal forces applied to said panel during said contacts.

4. A human-machine interface as defined in claim 1 wherein said at least one other attribute includes shear forces applied to said panel at each of said contact regions.

5. A human-machine interface as defined in claim 1 wherein said locations are determined by calculating coordinates of said contact regions.

6. A human-machine interface as defined in claim 5 wherein said locations are determined by calculating the coordinates of the centers of said contact regions.

7. A human-machine interface as defined in claim 1 wherein said at least one other attribute further includes the instant contacts are made with said panel and the instant said contacts are ended.

8. A human-machine interface as defined in claim 7 wherein said at least one other attribute further includes the duration of each of said contacts.

9. A human-machine interface as defined in claim 1 wherein contacts are made on the major contact surface of said panel with a user'"'"'s fingers, contact between the panel and said fingers altering the direction of energy transmitted by said panel, said contact regions being generally elliptical in shape.

10. A human-machine interface as defined in claim 9 wherein said processor calculates the normal forces applied to said panel at each of said contact regions by determining the areas of said elliptical contact regions.

11. A human-machine interface as defined in claim 1 further comprising a secondary light source illuminating a region over said panel thereby to illuminate objects or portions thereof proximal to but spaced from said major contact surface.

12. A human-machine interface as defined in claim 11 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact regions;

the instant contacts are made with said panel and the instant said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the durations of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

13. A human-machine interface as defined in claim 12 wherein said processor determines the positions of the tips of fingers contacting said panel and calculates the shear forces by determining the coordinates of displaced finger tips with respect to the centers of said contact regions.

14. A human-machine interface as defined in claim 12 wherein said image acquisition device acquires images at selected intervals, each of said images including a plurality of pixels, each of said pixels attaining a pixel value proportional to its exposure to the at least one parameter of said energy during said interval, said processor generating histograms of said pixel values for pixels corresponding to contact regions in said images.

15. A human-machine interface as defined in claim 11 wherein said processor further calculates distances at which said objects or portions thereof are spaced above said major contact surface.

16. A human-machine interface as defined in claim 1 wherein energy directed to said panel is transmitted within said panel to said major contact surface as a result of total internal reflection.

17. A human-machine interface as defined in claim 16 wherein said energy source is a light source, said at least one parameter is the intensity of light, and light transmitted within said panel escapes from said panel at said contact regions to alter said energy.

18. A human-machine interface as defined in claim 17 wherein said image acquisition device acquires images of said panel at selected intervals, each image including a plurality of pixels that attain pixel values proportional to their exposure to light intensity, said pixel values being output by said image acquisition device.

19. A human-machine interface as defined in claim 18 wherein said light source radiates light, said image acquisition device is positioned adjacent said second major surface, and said light that escapes from said panel at said contact regions appears as intense regions.

20. A human-machine interface as defined in claim 19 further comprising a secondary light source illuminating a region over said panel to illuminate objects or portions thereof proximal to but spaced from said major contact surface.

21. A human-machine interface as defined in claim 20 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact regions;

the instant contacts are made with said panel and the instant said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the durations of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

22. A human-machine interface as defined in claim 21 wherein the normal forces are determined by examining the areas of the intense regions in said images.

23. A human-machine interface as defined in claim 22 wherein said secondary light source produces a sheet of light over said major contact surface, said sheet of light illuminating tips of fingers whose pads contact said major contact surface sufficiently such that lunate highlights are included in images acquired by said image acquisition device, said processor using pixel values corresponding to the lunate highlights in said images to determine the central longitudinal axes of said fingers.

24. A human-machine interface as defined in claim 23 wherein shear forces are determined by comparing the positions of said intense regions with respect to the geometrical tips of the lunate highlights.

25. A human-machine interface as defined in claim 22 wherein said secondary light source illuminates said panel from the second major surface, said secondary light source illuminating fingers whose pads contact said major contact surface sufficiently such that outlines of said fingers are included in images acquired by said image acquisition device, said processor using pixel values corresponding to the outlines of said fingers in said images to determine the central regions of said fingers.

26. A human-machine interface as defined in claim 25 wherein said shear forces are determined by comparing the positions of said intense regions with respect to the geometrical tips of the finger outlines.

27. A human-machine interface as defined in claim 26 wherein said image acquisition device is optically coupled to said panel via a prism, and wherein each image includes a plurality of pixels that attain pixel values proportional to their exposure to the at least one parameter of said energy, said pixel values being output by said image acquisition device.

28. A human-machine interface as defined in claim 27 further including a light valve associated with said prism, said light valve being actuable to couple selectively said image acquisition device and panel.

29. A human-machine interface as defined in claim 28 wherein said image acquisition device is oriented at an acute angle to said panel.

30. A human-machine interface as defined in claim 28 wherein said light valve is a shutter actuable to select different angles of view for said image acquisition device.

31. A human-machine interface as defined in claim 22 wherein said processor further calculates distances at which said objects or portions thereof are spaced above said major contact surface.

32. A human-machine interface as defined in claim 20 wherein said processor further calculates distances at which said objects or portions thereof are spaced above said major contact surface.

33. A human-machine interface as defined in claim 1 wherein said energy source directs said energy towards said second major surface, said energy passing through said panel and escaping via said major contact surface, energy being reflected off of objects in contact with said major contact surface and propagating within said panel, said image acquisition device detecting energy propagating in said panel and acquiring images of said panel at selected intervals.

34. A human-machine interface as defined in claim 23 wherein said energy source directs light towards said second major surface at an acute angle.

35. A human-machine interface as defined in claim 34 wherein said second major surface and said major contact surface are precisely parallel and closely spaced to define a self-imaging waveguide.

36. A human-machine interface as defined in claim 35 further including a generally transparent sheet disposed on said major contact surface, said sheet having a roughened surface.

37. A human-machine interface as defined in claim 33 wherein contacts are made on said panel with a user'"'"'s fingers, said contact regions being generally elliptical in shape.

38. A human-machine interface as defined in claim 37 wherein a region over said panel is illuminated to illuminate fingers or portions thereof proximal to but spaced from said major contact surface.

39. A human-machine interface as defined in claim 38 wherein said at least one other attribute includes one or more of:

the normal forces applied to said panel at each of said contact regions;

shear forces applied to said panel at each of said contact locations;

the instant contacts are made with said panel and the instant said contacts are ended;

the velocities at which the contacts are made with said panel and the velocities at which said contacts are ended;

the durations of said contacts; and

oscillations in the normal forces applied to said panel during said contacts.

40. A human-machine interface as defined in claim 39 wherein said normal forces are calculated by determining the areas of said elliptical contact locations.

41. A human-machine interface as defined in claim 1 further including means to project an image onto said panel, said image being visible on said major contact surface.

42. A human-machine interface as defined in claim 41 wherein said energy source is a cathode ray tube (CRT) positioned adjacent said second major surface, said CRT projecting said image on said panel by scanning said major contact surface with an energy beam, objects in contact with said major contact surface reflecting energy of said beam thereby to alter the same, said image acquisition device detecting reflected energy and outputting pixel values corresponding thereto.

43. A human-machine interface as defined in claim 42 wherein said image acquisition device includes red, blue, green (RBG) photodiodes to detect said reflected energy.

44. A human-machine interface as defined in claim 43 wherein said image acquisition device further includes signal dividers receiving red, green and blue intensity signals driving said CRT and the output of said photodiodes, said signal dividers calculating the reflectance of reflected energy thereby to generate said pixel values.

45. A human-machine interface as defined in claim 44 wherein said CRT is spaced from said panel, said human-machine interface further including at least one lens disposed between said CRT and said panel and at least one lens disposed between said RBG photodiodes and said panel.

46. A human-machine interface as defined in claim 43 wherein said CRT is proximal to said panel, said human-machine interface further including a sheet of transmissive material having at least one lens molded therein, disposed on said panel and positioned between said panel and said CRT.

47. A human-machine interface as defined in claim 46 wherein said major contact surface has a roughened surface.

48. A human-machine interface as defined in claim 1 wherein said energy source includes a scanner directing a beam of light into said panel and scanning said beam across said major contact surface in a raster pattern.

49. A human-machine interface as defined in claim 48 wherein said scanner includes a light source to generate said beam of light and a movable compound optical element disposed in the path of said beam, said compound optical element scanning said beam across said major contact surface in said raster pattern as said compound optical element moves.

50. A human-machine interface as defined in claim 49 wherein said compound optical element includes at least two spaced, rotatable holographic optical elements (HOEs), each HOE including at least one ring of circumferentially spaced diffraction gratings, each of said diffraction gratings scanning said beam across a respective row of said raster pattern when said beam impinges thereon and as said compound optical element is rotated.

51. A human-machine interface as defined in claim 50 wherein said scanner is off-axis and wherein said diffraction gratings have grating functions to compensate for keystoning.

52. A human-machine interface as defined in claim 50 wherein each HOE includes concentric rings of diffraction gratings, the diffraction gratings of one of said rings scanning said beam across said panel in said raster pattern, another of said rings projecting an image on said panel that is visible when looking at said major contact surface.

53. A human-machine interface as defined in claim 50 further comprising an array of rows of diffraction gratings disposed on a surface of said panel opposite said major contact surface, each row of diffraction gratings in said array being associated with a respective diffraction grating on said HOE and assisting to focus said beam on said major contact surface during scanning of said beam across said respective row.

54. A human-machine interface as defined in claim 1 wherein said energy source includes a pair of scanners directing a beam of light into said panel and scanning said beam across said major contact surface in a raster pattern.

55. A human-machine interface as defined in claim 54 wherein each of said scanners includes a pair of diffraction gratings movable relative to one another, wherein movement of the diffraction gratings of one of said scanners provides scanning of said beam across said panel in an X-direction and movement of the diffraction gratings of the other scanner provides movement of said beam in a Y-direction.

56. A human-machine interface as defined in claim 55 wherein said panel includes mirrors at opposite edges thereof, said mirrors reflecting the beam entering said panel back and forth a plurality of times prior to impingement of said beam on said major contact surface to lengthen the optical path of said beam within said panel.

57. A method of detecting contacts on a panel having a substantially non-deformable major contact surface and a second major surface generally parallel with said contact surface and being formed of energy transmissive material comprising steps of:

directing energy to the panel, said panel transmitting energy to said major contact surface and at least one parameter of energy transmitted by the panel being altered at regions where contacts are made with the major contact surface;

using an image acquisition device to acquire, through said panel via said second major surface, an image generally over the area of the major contact surface including at least the alterations of the at least one parameter of the energy representative of at least one object contact with said major contact surface; and

processing said image to determine the location of each contact region on the major contact surface and at least one other attribute of each contact.

58. The method of claim 57 further comprising illuminating a region over said major contact surface.

59. The method of claim 58 wherein said image is processed to determine at least one of:

the normal forces applied to the panel at each of the contact regions;

shear forces applied to the panel at each of the contact regions;

the instant contacts are made with the panel and the instant the contacts are ended;

the velocities at which the contacts are made with the panel and the velocities at which the contacts are ended;

the durations of the contacts; and

oscillations in the normal forces applied to the panel during the contacts.

60. The method of claim 59 further comprising the step of projecting an image on the panel that is visible when looking at the major contact surface.

61. An interface for use with a panel formed of energy transmissive material and having a substantially non-deformable major contact surface on which one or more contacts is simultaneously made and a second major surface generally parallel with said major contact surface, the panel transmitting energy received from an energy source to the major contact surface wherein at least one parameter of the energy transmitted by the panel is altered at regions where contacts are made with the major contact surface, said interface comprising:

an image acquisition device acquiring, through said panel via said second major surface, an image generally over the area of the major contact surface including at least the alterations of the at least one parameter of the energy representative of said object; and

a processor in communication with said image acquisition device, said processor processing said image to determine the location of each contact region on the major contact surface and at least one other attribute of each contact.

62. An interface as defined in claim 61 wherein said at least one other attribute includes the normal force applied to said panel at each of said contact regions.

63. An interface as defined in claim 62 further comprising a light source illuminating a region over said major contact surface and wherein said at least one other attribute further includes shear forces applied to said panel at each of said contact regions.

64. An interface as defined in claim 62 wherein said at least one other attribute further includes the instant contacts are made with said panel and the instant said contacts are ended.

65. An interface as defined in claim 64 wherein said at least one other attribute further includes the duration of each of said contacts.

66. An interface as defined in claim 62 wherein said at least one other attribute further includes oscillations in the normal forces applied to said panel during said contacts.

67. An interface as defined in claim 61 wherein said locations are determined by calculating coordinates of said contact regions.

68. An interface as defined in claim 67 wherein said locations are determined by calculating the coordinates of the centers of said contact regions.

69. A human-machine interface comprising:

a panel formed of energy transmissive material and having a substantially non-deformable major contact surface on which one or more object contacts is made and a second major surface generally parallel with said major contact surface;

an energy source directing energy into said panel, said energy propagating within said panel, energy escaping from said panel in response to each object contact with said major contact surface;

an image acquisition device acquiring, through said panel via said second major surface, an image generally over the area of the major contact surface including at least alterations of at least one parameter of said escaping energy representative of said object; and

a processor in communication with said image acquisition device, said processor processing said image to determine the location of each contact region on said major contact surface and at least one other attribute of each contact.

70. A human-machine interface according to claim 69 wherein the energy source directs energy into a side of said panel.

71. A human-machine interface according to claim 69 wherein the energy escaping from said panel at each contact region illuminates the object in contact with said contact region.

72. A human-machine interface comprising:

a panel formed of energy transmissive material and having a substantially non-deformable major contact surface on which one or more object contacts is made and a second major surface generally parallel with said major contact surface;

an energy source directing energy to said panel, said panel transmitting energy received from said energy source to said major contact surface, at least one parameter of energy transmitted by said panel being altered at regions where object contacts are made with said major contact surface;

an image acquisition device having a field of view aimed at said second major surface, said image acquisition device acquiring, through said panel and from a region above said major contact surface, an image generally over the area of said major contact surface including at least the alterations of the at least one parameter of the transmitted energy representative of said object; and

a processor in communication with said image acquisition device, said processor processing said image to determine the location of each contact region on said major contact surface and at least one other attribute of each contact.