Man machine interfaces for entering data into a computer

• US 6,008,800 A
• Filed: 08/15/1994
• Issued: 12/28/1999
• Est. Priority Date: 09/18/1992
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Timothy R. Pryor
• Inventors: Pryor, Timothy R.
• Primary Examiner(s): NGUYEN, CHANH DUY
• Application Number:
  US08/290,516
• Time in Patent Office: 1,961 Days
• Field of Search: 340/706, 340/709, 340/710, 340/711, 340/707, 356/237, 356/372, 356/375, 356/376, 356/388, 356/389, 345/156, 345/157, 345/173, 345/174, 345/175-176, 345/180
• US Class Current: 345/173
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An optical method and apparatus for computer input, most typically man-machine interfaces, particularly of the class known as "touch screens" and "digitizers," are based on surface distortion imaging, i.e., magnified optical imaging of a surface temporarily distorted by the input data of interest such as a finger, pencil point, key impact, projectile, etc. Typically, the surface (or material) distorted or disturbed, is extensive, that is large relative to the size of the distortion(s). This imaging allows 3 dimensional displacement X, Y, Z and vector inputs, low cost, high speed, tactile feedback, and an ability to be used on any size or type of surface or screen, from small tablet sized data entry devices to wall murals and beyond, with essentially the same hardware configurations.

Claims

1. A method for entering data to a computer or data storage device, comprising the steps of:

(a) generating, via an input on a front surface of an extensive material having thickness separating the front surface from a back surface of said material, a temporary disturbance or distortion on said back surface;

(b) electro-optically forming an image of at least a substantial portion of said back surface onto an electro-optical imaging means, said image including an image of said temporary disturbance or distortion of said back surface;

(c) determining from said image of said back surface at least one characteristic of said input;

(d) generating at least one data entry signal representative of the determined characteristic of said input; and

(e) entering said data signal to said computer or data storage device.

2. A method according to claim 1 wherein steps (b) and (c) occur substantially simultaneously for a plurality of separate discrete areas of input on said front surface.

3. A method according to claim 1 wherein electro-optical image forming step (b) is accomplished with a TV camera.

4. A method according to claim 3 wherein an image of substantially the entire area of said back surface is formed on said camera.

5. A method according to claim 3 wherein an image of a grid on said back surface is formed on said camera.

6. A method according to claim 3 wherein said image forming step (b) comprises stroboscopically illuminating said back surface.

7. A method according to claim 3 wherein said image-forming step (b) comprises detecting light retro-reflected to said back surface.

8. A method according to claim 1 wherein step (c) includes determining from said image a vector along a longitudinal axis of distortion of said back surface.

9. A method according to claim 1 wherein said extensive material comprises a display screen for TV images.

10. A method according to claim 1 wherein step (c) comprises comparing instant data to a previously recorded electro-optical image of the back surface prior to said temporary disturbance or distortion.

11. A method according to claim 1 wherein step (d) comprises generating said data entry signal with respect to a previous condition of the surfaces.

12. A method according to claim 1 wherein step (c) comprises obtaining a signature of said back surface.

13. A method according to claim 12 wherein said signature comprises a distortion or disturbance of said back surface at a plurality of points.

14. A method according to claim 12 further comprising utilizing a characteristic of said signature to identify at least a portion of a distortion or disturbance for generating said data entry signal.

15. A method according to claim 1 further comprising providing tactile feedback in response to said disturbance or distortion generating step (a).

16. A method according to claim 1 wherein the input of step (a) comprises applying a force to said front surface, said force being produced by at least one of pressure, vacuum, heat, electromagnetism, and weight.

17. A method according to claim 1 wherein the input of step (a) comprises striking said front surface with an object.

18. A method according to claim 1 wherein said extensive material comprises a single membrane element.

19. A method according to claim 1 wherein said extensive material comprises multiple substantially independently movable elements.

20. A method according to claim 1 wherein said image-forming step (b) comprises forming an image of substantially the entire area of said back surface simultaneously.

21. A method according to claim 1 wherein said extensive material is contoured when undistorted.

22. A method according to claim 1 wherein said determining step (c) comprises determining the location of said temporary distortion or disturbance and determining an intensity centroid of light reflected from said back surface.

23. A method according to claim 1 wherein said determining step (c) comprises detecting light above or below a given threshold intensity.

24. A method according to claim 1 wherein, in step (a), transmission of light through said material occurs simultaneously with said distortion due to photoelastic properties of said material under stress of said distortion.

25. A method according to claim 1 wherein, in step (c), said at least one characteristic comprises the location of said input, and wherein step (c) includes determining the source of said input from said location characteristic.

26. A method according to claim 1 wherein said extensive material is large with respect to the temporary distortion or disturbance.

27. A method according to claim 1 wherein the image formed in step (b) comprises a "D Sight" image.

28. A method according to claim 1 wherein the image formed in step (b) comprises a triangulation image.

29. A method according to claim 1 wherein said temporary distortion or disturbance comprises a convex distortion of said back surface.

30. Apparatus for entering data to a computer or a data storage device, comprising:

(a) a video monitor having an extensive video display screen for viewing video images, said screen having a front surface and a back surface;

(b) means for generating, via an input on said front surface, a temporary disturbance or distortion on said back surface of said screen;

(c) image-forming means for electro-optically forming thereon an image of at least a substantial portion of said back surface, said image including an image of said temporary disturbance or distortion of said back surface;

(d) determining means for determining from said image of said back surface at least one characteristic of said input and generating at least one data entry signal representative of the determined characteristic of said input; and

(f) means for entering said data entry signal to said computer or data storage device.

31. Apparatus according to claim 30 further comprising means for actuating said image-forming means and said determining means substantially simultaneously for a plurality of discrete areas of input on said front surface.

32. Apparatus according to claim 30 wherein said image forming means comprises a TV camera.

33. Apparatus according to claim 32 wherein the field of view of said TV camera includes substantially the entire area of said back surface of said screen.

34. Apparatus according to claim 33 further comprising a grid on said back surface.

35. Apparatus according to claim 30 wherein said determining means comprises means for determining a vector along a longitudinal axis of a distortion of said back surface.

36. Apparatus according to claim 30 wherein said determining means comprises means for comparing said image to a previously recorded electro-optical image of the back surface.

37. Apparatus according to claim 30 further comprising means for providing tactile feedback in response to said input on said front surface.

38. Apparatus according to claim 30 wherein said image-forming means comprises means for forming an image of substantially the entire area of said back surface simultaneously.

39. Apparatus according to claim 30 wherein said screen is contoured when distorted.

40. Apparatus according to claim 30 wherein said image-forming means comprises means for stroboscopically illuminating said back surface.

41. Apparatus according to claim 30 wherein said image-forming means comprises means for detecting light retro-reflected to said back surface.

42. Apparatus according to claim 30 wherein said characteristic determining means comprises means for determining the location of said temporary distortion and comprises means for determining an intensity centroid of light reflected from said back surface.

43. Apparatus according to claim 30 wherein said image forming means comprises means for forming a "D sight" image.

44. Apparatus according to claim 30 wherein said image forming means comprises means for forming a triangulation image.

45. An interactive video system comprising;

(a) a plurality of targets connectable to a user;

(b) detector means for monitoring the location and motion of said targets and for outputting target data based on the monitored location and motion of said targets;

(c) a user activity processor for receiving said target data and for outputting user data;

(d) a video monitor having an extensive display screen for displaying video images, said screen having a front surface and a back surface;

(e) means for generating, via an input on said front surface of said screen, a temporary disturbance or distortion on said back surface;

(f) image-forming means for electro-optically forming thereon an image of at least a substantial portion of said back surface of said screen, said image including an image of said temporary disturbance or distortion of said back surface;

(g) determining means for determining from said image of said back surface at least one characteristic of said input and

generating at least one data entry signal representative of the determined characteristic of said input;

(h) means for entering said data entry signal into a control processor; and

(i) a control processor for receiving said data entry signal and said user data, and for providing an output in accordance with said data entry signal and said user data.

46. An interactive video system as recited in claim 45, wherein said output from said control processor comprises a video image displayed on said front surface.

47. An interactive video system as recited in claim 45, wherein said targets comprise reflectors and said detector means comprise video cameras.

48. An interactive video system as recited in claim 45, further comprising storage means for storing said user data.