Computer data entry and manipulation apparatus and method

• US 4,988,981 A
• Filed: 02/28/1989
• Issued: 01/29/1991
• Est. Priority Date: 03/17/1987
• Status: Expired due to Term

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Original Assignee: VPL Research, Inc.
• Inventors: Zimmerman, Thomas G., Lanier, Jaron Z.
• Primary Examiner(s): Oberley, Alvin E.
• Application Number:
  US07/317,107
• Time in Patent Office: 700 Days
• Field of Search: 340/709, 340/706, 340/712, 340/705, 340/725, 340/6, 352/50
• US Class Current: 345/158
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1125 Grasping motions of hands
• CPC Class Codes: A61B 5/1126 using a particular sensing technique
• CPC Class Codes: A61B 5/1127 using markers
• CPC Class Codes: A61B 5/6806 Gloves
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Apparatus is disclosed for generating control signals for the manipulation of virtual objects in a computer system according to the gestures and positions of an operator'"'"'s hand or other body part. The apparatus includes a glove worn on the hand which includes sensors for detecting the gestures of the hand, as well as hand position sensing means coupled to the glove and to the computer system for detecting the position of the hand with respect to the system. The computer system includes circuitry connected to receive the gesture signals and the hand position signals for generating control signals in response thereto. Typically, the control signals are used to manipulate a graphical representation of the operator'"'"'s hand which is displayed on a monitor coupled to the computer system, and the graphical representations of the operator'"'"'s hand manipulates virtual objects or tools also displayed by the computer.

Claims

1. Apparatus for interacting with a computer program comprising:

display means connected to the computer for displaying objects on a screen;

glove means adapted to be worn on a hand of a user, the glove means including gesture sensing means coupled to the glove means for detecting flexure of fingers of the user'"'"'s hand, and position sensing means coupled to the glove means for detecting a position of the hand with respect to the display means;

interface means for coupling the glove means to the computer; and

control means for controlling a cursor indicated on the display means in response to and emulating the flexure of fingers and the position of the hand, the cursor being capable of interactivity acting upon a virtual object represented within the computer to allow communication and interaction between the user and the program.

2. Apparatus as in claim 1 wherein the position sensing means comprises transmitting means affixed to the glove for transmitting signals to receiving means disposed about the display means in close proximity thereto.

3. Apparatus as in claim 2 wherein the gesture sensing means further comprises flexure sensors affixed to the glove means to detect the extent of bending of the fingers of the user.

4. Apparatus as in claim 2 wherein the transmitting means transmits ultrasonic signals.

5. Apparatus as in claim 4 wherein the receiving means comprises a plurality of receivers of ultrasonic signals disposed about the screen.

6. Apparatus as in claim 5 wherein the plurality of receivers is three and the three receivers are not linearly arranged.

7. Apparatus as in claim 5 wherein the interface means includes circuitry coupled to the flexure sensors and to the plurality of receivers.

8. Apparatus as in claim 1 wherein the display means includes means for displaying the cursor as a representation of a hand which mirrors the position and flexure of fingers in the user'"'"'s hand.

9. Apparatus as in claim 4 wherein the transmitting means comprises a plurality of transmitters.

10. Apparatus as in claim 1 wherein the interface means includes a cable connecting the glove means to the computer.

11. Apparatus as in claim 1 wherein the interface means includes an electromagnetic transmitter connecting the glove means to the computer.

12. Apparatus as in claim 1 wherein the interface means includes an optical transmitter/receiver connecting the glove means to the computer.

13. An apparatus for controlling a computer display of the type having a virtual object depicted thereon that is used for communicating and interacting with a computer program comprising:

flex sensing means disposed in close proximity to a part of the body of the user, for sensing flexure of the associated part of the body of the user; and

cursor display means, coupled to the flex sensing means and to the computer display, for displaying a cursor emulating the flexure of the part of the body, the cursor being capable of interactivity acting upon the virtual object to allow communication and interaction between the user and the computer program.

14. The apparatus according to claim 13 wherein the flex sensing means senses the degree of flexure of the part of the body.

15. The apparatus according to claim 14 wherein the cursor display means displays a cursor depicting the degree of flexure of the part of the body.

16. The apparatus according to claim 13 wherein the flex sensing means senses flexure of a user'"'"'s hand.

17. The apparatus according to claim 16 wherein the flex sensing means senses flexure of a finger of the user'"'"'s hand.

18. The apparatus according to claim 17 wherein the flex sensing means senses the degree of flexure of the user'"'"'s hand.

19. The apparatus according to claim 18 wherein the cursor display means displays the degree of flexure of the user'"'"'s hand.

20. The apparatus according to claim 19 wherein the flex sensing means is disposed in a glove that is worn on the user'"'"'s hand.

21. The apparatus according to claim 20 wherein the cursor display means displays a representation of a hand which mirrors the flexure of fingers of the user'"'"'s hand.

22. The apparatus according to claim 21 further comprising position sensing means for sensing the position of the user'"'"'s hand with respect to the display.

23. The apparatus according to claim 22 wherein the cursor display means is coupled to the position sensing means and displays the position of the user'"'"'s hand with respect to the display.

24. The apparatus according to claim 13 wherein the flex sensing means senses the degree of flexure of the part of the user'"'"'s body and provides an analog signal to the cursor display means that indicates the degree of flexure of the part of the body;

and wherein the cursor display means depicts the flexure of the part of the body based on the intensity of the analog signal.

25. The apparatus according to claim 24 wherein the analog signal is a function of voltage.

26. An apparatus of interacting with a virtual object represented within a computer, the virtual object being used for communicating and interacting with a computer program, comprising:

position sensing means, disposed in close proximity to a part of a body of a user for movement therewith, for sensing the position of the associated part of the body of the user with respect to the computer;

flex sensing means, disposed in close proximity to a part of the user'"'"'s body for movement therewith, for sensing flexure of the associated part of the user'"'"'s body;

interface means for coupling the position sensing means and the flex sensing means to the computer and for controlling movement of a cursor represented within the computer in response to the position sensing means and the flex sensing means, the cursor emulating the position and flexure of the part of the user'"'"'s body for interactivity acting upon the virtual object to allow communication and interaction between the user and the program; and

wherein the computer includes contact detecting means for detecting contact between the cursor and the virtual object.

27. The apparatus according to claim 26 wherein the virtual object may be manipulated by the cursor and wherein the interface means includes object manipulating means for manipulating the virtual object with the cursor in response to the contact detecting means and in response to the position of the part of the user'"'"'s body with respect to the computer.

28. The apparatus according to claim 27:

wherein the part of the user'"'"'s body is a hand.

29. The apparatus according to claim 28 wherein the flex sensing means detects flexure of a finger of the user'"'"'s hand.

30. The apparatus according to claim 29 wherein the interface means represents within the computer a virtual hand which mirrors the position and flexure of the fingers of the user'"'"'s hand.

31. The apparatus according to claim 30 wherein the contact detecting means detects when the virtual object is grasped by the virtual hand.

32. The apparatus according to claim 31 wherein the flex sensing means detects the degree of flexure of the fingers of the user'"'"'s hand, and wherein the interface means represents within the computer the degree of flexure of the fingers of the user'"'"'s hand.

33. The apparatus according to claim 32 further comprising display means for visually displaying the cursor as the position and flexure of the fingers of the user'"'"'s hand.

34. The apparatus according to claim 33 wherein the display means visually displays the virtual object together with the cursor.

35. The apparatus according to claim 34, wherein the interface means causes the virtual object to be manipulated according to commands represented by the position and flexure of the fingers of the user'"'"'s hand.

36. The apparatus according to claim 35 wherein the interface means causes an effect of the manipulation of the virtual object to be substantially the same as if a real object depicted by the virtual object were being manipulated in the same way of the user'"'"'s hand.

37. The apparatus according to claim 36 wherein the user'"'"'s hand does not act upon a real object depicted by the virtual object.

38. The apparatus according to claim 36 wherein the interface means causes the virtual object to be coupled to the virtual hand and to move together with the virtual hand when the virtual object is grasped by the virtual hand.

39. The apparatus according to claim 38 wherein the interface means further comprises linkage means for linking action of the virtual object to a second virtual object.

40. The apparatus according to claim 39 wherein the position sensing means and the flex sensing means operate without using a camera.

41. The apparatus according to claim 39 wherein the virtual object is a tool which acts upon the second virtual object.

42. The apparatus according to claim 26 wherein the flex sensing means senses the degree of flexure of the part of the user'"'"'s body and provides and analog signal to the interface means that indicates the degree of flexure of the part of the body;

and wherein the interface means depicts the flexure of the part of the body based on the intensity of the analog signal.

43. The apparatus according to claim 42 wherein the analog signal is a function of voltage.

44. The apparatus according to claim 28 further comprising display means for visually displaying the cursor as the position and flexure of the part of the user'"'"'s body with respect to the display.

45. The apparatus according to claim 44 wherein the display means visually displays the virtual object together with the cursor.

46. An apparatus for controlling a computer display of the type having a virtual object depicted thereon that is used for communicating and interacting with a computer program comprising:

orientation sensing means, disposed in close proximity to a part of a body of a user, for sensing the orientation of the associated part of the body of the user with respect to the display; and

cursor display means, coupled to the orientation sensing means and to the computer display, for displaying a cursor emulating the orientation of the part of the user'"'"'s body, the cursor being capable of interactivity acting upon the virtual object to allow communication and interaction between the user and the computer program.

47. The apparatus according to claim 46 further comprising flex sensing means, disposed in close proximity to a part of the user'"'"'s body, for sensing flexure of the part of the user'"'"'s body.

48. The apparatus according to claim 47 wherein the cursor display means is coupled to the flex sensing means and displays a cursor depicting the flexure of the part of the user'"'"'s body.

49. The apparatus according to claim 48 wherein the flex sensing means causes the degree of flexure of the part of the user'"'"'s body.

50. The apparatus according to claim 49 wherein the cursor display means displays a cursor depicting the degree of flexure of the part of the user'"'"'s body.

51. The apparatus according to claim 50 wherein the part of the user'"'"'s body is the user'"'"'s hand.

52. The apparatus according to claim 51 wherein the flex sensing means senses flexure of a finger of the user'"'"'s hand.

53. The apparatus according to claim 52 wherein the flex sensing means senses the degree of flexure of the user'"'"'s hand.

54. The apparatus according to claim 53 wherein the cursor display means displays the degree of flexure of the user'"'"'s hand.

55. The apparatus according to claim 54 wherein the flex sensing means and the orientation sensing means are disposed in a glove that is worn on the user'"'"'s hand.

56. The apparatus according to claim 55 wherein the cursor display means displays a virtual hand which mirrors the orientation and flexure of fingers of the user'"'"'s hand.

57. The apparatus according to claim 56 further comprising position sensing means for sensing the position of the user'"'"'s hand with respect to the display.

58. The apparatus according to claim 57 wherein the cursor display means is coupled to the position sensing means and depicts the position of the user'"'"'s hand with respect to the display.

59. The apparatus according to claim 58 wherein the cursor display means further comprises contact detecting means for detecting when the virtual object is grasped by the virtual hand.

60. The apparatus according to claim 59 wherein the cursor display means includes object manipulating means for manipulating the virtual object in response to the contact detecting means and in response to the position, flexure, and orientation of the user'"'"'s hand.

61. The apparatus according to claim 46 wherein the cursor display means further comprises means for displaying a cursor depicting the roll orientation of the part of the user'"'"'s body.

62. The apparatus according to claim 46 wherein the cursor display means further comprises means for displaying a cursor depicting the pitch orientation of the part of the user'"'"'s body.

63. The apparatus according to claim 46 wherein the cursor display means further comprises means for displaying a cursor depicting the yaw orientation of the part of the user'"'"'s body.

64. The apparatus according to claim 46 wherein the cursor display means further comprises means for displaying a cursor depicting the pitch, roll and yaw orientation of the part of the body.

65. The apparatus according to claim 47 wherein the flex sensing means sense the degree of flexure of the part of the user'"'"'s body and provides an analog signal to the interface means that indicates the degree of flexure of the part of the body;

and wherein the interface means depicts the flexure of the part of the body based on the intensity of the analog signal.

66. The apparatus according to claim 65 wherein the analog signal is a function of voltage.