Method and apparatus for providing a cursor control interface with force feedback

• US 5,701,140 A
• Filed: 02/16/1996
• Issued: 12/23/1997
• Est. Priority Date: 07/16/1993
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Human Interface Corporation
• Inventors: Rosenberg, Louis B., Jackson, Bernard G.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Acosta, Joseph
• Application Number:
  US08/583,032
• Time in Patent Office: 676 Days
• Field of Search: 345/157, 345/158, 345/162, 345/156
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G01B 5/008 using coordinate measuring machines
• CPC Class Codes: G01B 7/004 for measuring coordinates of points
• CPC Class Codes: G05B 19/4207 in which a model is traced or scanned and corresponding data recorded
• CPC Class Codes: G05G 5/03 Means for enhancing the operator's awareness of arrival of the controlling member at a command or datum position; Providing feel, e.g. means for creating a counterforce arrangements for indicating the position of the controlling member G05G1/015
• CPC Class Codes: G05G 9/04737 with six degrees of freedom
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: H01H 2003/008 with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction

Abstract

A method and apparatus for use with a computer for providing commands to a computer through tracked manual gestures and for providing feedback to the user through forces applied to the interface. A user manipulatable object is coupled to a mechanical linkage which is, in turn, supportable on a fixed surface. The mechanical linkage or the user manipulatable object is tracked by sensors for sensing the location and/or orientation of the object. A multi-processor system architecture is disclosed wherein a host computer system is interfaced with a dedicated microprocessor which is responsive to the output of the sensors and provides the host computer with information derived from the sensors. The host computer has an application program which responds to the information provided via the microprocessor and which can provide force-feedback commands back to the microprocessor, which is felt by a user via the user manipulatable object.

Claims

1. A human interface device for enabling manual interactions with application software running on a host computer, said software providing images displayed on a computer display, said device comprising:

(a) a user manipulatable physical object;

(b) a support mechanism which supports said user manipulatable physical object while allowing a plurality of degrees of freedom of said user manipulatable physical object with respect to a surface;

(c) a sensor apparatus coupled to at least one of said user manipulatable physical object and said support mechanism and that produces a locative signal responsive to and corresponding with the position of the user manipulatable physical object with respect to said surface at points in time during normal operation;

(d) a communication bus coupled to said host computer;

(e) a device microprocessor separate from said host computer and coupled to said host computer by said communication bus, said device microprocessor being coupled to said sensor apparatus, said device microprocessor running a program contained at least in part in a non-volatile memory coupled to said device microprocessor and separate from said host computer, said device microprocessor providing information for use by said host computer running an application program simultaneously with said microprocessor running said program, said information including a representation of said locative signal,

wherein said application program of said host computer can provide images on a computer display, said images updated on said computer display in response to said locative signal, and

wherein said host computer can provide host commands, said host commands being communicated to said device microprocessor by said communication bus, wherein said device microprocessor;

(i) monitors said communication bus for said host commands; and

(ii) decodes said host commands, wherein

at least one of said host commands causes information to be reported from said device microprocessor to said host computer, and

at least one of said host commands causes said device microprocessor to output control signals to cause a force to be imposed on said user manipulative physical object, said at least one host command and said force being correlated with at least one of said images developed by said host computer on said computer display; and

(f) a force generator controlled by said device microprocessor for providing a resistance force to motion of said user manipulatable physical object in response to at least one of said control signals.

2. A device as recited in claim 1 wherein said support mechanism is a mechanical linkage coupled between said user manipulatable physical object and a fixed surface.

3. A device as recited in claim 2 wherein said sensor apparatus is coupled to said mechanical linkage.

4. A device as recited in claim 1 wherein said sensor apparatus is coupled to said user manipulatable physical object.

5. A device as recited in claim 1 further comprising:

a switch capable of being in an on state and an off state, said switch being coupled to said device microprocessor such that said device microprocessor can read said states of said switch and generate a command signal for receipt by said host computer in response to a state of said switch.

6. A device as recited in claim 5 wherein said switch comprises a foot pedal unit.

7. A device as recited in claim 2 wherein said mechanical linkage includes a joint with two rotary degrees of freedom, wherein one of said rotary degrees of freedom is coupled to said surface.

8. A device as recited in claim 7 wherein a sensor included in said sensor apparatus is associated with each of said rotary degrees of freedom.

9. A device as recited in claim 1 wherein said user manipulatable physical object has a pencil-like stylus configuration that can be manually manipulated by a user of said device.

10. A device as recited in claim 1 wherein said force generator transmits a force via said support mechanism in response to said force signals, and wherein said force signals are correlated to information displayed on said computer display said information including a cursor interacting with a graphical surface.

11. A device as recited in claim 1 wherein said device microprocessor reads said locative signal directly from said sensor apparatus.

12. A device as recited in claim 11 wherein said device microprocessor computes the position or orientation of said user manipulatable object from said locative signal produced by said sensor apparatus, said microprocessor reporting said position or orientation to said host computer.

13. A device as recited in claim 2 wherein said device microprocessor has access to information describing physical dimensions of said support mechanism and information of device kinematics such that said locative signal produced by said sensor apparatus can be converted into coordinate data describing a position of said physical object.

14. A device as recited in claim 11 wherein said device microprocessor computes a motion of said user manipulatable object and reports said motion to said host computer.

15. A device as recited in claim 1 wherein said device microprocessor supervises communication with said host computer.

16. A device as recited in claim 1 wherein said device microprocessor implements a command loop that monitors and decodes host commands received by said microprocessor from said application program on said host computer.

17. A device as recited in claim 16 wherein said device microprocessor executes a routine stored in said non-volatile memory based on said host command.

18. A device as recited in claim 17 wherein said routine processes said locative signal into angle data, and wherein said routine sends a single set of said angle data to said host computer.

19. A device as recited in claim 17 wherein said routine processes said locative signal into angle data, and wherein said routine continually reports said angle data from said device microprocessor to said host computer until commanded by said host computer to cease said reporting.

20. A device as recited in claim 17 wherein said locative signal is included in raw data provided by said sensor apparatus, wherein said device microprocessor reports a single set of said raw data or a single set of angle data, processed from said raw data, to said host computer.

21. A device as recited in claim 17 wherein said locative signal is included in raw data provided by said sensor apparatus, wherein said device microprocessor continually reports said raw data or continually reports angle data that is processed from said raw data to said host computer.

22. A device as recited in claim 1 wherein at least one of said host commands sets parameters on said device microprocessor.

23. A device as recited in claim 11 wherein said device includes a memory device accessible to said microprocessor and wherein said microprocessor stores sensor data representative of said locative signal in said memory device.

24. A device as recited in claim 22 wherein said host command from said host computer causes an internal register on said microprocessor to be set.

25. A device as recited in claim 22 wherein said host command from said host computer causes said device microprocessor to send control signals to a communications sub-unit of the microprocessor to provide data to said host computer.

26. A device as recited in claim 20 wherein said host command is decoded by said microprocessor, said host command calling a subroutine stored on said microprocessor to change the resistance at a joint of said support mechanism.

27. A device as recited in claim 1 wherein said host computer interacts with said force generator by sending a command to said microprocessor, said microprocessor decoding said command and running a subroutine to control said force feedback.

28. A device as recited in claim 1 wherein forces commanded by said host computer to said device microprocessor and felt by said user correspond to images displayed on said computer display.

29. A device as recited in claim 28 wherein said images include a cursor interacting with another object displayed on said display screen.

30. A device as recited in claim 29 wherein said cursor interacts with a surface image displayed on said display screen.

31. A device as recited in claim 1 wherein said parameters are communication parameters for communicating with said host computer.

32. A device as recited in claim 5 wherein at least one of said host commands causes said device microprocessor to report a state of said switch to said host computer.

33. A device as recited in claim 9 wherein a button is provided on said stylus, and wherein when said button is activated, the current position of said stylus is reported to said microprocessor and reported from said microprocessor to said host computer.

34. A device as recited in claim 1 wherein said physical object is capable of moving in only two degrees of freedom with respect to said surface.

35. A device as recited in claim 34 wherein said two degrees of freedom are rotary degrees of freedom.

36. A device as recited in claim 34 wherein said two degrees of freedom are linear degrees of freedom.

37. A device as recited in claim 1 wherein said physical object is capable of moving in six rotary degrees of freedom with respect to said surface.

38. A device as recited in claim 1 wherein said interface device includes springs operative to provide resistance to movement of said physical object in addition to said force provided by said force generator.

39. A device as recited in claim 1 wherein said interface device includes counterweights to reduce an adverse influence of gravity when said user moves said physical object.

40. A device as recited in claim 1 wherein said device microprocessor stores sensor data representative of said locative signal in said memory device including multiple locative values while said device microprocessor waits to send at least a representation of said values to said host computer.

41. A device as recited in claim 40 wherein said multiple locative values includes information such that said host computer can determine which of said host commands said multiple locative values are in response to.

48. A method as recited in claim 1 wherein said mechanical linkage includes a joint with two rotary degrees of freedom, wherein one of said rotary degrees of freedom is coupled to said surface.

49. A method as recited in claim 48 wherein said object locative signal includes signals from a sensor associated with each of said rotary degrees of freedom.

42. A method for interfacing a user and a host computer including a computer display comprising:

providing a user manipulatable physical object supported by a support apparatus which allows at least a plurality of degrees of freedom in the motion of said user manipulatable physical object;

producing an object locative signal corresponding with the position of the user manipulatable physical object at points in time during its normal operation, said object locative signal providing information for use by a host computer including a computer display which can be used by an application program running on said host computer to control images on said computer display;

providing a force feedback command from said host computer implementing a display process for displaying images on said computer display, said force feedback command being correlated to at least one of said images on said computer display;

parsing said force feedback command on a processor under at least partial control of a program stored in non-volatile memory, said processor and said non-volatile memory being located at said support apparatus and being entirely separate from said host computer, to develop force signal for controlling a force generator to cause a force to be exerted on said user manipulatable physical object that is correlated to an image on said computer display, said program in said non-volatile memory being executed simultaneously with said display process on said host computer.

43. A method as recited in claim 42 wherein said support apparatus comprises a mechanical linkage.

44. A method as recited in claim 43 wherein said locative signal is generated by a sensor apparatus that is coupled to said mechanical linkage.

45. A method as recited in claim 42 wherein said locative signal is generated by a sensor apparatus that is coupled to said user manipulatable physical object.

46. A method as recited in claim 42 further comprising providing a switch capable of being in an on state and an off state, said states of said switch read by said device microprocessor;

and

providing a command signal from said device microprocessor to said host computer in response to manipulation of said switch by said user.

47. A method as recited in claim 46 wherein said switch is located on said user manipulatable physical object.

50. A method as recited in claim 42 wherein said force exerted on said user manipulatable physical object comprises a resistance to the motion of said user manipulatable physical object.

51. A method as recited in claim 42 wherein said user manipulatable physical object has a pencil-like stylus configuration that can be manually manipulated.

52. A method as recited in claim 45 wherein said force feedback command from said host computer is received by a microprocessor which causes said force to be exerted on said physical object, said microprocessor receiving said object locative signal and providing said object locative signal to said host computer.

53. A method as recited in claim 45 wherein said host computer issues a command to said microprocessor, and said microprocessor takes an action in response to said command.

54. A method as recited in claim 53 wherein said action includes reporting data back to said host.

55. A device for use in conjunction with a host computer including a computer display, said device comprising:

a user manipulatable physical object;

a support mechanism which supports said user manipulatable physical object while allowing a plurality of degrees of freedom in the motion of said user manipulatable physical object;

a sensor apparatus coupled to at least one of said user manipulatable physical object and said support mechanism and that produces a locative signal responsive to and corresponding with the position of the user manipulatable physical object at points in time during normal operation;

a device microprocessor coupled to said sensor apparatus such that said device microprocessor can read said locative signal from said sensor apparatus, said device microprocessor running a program contained, at least in part, in a non-volatile memory coupled to said device microprocessor, said device microprocessor providing information for use by a host computer running an application program which can provide images on a computer display, said application program providing force commands which can be communicated to said device microprocessor for parsing into force signals, wherein said program and said application program are running simultaneously;

a force generator controlled by said device microprocessor for providing force feedback to said user manipulatable physical object in response to said force signals; and

a memory device accessible to said device microprocessor;

wherein said device microprocessor stores an output array of multiple locative values in said memory device while said device microprocessor waits to send a representation of said output array to said host computer.

56. A device as recited in claim 55 wherein multiple locative values includes information such that said host computer can determine which of said host commands said multiple locative values is in response to.

57. A device as recited in claim 55 wherein said multiple locative values are stored in a designated area of said memory for eventual output of said at least a representation of said values to said host computer.

58. A device as recited in claim 57 wherein said multiple locative values are stored as an output array.

59. A device as recited in claim 57 wherein said multiple locative values comprise angle values, and wherein each of said angle values is associated with a separate degree of freedom.

60. A device as recited in claim 59 wherein said multiple locative angles comprise coordinate values, each of said coordinate values being associated with a separate degree of freedom.

61. A multi-degree-of-freedom force feedback cursor control interface for use with a host computer running an application program which displays images on a display apparatus, said cursor control enabling a user to control the position of a cursor displayed by said host computer and enabling said host computer to convey force feedback to a user in accordance with said position of said cursor, said force feedback cursor control:

a user manipulatable physical object;

a support mechanism which supports said user manipulatable physical object while allowing a plurality of degrees of freedom in the motion of said user manipulatable object;

a sensor coupled to said user manipulatable physical object to provide a locative signal responsive to and corresponding with a user'"'"'s manipulation of said user manipulatable object;

a force generator coupled to said user manipulatable physical object providing resistance to motion of said user manipulatable physical object in at least one of said degrees of freedom, said resistance corresponding to host commands from a host computer running an application program which displays images on an associated display apparatus;

a microprocessor separate from said host computer, coupled to said sensor, and interacting with said force generator, said microprocessor supervising communication between said cursor control and said host computer by monitoring and decoding said host commands received from said host computer and by sending data to said host computer, where at least one of said host commands sent by said host computer to said microprocessor is a force feedback command causing the setting of force resistance on said user manipulatable object, and wherein at least part of said data sent to said host computer is representative of the motion of said user manipulatable object, wherein said supervising and setting of force resistance occur simultaneously with said display of images on said display apparatus by said host computer, thereby allowing resistance to be output from said force generator in response to and in conjunction with cursor position on said display apparatus;

memory locally coupled to said microprocessor and separate from memory of said host computer for locally storing a representation of said locative signal; and

a communication bus for coupling said microprocessor to said host computer, said bus being adapted to convey said force feedback commands from said host computer to said microprocessor and adapted to convey said data from said microprocessor to said host computer;

whereby said display apparatus of said host computer can display a cursor positioned in accordance with a position of said user manipulatable object such that force feedback commands are sent from said host computer to said cursor control in accordance with displayed cursor interaction with other display images on said display apparatus.

62. A force feedback cursor control interface device as recited in claim 61 wherein said local memory includes non-volatile memory.

63. A force feedback cursor control interface as recited in claim 62 wherein program instruction are stored in said non-volatile memory, said program instruction enabling communication between said microprocessor and said host computer and decoding at least one of said host commands.

64. A force feedback cursor control interface as recited in claim 61 wherein a plurality of command routines are stored in said local memory, at least one of said command routines allowing said microprocessor to control said force generator in accordance with at least one of said decoded host commands, and at least one of said command routines reporting data to said host computer in accordance with at least one of said decoded host commands.

65. A force feedback cursor control interface as recited in claim 61 wherein said force generator provides force upon said user manipulatable physical object by applying resistance to a rotary joint within said support mechanism.

66. A force feedback cursor control interface as recited in claim 61 wherein said communication bus is a serial interface.

67. A force feedback cursor control interface as recited in claim 61 further comprising a peripheral switch coupled to said user manipulatable physical object for providing input signals to said microprocessor when a user manipulates said peripheral switch, wherein said microprocessor reports it state of said switch to said host computer, said state causing said host computer to modify a displayed process.

68. A force feedback cursor control interface as recited in claim 61 wherein said user manipulatable physical object includes a stylus that can be moved by said user at least five degrees of freedom.

69. A force feedback cursor control interface as recited in claim 61 wherein said microprocessor is provided on board an interface device coupled to said user manipulatable physical object, said interface device being physically separate from said host computer and coupled to said host computer by said communication bus.

70. A force feedback cursor control interface as recited in claim 61 wherein said host command causes said microprocessor execute a routine to set communication parameters for communication between said microprocessor and said host computer.

71. A force feedback cursor control interface as recited in claim 61 wherein said host command causes said microprocessor to execute a routine to read said locative signal and provide said locative signal to said host computer.

72. A force feedback cursor control interface as recited in claim 61 wherein said user manipulatable physical object is coupled to a mechanical linkage having a plurality of joints, and wherein said host command causes said microprocessor to call and execute a routine to set a force on a joint of said mechanical linkage.

73. A force feedback cursor control interface as recited in claim 61 wherein said sensor is a potentiometer.

74. A force feedback cursor control interface as recited in claim 61 wherein said sensor is optical.

75. A force feedback cursor control interface is recited in claim 62 wherein said non-volatile memory contains a look-up table for processing said locative signal.

76. A multi-degree-of-freedom cursor control interface for use in conjunction with a host computer running an application program which displays images on a display apparatus, said cursor control interface enabling a user to control the position of a graphical image displayed by said host computer on said display apparatus and enabling said host computer to convey force feedback to a user in accordance with graphical image manipulation, said cursor control interface comprising:

a user manipulatable physical object grasped by a user;

a support mechanism coupled to said user manipulatable physical object, said support mechanism allowing a plurality of degrees of freedom in the motion of said user manipulatable physical object;

a force generator coupled to said user manipulatable physical object and providing force resistance to motion of said user manipulatable physical object in a plurality of said degrees of freedom in response to commands from said host computer and in coordination with said displayed graphical image;

a device microprocessor separate from said host computer and interacting with said force generator, said device microprocessor supervising communication between said cursor control interface and said host computer by monitoring and decoding said commands received from said host computer and by sending data to said host computer, wherein at least one of said commands sent from said host computer to said device microprocessor is a force feedback command causing the setting of force resistance on said user manipulatable object, and wherein at least part of said data sent to said host computer is representative of motion of said user manipulatable physical object, said decoding, sending, and setting occurring simultaneously with host computer control of said, graphical image, thereby allowing resistance to be output from said force generator in response to and in conjunction with graphical image manipulation on said display apparatus;

a communication bus connecting said device microprocessor and said host computer, said bus used in conveying said force feedback commands from said host computer to said device microprocessor;

a sensor apparatus coupled to said device microprocessor, said sensor apparatus providing said device microprocessor with a locative signal responsive to and corresponding with manipulation of said user manipulatable object; and

local memory coupled to said device microprocessor and separate from host computer memory, said local memory locally storing a representation of said locative signal;

whereby said display apparatus coupled to said host computer displays a graphical image controlled by said host computer in accordance with said motion of said user manipulatable object, and wherein force feedback commands are sent from said host computer to said cursor control interface in accordance with said displayed graphical image.

77. A cursor control interface as recited in claim 76 further comprising:

a switch capable of being in an on state and an off state, said switch being coupled to said device microprocessor such that said device microprocessor can generate a command signal for receipt by said host computer in response to a state of said switch.

78. A cursor control interface as recited in claim 76 wherein device microprocessor reads said locative signal from said sensor apparatus according to a routine stored in said local memory.

79. A cursor control interface as recited in claim 78 wherein said device microprocessor has access to information describing physical dimensions of said support mechanism and information of device kinematics such that said locative signal produced by said sensor apparatus can be converted into coordinate data describing a position of said user manipulatable physical object.

80. A cursor control interface as recited in claim 76 wherein said host computer sends a command to said device microprocessor that sets communication parameters on said microprocessor for communicating with said host computer.

81. A cursor control interface as recited in claim 76 wherein said device microprocessor stores sensor data representative of said locative signal in said local memory including multiple locative values while said device microprocessor waits to send at least a representation of said data to said host computer, wherein said multiple locative values include information such that said host computer can determine which of said host commands said multiple locative values are in response to.

82. A cursor control interface as recited in claim 80 wherein when said host command is decoded by said microprocessor, said host command calls a routine stored in said local memory to change the resistance at a joint of said support mechanism.

83. A cursor control interface as recited in claim 76 wherein said graphical image on said display apparatus includes a cursor.

84. A cursor control interface as recited in claim 83 wherein forces commanded by said host computer to said device microprocessor and felt by said user correspond to said cursor interacting with another object displayed on said display screen.

85. A cursor control interface as recited in claim 84 wherein said cursor interacts with a surface image displayed on said display screen.