Method and apparatus for controlling human-computer interface systems providing force feedback

• US 5,739,811 A
• Filed: 09/27/1995
• Issued: 04/14/1998
• Est. Priority Date: 07/16/1993
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Human Interface Corporation
• Inventors: Rosenberg, Louis B., Jackson, Bernard G.
• Primary Examiner(s): BRIER, JEFFERY A
• Application Number:
  US08/534,791
• Time in Patent Office: 930 Days
• Field of Search: 345/156, 345/157, 345/161, 345/163, 318/568.11, 318/568.25, 74/471, 463/37, 463/38, 463/30, 434/45, 395/99, 244/223
• US Class Current: 345/161
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G01B 5/008 using coordinate measuring machines
• CPC Class Codes: G01B 7/004 for measuring coordinates of points
• CPC Class Codes: G05B 19/4207 in which a model is traced or scanned and corresponding data recorded
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: H01H 2003/008 with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction

Abstract

A method and apparatus for controlling and providing force feedback using an interface device manipulated by a user. A microprocessor is provided local to the interface device and reads sensor data from sensors that describes the position and/or other information about an object grasped and moved by the user, such as a joystick. The microprocessor provides the sensor data to a host computer that is coupled to the interface device by a communication bus that preferably includes a serial interface. In a "host-controlled" embodiment, the host computer calculates force values using the sensor data and other parameters of a host application program and sends the force values to the local microprocessor, which directly provides the force values to actuators to apply forces to the user object. In a "reflex" embodiment, the host computer sends high level supervisory commands to the local microprocessor, and the microprocessor independently implements a local process based on the high level command for reading sensor data and providing force values to the actuators using sensor data and other parameters.

Claims

1. An interface device for use with a host computer displaying a graphical simulation on a display device, said host computer displaying, executing, and updating a graphical simulation in response to user manipulation of said interface device and commanding force feedback sensations in response to said user manipulation and in coordination with events within said graphical simulation, the interface device comprising:

a user manipulable object coupled to a support system and grasped by a user;

a force generator coupled to said user object for receiving a force control signal and providing a resistance force to motion along at least one degree of freedom of said user object and in accordance with said force control signal;

a sensor that detects motion of said user manipulable object along said degree of freedom and outputs a locative signal including information representative of the position of said user object;

a user-adjustable switch apparatus providing a state signal representing a state of said switch apparatus; and

a microprocessor local to said interface apparatus, separate from said host computer system, coupled to said host computer system, to said sensor, and to said switch apparatus, said microprocessor receiving

host commands from said host computer system,

said state signal from said switch apparatus, and

said locative signal from said sensor,

said microprocessor executing a local process in parallel with said host execution of said graphical simulation and providing said force control signal to said force generator, said local process involving an execution of at least one local routine stored in a local memory coupled to said microprocessor, wherein said microprocessor

enables communication between said microprocessor and said host computer system,

decodes said host commands sent by said host computer system to said microprocessor,

controls said force generator in response to a decoded command received from said host computer system, wherein the control of the force generator is coordinated with an event occurring in said parallel host computer system execution of said graphical simulation, thereby locally creating a meaningful feel sensation that corresponds with a relevant displayed event within said graphical simulation,

reports a representation of said locative signal to said host computer system, wherein said host computer system updates said parallel execution of said graphical simulation in response to said representation of said locative signal, and

reports a representation of said state signal to said host computer system, wherein said host computer system updates said parallel execution of graphical simulation in response to said state of said switch apparatus.

2. An interface device as recited in claim 1 wherein said host command output by said host computer system is a force command that is relayed directly to said force generator by said microprocessor.

3. An interface device as recited in claim 1 wherein said microprocessor is operative in a low-level local force control loop to compute said resistance force and to provide said force control signal to said force generator in response to said motion of said object, said low level local force control loop being initiated in response to said host commands.

4. An interface device as recited in claim 3 wherein said host command is a high level command from said host computer system, and wherein said microprocessor implements one of a plurality of said local processes selected in accordance with said high level command to implement said local control loop.

5. An interface device as recited in claim 4 wherein said host computer system can override said local control loop so that said host provides said force control signal to said force generator in response to said motion of said object, wherein said host command includes a force command that is relayed directly to said force generator by said microprocessor.

6. An interface device as recited in claim 1 wherein said user object includes a joystick movable in a plurality of degrees of freedom.

7. A system as recited in claim 1 wherein said user object includes a mouse.

8. An interface device as recited in claim 1 further comprising a serial interface for outputting said host command from said host computer system and for receiving said locative signal to said host computer system.

9. An interface device as recited in claim 1 further comprising a parallel interface for outputting said host command from said host computer system and for receiving said locative signal to said host computer system.

10. An interface device as recited in claim 8 further comprising an additional interface for providing said locative signal to said host computer system through a game port of said host computer system.

11. An interface device as recited in claim 1 wherein said force generator is one of a motor and a voice coil.

12. An interface device as recited in claim 1 wherein said force generator is a brake, and wherein said force is a resistive force for resisting a user force applied to said user object by said user along said degree of freedom.

13. An interface device as recited in claim 1 wherein said graphical simulation updated by said host computer system includes simulation software or game software, wherein said host computer system displays images on a visual output device and manipulates said images in accordance with said position and motion of said object.

14. A interface device as recited in claim 1 further comprising a local clock coupled to said microprocessor, wherein said microprocessor accesses said local clock to determine, in part, said force provided by said force generator.

15. A interface device as recited in claim 1 further comprising a housing, wherein said force generator, said sensor, and said microprocessor are included within said housing.

16. An interface device as recited in claim 1 wherein said local routine that enables communication sets communication parameters on said microprocessor for enabling said communication between said microprocessor and said host computer.

17. An interface device as recited in claim 3 wherein said local control loop uses an algorithm to compute said resistance force based on said motion of said object.

18. An interface device as recited in claim 3 wherein said local control loop uses a digitized force profile having previously stored values, said values being used to determine a sequence of force values, each of said force values in said sequence being output to said force generator for a predetermined time duration.

19. An interface device as recited in claim 8 wherein said time interval can be varied so as to achieve a desired force sensation on said user.

20. An interface device as recited in claim 18 further comprising a local clock coupled to said microprocessor, wherein said microprocessor implements said predetermined time interval using said local clock.

21. An interface device as recited in claim 3 wherein said host computer determines an appropriate high level command to send to said microprocessor depending on an event in said application program, wherein said high level command designates a particular force routine for said microprocessor to call and execute and includes additional parameter information to command a desired force on said user object.

22. An interface device as recited in claim 21 further comprising multiple force generators for outputting said resistance force, wherein said said parameter information includes information designating which of said multiple force generators are to output forces.

23. An interface device as recited in claim 3 wherein upon activation of said interface device, said interface device contacts said host computer using said interface device and informs said host computer that said interface device is active and ready to receive said host commands.

24. An interface device as recited in claim 1 wherein said microprocessor uses a force routine to modulate force with respect to timing information from a local clock, separate from a host clock and coupled to said microprocessor, wherein said timing information is used for at least one of coordinating forces with said events of said graphical simulation, outputting forces at a predetermined rate and duration, and determining magnitudes of output forces.

25. An interface device as recited in claim 1 wherein said graphical simulation includes a displayed cursor, and wherein said host computer updates the location of said cursor on a display apparatus in accordance with the motion of said user manipulatable object, and wherein said host computer further sends force host commands to said microprocessor in accordance with interaction of said cursor with other simulated objects in said graphical simulation.

26. An interface device as recited in claim 1 further including a safety device mounted to said user manipulatable object for disabling power to said motors when said user is not properly engaging said user manipulatable object by blocking an optical sensor on said user manipulatable object.

27. An interface device as recited in claim 1 further including a safety device mounted to said user manipulatable object for disabling power to said motors when said user is not touching an electrostatic contact element on said user manipulatable object.

28. An interface device as recited in claim 1 further comprising memory local to said microprocessor and separate from said host computer for storing at least one of said local routines.

29. An interface device as recited in claim 28 further including factory calibration parameters stored in said memory, said factory calibration parameters including information unique to the given manufacture unit of said interface device.

30. An interface device as recited in claim 29 wherein said factory calibration parameters are used by said local process to compensate for manufacturing variations in physical properties of components within a given manufactured unit of said interface device and are used to ensure that force output characteristics of a given manufactured interface device is approximately the same as the force output characteristics of a large number of other manufactured interface devices of the same design.

31. An interface device as recited in claim 1 wherein said coordination between said event in said graphical simulation and generation of said force on said user manipulatable object has less than a 30 millisecond delay between onsets of said displayed host event and of said force.

32. An interface device as recited in claim 1 wherein said event in said host execution of said graphical simulation is a collision between simulated objects.

33. An interface device as recited in claim 32 wherein said collision event on said host computer is coordinated with a force routine for simulating a feel of an obstruction.

34. An interface device as recited in claim 32 wherein said collision event on said host computer is coordinated with a force routine for simulating a feel of damping.

35. An interface device as recited in claim 1 wherein said microprocessor commands the generation of a simulated spatial texture as a force on said user object, said texture being coordinated with an event in said graphical simulation wherein said texture is associated with a displayed surface within said graphical simulation.

36. An interface device as recited in claim 1 wherein said host computer further executes an audio event in said graphical simulation, said audio event coordinated with a force routine such that less than a 30 millisecond delay occurs between onsets of said host audio event and onset of said force.

37. An interface device as recited in claim 1 wherein host command includes parameter information used by said local force routine to generate appropriate feel sensation.

38. An interface device as recited in claim 1 wherein said force routine includes a local force algorithm that modulates force with respect to said locative signal.

39. An interface device as recited in claim 38 wherein said force algorithm generates a spring force proportional to displacement of said user manipulatable object.

40. An interface device as recited in claim 8 wherein said serial interface is a Universal Serial Bus.

41. A method for controlling a force feedback interface device using a host computer, said interface device manipulated by a user, said host computer displaying a graphical simulation including images and updating said graphical simulation in response to said manipulation of said interface device, said interface device allowing said host computer to command and update force feedback sensations on said user in response to said manipulations and in coordination with events in said graphical simulation, the method comprising:

inputting a position signal to a host computer system from a sensor, said position signal including information representative of the position and motion of a user manipulatable object of said interface device in a plurality of degrees of freedom with respect to a fixed surface, said user maniupulatable object being grasped by said user, wherein said host computer updates said graphical simualtion in reponse to said position signal;

outputting a host force command from a host computer system to a microprocessor local to said force feedback interface device and separate from said host computer system, said microprocessor executing a local process in parallel with said graphical simulation executed by said host computer system, said local process including a plurality of routines to control reception of said host force commands and outputting force signals in response to said host force commands;

outputting said force signal from said microprocessor to an actuator, said force signal derived from said host force command; and

providing a resistance force from said actuator on said user manipulatable object grasped by said user, wherein a direction and a magnitude of said force is generated in response to and in accordance with said force signal.

42. A method as recited in claim 41 wherein said position signal is input to said microprocessor and is input to said host computer system from said microprocessor, wherein said microprocessor executes at least one of said routines within said local process to receive said position signal and to report said position signal to said host computer system.

43. A method as recited in claim 42 wherein said host force command is substantially equivalent to said force signal.

44. A method as recited in claim 42 wherein said host force command is a high level command instructing said microprocessor to select one of a plurality of force subroutines available as said routines to said microprocessor, wherein each of said force subroutines instructs said microprocessor to output force commands to said actuator and input position signals from said sensor independently of said host computer system in a different way.

45. A method as recited in claim 42 wherein said host force command can be either a low level force command substantially equivalent to said force signal, or a high level force command instructing said microprocessor to select one of a plurality of force subroutines to output force commands to said actuator and input position signals from said sensor independently of said host computer system.

46. A method as recited in claim 41 wherein said object includes a joystick that can be moved by said user in two degrees of freedom.

47. A method as recited in claim 42 wherein said displaying images by said host computer on a visual output device includes manipulating said images in accordance with said position signal.

48. A method as recited in claim 42 wherein said host computer receives said position signal and outputs said host force command signal using a serial interface.

49. A method as recited in claim 48 wherein said serial interface is a Universal Serial Bus (USB) interface.

50. A method as recited in claim 48 further comprising a step of determining a magnitude of said force to be provided by said actuator.

51. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from a position of said user manipulatable object along said degree of freedom.

52. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from a position and a velocity of said user manipulatable object moving along said degree of freedom.

53. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from an acceleration of said user manipulatable object along said degree of freedom.

54. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from a clock providing timing information for said force commands.

55. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from input data provided by a button input device included in said force feedback interface device.

56. A method as recited in claim 50 wherein said magnitude of said force is determined, in part, from a position signal input from a second force feedback interface device coupled to said host computer system and interacting with a host application program implemented by said host computer system.

57. A method as recited in claim 48 further comprising a step of using calibration parameters stored in a memory device for adjusting said force consistently with output forces of a plurality of other force feedback interface devices having variations in physical properties resulting from a manufacturing process.

58. A force feedback interface device manipulated by a user and communicating with a host computer system implementing a host application program, said host application program displaying images on a computer display apparatus, said host computer system updating said host application program in response to user manipulation of said interface device and commanding force feedback sensations on said user in response to said manipulations and in coordination with events in said host application program, said force feedback interface device comprising:

a user object movable in a degree of freedom by a user and being physically contacted by said user;

a gimbal mechanism rotationally coupled to a support, said gimbal mechanism providing a first rotary degree of freedom to said user manipulatable object with respect to said support and providing a second rotary degree of freedom to said user manipulable object with respect to said support, wherein an axis of said first degree of freedom and an axis of said second degree of freedom are substantially mutually perpendicular and are intersecting at an origin point;

a device microprocessor, separate from said host computer system, for communicating with said host computer system via a communication bus by receiving a host force command from said host computer system, said host force command being output from said host computer system in response to and coordinated with events within said host application program, said microprocessor executing a local process in parallel with said execution of said host application program for receiving and decoding said host force command and outputting a force signal in response to a decoded host command, wherein said local process is stored in a local memory separate from said host computer and accessible to said device microprocessor;

an actuator coupled between said gimbal mechanism and said support and controlled by said microprocessor for applying a rotational force along a degree of freedom to said user object in accordance with a force signal from said microprocessor, said force signal being derived from said host force command and causing said force to be coordinated with said host application program, thereby causing a feel sensation that corresponds with a relevant event within said host application program; and

a sensor for detecting a position of said user object along at least one of said degrees of freedom and outputting locative signals to said host computer system, said locative signals including information representative of said position of said user object, wherein said host application program updates said images in response to said locative signals and said position of said user object and said user interacts with said host application program by moving said user object.

59. A force feedback interface device as recited in claim 58 wherein said sensor is electrically coupled to said microprocessor, and wherein said sensor outputs said locative signals to said microprocessor, and wherein said local process of said microprocessor reports a representation of said locative signals to said host computer system.

60. A force feedback interface device as recited in claim 59 wherein said host force command is a high level command instructing said microprocessor to select one of a plurality of force subroutines available to said microprocessor, wherein each of said force subroutines instructs said microprocessor to output force commands to said actuator and input position signals from said sensor independently of said host computer system.

61. A force feedback interface device as recited in claim 60 wherein said host force command can be either a low level force command substantially equivalent to said force signal, or a high level force command instructing said microprocessor to select one of a plurality of force subroutines to output force commands to said actuator and input position signals from said sensor independently of said host computer system.

62. A force feedback interface device as recited in claim 59 further comprising circuitry for receiving position information in said locative signals from said sensor and for calculating velocity or acceleration values from said position information, said velocity or acceleration values being used in part by microprocessor to determine force values output to said actuator in said force signals.

63. A force feedback interface device as recited in claim 59 wherein said user object is a joystick or a mouse movable in two degrees of freedom.

64. A force feedback interface device as recited in claim 59 wherein said actuator is a first actuator and further comprising a second actuator, wherein both of said actuators are grounded and are coupled to said gimbal and to said support.

65. A force feedback interface device as recited in claim 59 wherein said actuator is one of a servo motor and a voice coil.

66. A force feedback interface device as recited in claim 59 wherein said gimbal mechanism includes a closed loop five member linkage.

67. A force feedback interface device as recited in claim 63 wherein said gimbal mechanism is a slotted yoke mechanism coupled to said joystick.

68. An interface device as recited in claim 58 wherein said local process sets communication parameters for enabling communication between said microprocessor and said host computer.

69. An interface device as recited in claim 58 wherein said local process includes an execution of a plurality of local routines stored in a local memory coupled to said microprocessor, wherein at least one of said local routines enables communication between said microprocessor and said host computer system, at least one of said local routines decodes said host commands sent by said host computer system to said microprocessor, and at least one of said local routines is a force routine for controlling said actuator in response to a decoded command received from said host computer system, an execution of said force routine being coordinated with an event occurring in said parallel host computer system execution of said application program, thereby locally creating a meaningful feel sensation that corresponds with a relevant displayed event within said application program.

70. An interface device as recited in claim 69 wherein at least one of said local routines reports a representation of said locative signals to said host computer system, and at least one of said local routines reports a representation of a state signal to said host computer system based on a state of a user-actuable switch coupled to said microprocessor, wherein said host computer system updates said parallel execution of application program in response to said state of said switch.

71. An interface device as recited in claim 58 wherein said host application program includes a graphical simulation which includes a displayed cursor, wherein said host computer updates the location of said cursor in response to said representation of said locative signal, and wherein said host computer further sends force commands to said local microprocessor in response to interaction of said cursor with other simulated objects in said graphical simulation.

72. An interface device as recited in claim 71 wherein said event in said host execution of said graphical simulation is a collision between simulated objects, wherein said collision event is coordinated with a force routine for simulating the feel of an obstruction.

73. An interface device as recited in claim 58 wherein said host command includes parameter information used by said force routine to generate appropriate feel sensation, said parameter information including a damping constant.

74. An interface device as recited in claim 58 further including a safety device mounted to said user manipulatable object for disabling power to said motors when user is not properly engaging said user manipulatable object.

75. An interface device as recited in claim 62 wherein said circuitry includes an application specific integrated circuit (ASIC).

76. An interface device as recited in claim 62 wherein said velocity and acceleration values are provided with said input representations of said locative signals to said host computer.

77. An interface device as recited in claim 62 wherein said velocity and acceleration values are only used by said microprocessor and are not sent to said host computer.

78. A method for interfacing motion of user manipulable object with a host computer system running a graphical application program, said program being responsive to manipulations of said user object by said user and providing force feedback on said user object in response to said manipulations and in coordination with events within said program, the method comprising:

providing a user manipulable object having a degree of freedom and being grasped by a user;

sensing positions of said user manipulable object along said degree of freedom with a sensor and producing electrical sensor signals therefrom;

utilizing a device microprocessor local to said user manipulable object and separate from said host computer system to receive said sensor signals from said sensor, to communicate with said host computer system to provide said sensor signals to said host computer system so that said host computer system can update said graphical application program in response to said sensor signals, and to receive host force commands from said host computer system, wherein a host force command is output by said host computer system when an event within said graphical application program implemented by said host computer requires a force to be applied to said user object; and

creating a force on said user manipulable object along said degree of freedom by using said microprocessor and said host force commands to control an actuator coupled to said user manipulable object, said creating a force including;

selecting a force subroutine by said microprocessor in accordance with said host force commands;

outputting force signals from said microprocessor to said actuator utilizing said force subroutine and said sensor signals to cause said actuator to output said force.

79. A method as recited in claim 78 wherein said change in force is based on said electrical sensor signals and timing information provided by a clock coupled to said microprocessor.

80. A method as recited in claim 78 wherein said force includes one of the group consisting of a spring force, damping force, and inertia force.

81. A method as recited in claim 78 further comprising a step of said microprocessor retrieving a plurality of stored digital force values from a memory device coupled to said microprocessor and outputting said stored force values to said actuator.

82. A method as recited in claim 81 wherein said stored digital force values are output to said actuator based on timing information provided by a system clock.

83. A method as recited in claim 81 wherein said stored digital force values are output to said actuator based on said electrical sensor signals.

84. A method as recited in claim 78 wherein said microprocessor and said host computer system are coupled together by a serial communication interface.

85. A method as recited in claim 78 further comprising a step of said host computer system outputting audio feedback to said user when an audio event occurs in a host application program, said audio feedback being output in synchronization with said output of said force within a tolerance of about 30 milliseconds when said force is desired to correspond to said audio event.

86. A method as recited in claim 85 wherein said said onset of said audio feedback occurs within about 30 milliseconds of an onset of said corresponding force, and wherein said audio feedback has an amplitude in direct proportion to a magnitude of said force.

87. A method as recited in claim 78 wherein said host computer system outputs visual images on a display screen according to visual events in a host application program, said visual events being synchronized with said output of said force within a tolerance of about 30 milliseconds when said force is desired to correspond to said visual event.

88. An interface device for use with a host computer executing and displaying a graphical simulation, said interface allowing said host computer to update said graphical simulation in response to user manipulation of said interface device, said interface device also allowing said host computer to command force feedback sensations in response to said manipulations and in coordination with events within said graphical simulation, said interface device comprising:

a user manipulatable object grasped by a user;

a support mechanism which supports said user manipulatable object on a fixed surface while allowing a plurality of degrees of freedom in the motion of said user manipulatable object with respect to said fixed surface;

a sensor apparatus coupled to said support mechanism, said sensor apparatus providing a locative signal corresponding with the location of said user manipulatable object within said plurality of degrees of freedom;

a force generator coupled to said user manipulatable object for providing resistance to motion of said user manipulatable physical object in at least one of said degrees of freedom with respect to said fixed surface, said resistance to motion generated in response to host signals from said host computer; and

a communication interface coupling said host computer to said interface device, said communication interface including a game port and a serial port functioning in parallel wherein host signals are transmitted from host computer to said interface device across said serial port and wherein locative signals are transmitted from interface to host computer across said game port.

89. A device as recited in claim 88 further including a local microprocessor separate from said host computer, said local microprocessor executing a local process in parallel with host execution of said graphical simulation, said local process enabling communication between said interface device and said host computer across said serial interface and controlling said force generator in response to said host signals.