Electromechanical human-computer interface with force feedback

• US 5,576,727 A
• Filed: 06/05/1995
• Issued: 11/19/1996
• Est. Priority Date: 07/16/1993
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Human Interface Corporation
• Inventors: Rosenberg, Louis B., Jackson, Bernard G.
• Primary Examiner(s): BRIER, JEFFERY A
• Application Number:
  US08/461,170
• Time in Patent Office: 533 Days
• Field of Search: 345/156-158, 345/161, 345/184, 901/3, 901/4, 901/16, 33/1 M, 33/1 CC, 178/18, 400/475
• US Class Current: 345/179
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G01B 5/008 using coordinate measuring machines
• CPC Class Codes: G01B 7/004 for measuring coordinates of points
• CPC Class Codes: G05B 19/4207 in which a model is traced or scanned and corresponding data recorded
• CPC Class Codes: G05G 5/03 Means for enhancing the operator's awareness of arrival of the controlling member at a command or datum position; Providing feel, e.g. means for creating a counterforce arrangements for indicating the position of the controlling member G05G1/015
• CPC Class Codes: G05G 9/04737 with six degrees of freedom
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: H01H 2003/008 with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction

Abstract

A method and apparatus for use with a computer for providing commands to a computer through tracked manual gestures and for providing feedback to the user through forces applied to the interface. A user manipulatable object is coupled to a mechanical linkage which is, in turn, supportable on a fixed surface. The mechanical linkage or the user manipulatable object is tracked by sensors for sensing the location and/or orientation of the object. A multi-processor system architecture is disclosed wherein a host computer system is interfaced with a dedicated microprocessor which is responsive to the output of the sensors and provides the host computer with information derived from the sensors. The host computer has an application program which responds to the information provided via the microprocessor and which can provide force-feedback commands back to the microprocessor. The force feedback is felt by a user via the user manipulatable object.

Claims

1. An interactive device for use in conjunction with a host computer, images displayed on a computer display screen, and a fixed surface, comprising:

a stylus having a pencil-like configuration to allow writing-like manipulations between fingers of a user;

a mechanical linkage coupled to a fixed surface and coupled to said stylus for supporting said stylus while allowing at least five degrees of freedom in the motion of said stylus, said mechanical linkage providing a user the ability to manipulate both the orientation and location of said stylus in three-dimensional space; and

a sensor for producing an interactive stylus locative signal which is responsive to and corresponding with the position and movement of the stylus at any point in time during its normal operation, said stylus locative signal providing information about the orientation, location, and movement of said stylus for use by said host computer and said computer display screen to manipulate images displayed by said computer display screen in accordance with said orientation, location, or movement of said stylus, said images including a cursor whose position on said computer display screen is controlled by said stylus locative signal; and

a force generator for generating a force on said stylus in at least one of said five degrees of freedom in response to force signals provided to said interactive device, said force signals correlated to information displayed on said computer display screen.

2. A device as recited in claim 1 further comprising:

a remote unit having a switch capable of being in an on state and an off state; and

command means triggered by said switch when said switch is in its on state for generating a command signal for receipt by said host computer, wherein an action taken by said computer upon receipt of said command is dependent on said state of said switch.

3. A device as recited in claim 5 wherein said remote unit is a foot pedal unit.

4. A device as recited in claim 1 wherein said mechanical linkage includes at least five joints, wherein a configuration of said joints allows said stylus to spin freely about an axis extending through the length of said stylus while all of said other joints remain fixed in position, and a sensor for sensing said spin and providing a signal describing said spin to said host computer.

5. A device as recited in claim 4 wherein three joints of said mechanical linkage closest to said stylus control said orientation of said stylus, said orientation being variable by a user while a position of a point on said stylus remains fixed.

6. A device as recited in claim 1 further comprising a button provided on said stylus, said button generating a command signal for receipt by said host computer when said button is pressed by said user.

10. A device as recited in claim 1 wherein said mechanical linkage provides said stylus with six degrees of freedom.

13. A device as recited in claim 1 wherein said feedback means generates a force on said stylus by generating a force on a joint included in said mechanical linkage in response to said force signals.

7. A method for interactively interfacing a user and a computer display apparatus, comprising the steps of:

providing a stylus having a pencil-like configuration that allows writing-like manipulations between fingers of said user;

coupling to said stylus a mechanical linkage coupled to a fixed surface for supporting said stylus while allowing at least five degrees of freedom in the motion of said stylus, said mechanical linkage for providing a user the ability to manipulate the orientation and location of said stylus in three-dimensional space;

producing an interactive stylus locative signal which is responsive to and corresponding with the position and movement of the stylus at any point in time during its normal operation, said stylus locative signal providing information about the orientation and location of said stylus;

displaying a cursor on said computer display apparatus, said computer display apparatus using said stylus locative signal to position and move said cursor in accordance with the location, orientation, or movement of said stylus;

providing feedback means for generating force on said stylus in at least one of said degrees of freedom in response to force signals provided by said host computer to said mechanical linkage, said force signals correlated to information displayed on said computer display apparatus; and

providing a remote unit switch capable of being in at least two states and a command device for generating a command signal representing said state of said switch and for receipt by said host computer, wherein an action taken by said computer when receiving said command signal depends on said state of said switch.

8. A method as recited in claim 7 wherein said remote unit switch is a foot pedal unit.

9. A method as recited in claim 7 wherein said mechanical linkage includes counterweights for reducing an adverse influence of gravity on said motion of said stylus when said user moves said stylus.

11. A method as recited in claim 7 wherein said mechanical linkage provides said stylus with six degrees of freedom.

12. A method as recited in claim 7 wherein said said stylus locative signal is produced by appropriately placed sensors.

14. An interactive device for use in conjunction with a host computer, a computer display apparatus and a fixed surface, comprising:

a stylus that can be grasped and manipulated by a user;

a mechanical linkage coupled to a fixed surface and coupled to said stylus for supporting said stylus while allowing a plurality of degrees of freedom in the motion of said stylus;

a sensor coupled to said mechanical linkage for sensing a location of said stylus and providing a stylus locative signal to a computer display apparatus, said stylus locative signal providing information about said orientation and location of said stylus for use by said computer display apparatus to manipulate an image displayed by said computer display apparatus in accordance with said location of said stylus, said image including a computer cursor having a position controlled by said location of said stylus;

a feedback device for generating a force in at least one of said plurality of degrees of freedom of said stylus in response to a stylus force signal provided by said host computer to said interactive device, said force signal being output to said feedback device when said computer cursor interacts with other images displayed on said computer display apparatus; and

a user actuated switch capable of being in a least two states and a command device for generating a command signal for receipt by said host computer, said command signal representing a state of said switch and being received by said host computer, wherein an action taken by said computer when receiving said command signal depends on said state of said switch.

15. An interactive device as recited in claim 14 wherein when said cursor displayed on said computer display apparatus moves into a different image displayed on said computer display apparatus, a force signal is output and a force is generated in at least one of said plurality of degrees of freedom.

16. An interactive device as recited in claim 14 wherein said mechanical arm linkage allows six degrees of freedom in the motion of said stylus.

17. A system for controlling an electromechanical interface apparatus manipulated by a user, the system comprising:

a host computer system for receiving a microprocessor input control signal and for providing high-level host commands, wherein said host computer system modifies a displayed process in response to said microprocessor input control signal and in proportion to a position or orientation of a physical object manipulated by a user;

a microprocessor separate from said host computer system for receiving said high-level host commands from said host computer system and providing a microprocessor output control signal;

an actuator coupled to said physical object and controlled by said microprocessor output control signal and providing a force along a degree of freedom to said user manipulable physical object in response to said microprocessor output control signal;

a sensor for tracking motion of said manipulable physical object along said degree of freedom and for outputting a locative signal which is responsive to and represents the position or orientation of said physical object, wherein said microprocessor is responsive to said locative signal, derives said microprocessor input control signal at least in part from said locative signal, and sends said microprocessor input control signal to said host computer system;

local memory separate from memory of said host computer system, said local memory comprising non-volatile memory;

program instructions stored in said non-volatile memory for enabling communication between said microprocessor and said host computer system and for decoding at least one of said high level host commands; and

a plurality of command routines stored in said local memory at least one of said command routines allowing said microprocessor to control said actuator in accordance with at least one of said decoded high-level commands, and at least one of said command routines reporting said input control signal to said host computer in accordance with at least one of said decoded high-level commands.

18. A system as recited in claim 17 wherein said microprocessor selects one of said command subroutines as instructed by said host commands and controls said actuator by following instructions of said selected command subroutine.

19. A system as recited in claim 17 wherein said physical object can be moved by said user in a plurality of degrees of freedom, and wherein said system further comprises, for each of said plurality of degrees of freedom, an actuator for providing a force along a degree of freedom of said object, and a sensor for detecting motion of said object in said degree of freedom.

20. A system as recited in claim 17 wherein said force provided by said actuator is a resistive force to motion of said physical object in said degree of freedom.

21. A system as recited in claim 17 further comprising a serial interface coupled between said host computer and said microprocessor for outputting said host command from said host computer system to said microprocessor and for receiving said microprocessor input control signal at said host computer system from said microprocessor.

22. A system as recited in claim 17 wherein said host computer system displays images on a visual output device and manipulates said images in accordance with said position of said physical object.

23. A system as recited in claim 17 further comprising a peripheral switch coupled to said microprocessor for providing input signals to said microprocessor when a user manipulates said peripheral switch, wherein said microprocessor reports a state of said switch to said host computer system, said state causing said host computer system to modify said displayed process.

36. A system as recited in claim 17 wherein said command routines are stored in said non-volatile memory.

37. A system as recited in claim 23 wherein said reporting of said state of said switch to said host computer system is controlled by at least one of said command subroutines.

38. A system as recited in claim 37 wherein a peripheral switch coupled to said microprocessor for providing input signals to said microprocessor when a user manipulates said peripheral switch, wherein said microprocessor reports a state of said switch to said host computer system, said state causing said host computer system to modify said computer-implemented process, and wherein said reporting of said state by said microprocessor is controlled by one or more of said command routines.

24. A method for interfacing motion of an object with a host computer system, the method comprising the steps of:

providing a physical object having a degree of freedom;

sensing positions of said physical object along said degree of freedom with a sensor and producing electrical sensor signals therefrom;

utilizing a microprocessor separate from said host computer system to receive said electrical sensor signals, provide said electrical sensor signals to said host computer system, and to receive host commands from said host computer system;

creating a force on said object along said degree of freedom by using said microprocessor and said host commands to control an actuator coupled to said physical object;

providing a non-volatile memory device coupled to and provided local to said microprocessor and being accessible by said microprocessor; and

providing program instructions stored in said non-volatile memory for enabling communication between said microprocessor and said host computer system and for allowing said microprocessor to control said actuators in accordance with force commands provided by said host computer system.

25. A method as recited in claim 24 wherein said microprocessor and said host computer system are coupled together by a serial communication interface.

26. A method as recited in claim 24 wherein said microprocessor provides processor commands to said actuator in accordance with a processor subroutine selected in accordance with said host commands and stored on a memory device coupled to said microprocessor.

27. A method as recited in claim 24 wherein said host computer system controls and displays visual images on a visual output apparatus in accordance with said positions of said physical object.

28. A method as recited in claim 27 wherein said physical object includes a stylus that can be moved by said user in at least five degrees of freedom.

29. A method as recited in claim 24 further comprising sending a peripheral command to said microprocessor from a peripheral input device, wherein said microprocessor sends said peripheral command to said host computer system.

39. An interface device as recited in claim 26 wherein said microprocessor is provided on board an interface device coupled to said physical object, said interface device being physically separate from said host computer system and coupled to said host computer system by a bus.

40. An interface device as recited in claim 24 wherein said program instructions include a processor routine to instruct said microprocessor to monitor and decode said host commands from said host computer system and wherein said subroutine is one of a plurality of available routines which said microprocessor calls and executes in accordance with said host command.

30. An interface device manipulated by a user and communicating with a host computer system displaying visual images on a screen, said host computer system updating said visual images in response to input signals, said interface device comprising:

a microprocessor, separate from said host computer system, for communicating with said host computer system via a communication interface by receiving a host command from said host computer system, said microprocessor being controlled by software instructions stored on a memory device coupled to said microprocessor, said software instructions enabling said communication between said host computer system and said microprocessor;

a user object movable in a degree of freedom by a user and being physically contacted by said user;

an actuator electrically coupled to said microprocessor for applying a force along a degree of freedom to said user object in accordance with a processor command from said microprocessor, said processor command being derived from said host command, wherein said software instructions on said memory device includes a routine that allows said microprocessor to control said actuator in accordance with said host command; and

a sensor for detecting a position of said user object along said degree of freedom and outputting sensor information that is included in said input signals received by said host computer system, said sensor information including information representative of said position of said user object.

31. An interface device as recited in claim 30 wherein said sensor is electrically coupled to said microprocessor, wherein said sensor outputs said sensor information to said microprocessor, and wherein said microprocessor sends said input signals that include said sensor information to said host computer system.

32. An interface device as recited in claim 31 wherein said microprocessor is operative to receive said sensor information from said sensor in accordance with a processor routine selected in accordance with said host command and stored in said memory device.

33. An interface device as recited in claim 31 wherein said user object is movable in at least two degrees of freedom.

34. An interface device as recited in claim 31 wherein said communication interface includes a serial interface.

35. An interface device as recited in claim 31 wherein said actuator applies a resistive force along said degree of freedom to said user object.

41. An interface device as recited in claim 30 wherein said microprocessor is provided on board said interface device.

42. An interface device as recited in claim 41 wherein said microprocessor monitors and decodes said host commands from said host computer system and wherein said routine in said memory device is one of a plurality of available routines which said microprocessor calls and executes in accordance with said host command.

43. An interface device as recited in claim 42 further comprising a peripheral switch electrically coupled to said microprocessor and capable of being in one of two states, and wherein said host command to said microprocessor causes said microprocessor to execute code in said memory that allows said microprocessor to report said state of said peripheral switch to said host computer system.

44. An interface device as recited in claim 42 wherein said host command causes said microprocessor to call and execute a routine to set communication parameters for communication between said microprocessor and said host computer.

45. An interface device as recited in claim 42 wherein said host command causes said microprocessor to call and execute a routine to read said sensor information and provide said input signals to said host computer.

46. An interface device as recited in claim 42 wherein said user object is coupled to a mechanical linkage having a plurality of joints, and wherein said host command causes said microprocessor to call and execute a routine to set a force on a joint of said mechanical linkage.

47. An electromechanical interface apparatus manipulated by a user for interfacing with a host computer system, said host computer system receiving a microprocessor input control signal and providing high-level host commands, wherein said host computer system modifies a displayed process in response to said microprocessor input control signal and based on a position or orientation of a physical object manipulated by said user, the interface apparatus comprising:

a microprocessor separate from said host computer system for receiving said high-level host commands from said host computer system and providing a microprocessor output control signal;

an actuator coupled to said physical object and controlled by said microprocessor output control signal and providing a force along a degree of freedom to said user manipulable physical object in response to said processor output control signal;

a sensor for tracking motion of said manipulable physical object along said degree of freedom and for outputting a locative signal which is responsive to and represents the position or orientation of said physical object, wherein said microprocessor is responsive to said locative signal, derives said microprocessor input control signal at least in part from said locative signal, and sends said microprocessor input control signal to said host computer system;

local memory separate from memory of said host computer system, said memory comprising non-volatile memory;

program instructions stored in said non-volatile memory for enabling communication between said microprocessor and said host computer system and for decoding said high level host commands; and

a plurality of command routines stored in said local memory, at least one of said command routines allowing said microprocessor to control said actuator in accordance with at least one of said decoded host commands, and at least one of said command routines reporting said input control signal to said host computer in accordance with at least one of said decoded host commands.

48. An electromechanical interface apparatus as recited in claim 47 wherein said microprocessor input control signal received by said host computer system includes information indicative of which host command said microprocessor is responding to.

49. An electromechanical interface apparatus as recited in claim 47 wherein said host command causes said microprocessor to repeatedly send said microprocessor input control signal to said host computer system until a different host command is received by said microprocessor to cease sending said microprocessor input control signal.

50. An electromechanical interface apparatus as recited 47 claim, wherein said microprocessor can compute a position and orientation of said physical object using kinematic equations and said locative signal.