Force feedback system including multi-tasking graphical host environment and interface device

• US 6,300,936 B1
• Filed: 11/14/1997
• Issued: 10/09/2001
• Est. Priority Date: 11/14/1997
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Rosenberg, Louis B., Braun, Adam C., Beamer, Jonathan L., Chang, Dean C.
• Primary Examiner(s): Jankus, Almis R.
• Assistant Examiner(s): Tran, Henry N.
• Application Number:
  US08/970,953
• Time in Patent Office: 1,425 Days
• Field of Search: 345/145, 345/156-167, 345/184, 345/326, 345/427, 345/333, 345/338, 345/520, 345/522, 341/20, 341/27, 364/709.01
• US Class Current: 345/156
• CPC Class Codes: G05B 19/00 Programme-control systems
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/0333 Ergonomic shaped mouse for one hand
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/03548 Sliders, in which the moving part moves in a plane
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A force feedback system provides components for use in a force feedback system including a host computer and a force feedback interface device. An architecture for a host computer allows multi-tasking application programs to interface with the force feedback device without conflicts, where a single active application may output forces. A background application also provides force effects to be output and allows a user to assign force effects to graphical objects in a graphical user interface. Force feedback effects and structures are further described, such as events and enclosures.

Claims

1. A method for interfacing a multi-tasking graphical environment implemented on a host computer with a force feedback interface device coupled to said host computer, wherein multiple application programs are simultaneously running in said multi-tasking environment, wherein one of said multiple application programs is active and the other application programs are inactive, the method comprising:

receiving force effects from each of at least two of said multiple application programs simultaneously running in said multi-tasking graphical environment, each of said force effects describing a force sensation that is output by said force feedback interface device;

storing said force effects into contexts allocated in memory of said host computer, wherein each of said contexts is associated with only one of said at least two application programs running on said host computer, and wherein each of said force effects is stored in said context associated with said application program that sent each of said force effects;

of said force effects in said contexts of said at least two application programs, providing only said force effects in said context of said active application program to be stored in said force feedback interface device such that only said force effects received from said active application program are output to a user of said force feedback interface device;

receiving instructions from each of said at least two application programs, said instructions instructing said force feedback interface device to start an output of at least one of said force effects sent from each of said application programs; and

of said instructions from said at least two application programs, sending only said instructions from said active application program to said force feedback interface device such that only said force effects from said context of said active application program are output to said user of said force feedback interface device.

2. A method as recited in claim 1 wherein said instructions include commands, and wherein commands from inactive running application programs to start or stop an output of force effects are ignored and not sent to said force feedback interface device.

3. A method as recited in claim 2 wherein when said active application program becomes inactive and a different one of said at least two application programs becomes active, new force effects are sent to said force feedback interface device to replace said force effects sent to said force feedback interface device for said previously active application program, said new force effects being included in a context associated with said different application program.

4. A method as recited in claim 1 further comprising:

sending force effects in a context associated with a background application program to said force feedback interface device regardless of whether said background application program is active or inactive in said multi-tasking environment; and

sending instructions from said background application program to said force feedback interface device such that said force effects from said background application program are output to said user of said force feedback interface device,

such that said force feedback interface device may output forces based on said force effects from said active application program and from said background application program.

5. A method as recited in claim 1 wherein said contexts further include at least one event that can be triggered by said force feedback device, wherein said event includes conditions based on interactions of a cursor in said graphical environment such that when said conditions are met, an event notification is sent to an application program associated with said event.

6. A method as recited in claim 1 wherein said force effects include a spring, a damper, and a vibration.

7. A method as recited in claim 1 wherein said multi-tasking graphical environment includes a Windows™

operating system environment.

8. A method as recited in claim 7 wherein said force effects are created as effect objects in an application programming interface in said Windo™

operating system environment, and wherein said effect objects are sent to said force feedback interface device when created by said active application program.

9. A method as recited in claim 1 wherein said force effects sent to said force feedback interface device are stored in memory of said force feedback interface device and output as force sensations when commanded to be output by said active application program.

10. A method as recited in claim 1 wherein at least one of said force effects describes a force sensation that provides an enclosure, said enclosure including at least one wall having an inner wall portion and an outer wall portion, each of said wall portions associated with different forces felt by said user from said device.

11. A method as recited in claim 10 wherein said enclosure is an ellipse.

12. A method as recited in claim 10 wherein said enclosure is a rectangular object including four sides, each of said sides providing said inner wall and said outer wall.

13. A method as recited in claim 12 wherein a magnitude of force sensations associated with top and bottom sides of said enclosure is greater than a magnitude of force sensations associated with left and right sides of said enclosure so that a user can perceive all of said sides as having equal magnitude when using said force feedback interface device.

14. A method as recited in claim 10 wherein at least one of said force sensations associated with said enclosure is adjusted after a user-controlled cursor is moved into said enclosure, wherein a particular one of said application programs is informed of said movement into said enclosure by using an event stored in said context associated with said particular application program and sent to said force feedback interface device.

15. A method as recited in claim 10 wherein said enclosure is used to control force sensations associated with other enclosures defined within the interior space of said enclosure.

16. A computer readable medium including program instructions executable by a host computer, said program instructions interfacing a multi-tasking graphical environment implemented on said host computer with a force feedback interface device coupled to said host computer, wherein multiple application programs are simultaneously running in said multi-tasking environment, wherein one of said plurality of application programs is active and the other application programs are inactive, the program instructions performing:

receiving force effects from each of at least two of said multiple application programs simultaneously running in said multi-tasking graphical environment, each of said force effects describing a force sensation that is output by said force feedback interface device;

storing said force effects into contexts allocated in memory of said host computer, wherein each of said contexts is associated with only one of said at least two application programs running on said host computer, and wherein each of said force effects is stored in said context associated with said application program that sent each of said force effects;

of said force effects in said contexts of said at least two application programs, providing only said force effects in said context of said active application program to be stored in said force feedback interface device;

receiving instructions from each of said at least two application programs, said instructions instructing said force feedback interface device to start an output of at least one of said force effects sent from said at least two application programs; and

of said instructions from said at least two application programs, sending only said instructions from said active application program to said force feedback interface device, said instructions causing only said force effects from said context of said active application program to be output to said user.

17. A computer readable medium as recited in claim 16 wherein when said active application program becomes inactive and a different one of said application programs becomes active, new force effects are sent to said force feedback interface device to replace said force effects sent to said force feedback interface device for said previously active application program, said new force effects being included in a context associated with said different application program.

18. A computer readable medium as recited in claim 16 further comprising performing a step of sending force effects in a context associated with a background application program to said force feedback interface device regardless of whether said background application program is active or inactive in said multi-tasking environment, such that said force feedback interface device may always output forces based on said force effects from said active application program or from said background application program.

19. A computer readable medium as recited in claim 18 wherein only said background application program may send said force effects to said force feedback interface device when inactive.

20. A computer readable medium as recited in claim 16 wherein said contexts further include at least one event that can be triggered by said force feedback device, wherein said event includes conditions based on interactions of a cursor in said graphical environment such that when said conditions are met, an event notification is sent to an application program associated with said event.

21. A computer readable medium as recited in claim 16 wherein said force effects include a spring, a damper, and a vibration.

22. A computer readable medium as recited in claim 16 wherein said multi-tasking graphical environment includes an operating system environment.

23. A computer readable medium as recited in claim 22 wherein said force effects are created as effect objects in an application programming interface in said operating system environment, and wherein said effect objects are sent to said force feedback interface device when created by said active application program.

24. A computer readable medium as recited in claim 16 wherein said force effects sent to said force feedback interface device are stored in memory of said force feedback interface device and output as force sensations when commanded to be output by said active application program.

25. A computer readable medium as recited in claim 16 wherein said force effects describe an enclosure force sensation, said enclosure force sensation providing at least one wall having an inner wall and an outer wall, each of said walls associated with said enclosure force sensation.

26. A method for interfacing a multi-tasking graphical environment implemented on a host computer with a force feedback interface device coupled to said host computer, wherein a plurality of application programs having force feedback functionality are simultaneously running in said multi-tasking environment, wherein only one of said application programs is active, the method comprising:

enabling a reception of force commands from at least two of said application programs having force feedback functionality, said force commands characterizing force effects to be output by said force feedback interface device;

enabling a storage of each of said force effects in one of a plurality of contexts, each of said contexts being allocated in memory of said host computer, and each of said contexts being associated with only one of said at least two application programs running on said host computer, and wherein each of said force effects is stored in said context associated with said application program that sent said force effect;

enabling a determination of which one of said at least two application programs sending said force commands is currently active in said multi-tasking environment; and

of said force effects in said contexts of said at least two application programs, enabling a sending of only said force effects in said context of said active application program to said force feedback interface device, said sent force effects being available to be output to said user by said force feedback interface device.

27. A method as recited in claim 26 further comprising

enabling a reception of instructions from said at least two application programs that sent said force commands to start an output of at least one of said force effects sent from said application programs; and

enabling a sending of only said instructions from said active application program to said force feedback interface device, said instructions causing only said force effects from said context of said active application program to be output to said user of said force feedback interface device.

28. A method as recited in claim 27 wherein if said application program providing said instructions is inactive, said instructions to cause said force effects to be output are ignored.

29. A method as recited in claim 27 wherein said sent force effects include force effect characterization data that are stored in memory of said force feedback interface device.

30. A method as recited in claim 29 wherein said instructions include a start command to start the output of a force effect stored in a context, wherein said force effect is commanded to be output on said force feedback interface device only if said application program providing said start command is active.

31. A method as recited in claim 26 further comprising receiving a stop command from one of said at least two application programs to stop the output of a force effect, wherein said interface device is commanded to stop said output of said force effect only if said application program providing said start command is active.

32. A method as recited in claim 26 further comprising enabling a background force feedback master application to run in said graphical user interface simultaneously with said plurality of application programs, said background master application providing force feedback to graphical objects in said graphical environment.

33. A method as recited in claim 32 wherein commands and force effects from said background master application are always sent to said force feedback interface device, regardless of whether said background application is active, such that said force feedback interface device outputs forces based on said force effects from said active application program and from said background application program.

34. A method as recited in claim 26 wherein said force commands include a command to set a device state structure.

35. A method as recited in claim 34 wherein said device state structure includes a force gain level for all forces output by said force feedback interface device.

36. A method as recited in claim 26 wherein said active application program is a first application program, and further comprising receiving an indication that said first application program has become inactive and that a second one of said running application programs has become active, wherein further force commands from said first application program are not sent to said interface device and force commands from said second application program are sent to said interface device while said second application program is active.

37. A method as recited in claim 26 wherein any commands from an inactive running application program to start, stop, or pause said output of force effects by said interface device are ignored and are not sent to said interface device.

38. A method as recited in claim 26 wherein said contexts further include data characterizing at least one event that can be triggered by said force feedback device, wherein said event includes conditions based on interactions of a cursor in said graphical environment such that when said conditions are met, an event notification is sent to an application program associated with said event.

39. A method for controlling a force feedback interface device coupled to a host computer displaying a graphical environment and a cursor within said graphical environment on a display screen, the method comprising:

reading sensor data and determining a position of a user manipulandum in a device frame of said force feedback interface device from said sensor data, said manipulandum being physically contacted and moveable by a user;

determining a device delta position of said manipulandum that includes the change in position of said manipulandum from a previous position, and reporting said device delta position to said host computer when said host computer is not enabled to receive an absolute position of said cursor from said force feedback interface device; and

when said host computer is enabled to receive said absolute position of said cursor from said force feedback interface device,

determining said absolute position from said device delta position, said absolute position provided in a screen frame of said display screen,

reporting said absolute position to said host computer to allow said host computer to display said cursor in said screen frame at a position derived from said absolute position, and

using a scaled position of said manipulandum related to said absolute position in determining a force to be output by actuators of said force feedback interface device.

40. A method as recited in claim 39 wherein said scaled position of said manipulandum is determined from said delta position, and wherein said absolute position is determined based on said scaled position.

41. A method as recited in claim 40 wherein said scaled position is determined based on a ballistic delta position, said ballistic delta position being derived from said delta position using a ballistics process.

42. A method as recited in claim 41 wherein said ballistics process uses a velocity of said manipulandum in said device frame in said determination of said ballistic delta position.

43. A method as recited in claim 42 wherein, in determining said ballistic delta position, said ballistics process multiplies said device delta position by one if said velocity of said manipulandum is below a first threshold velocity, by two if said velocity of said manipulandum is between said first threshold and a second threshold velocity, and by four if said velocity of said manipulandum is above said second threshold velocity.

44. A method as recited in claim 41 further comprising receiving ballistic parameters from said host computer, said ballistic parameters characterizing said ballistics process.

45. A method as recited in claim 40 further comprising receiving a current screen resolution of said display screen of said host computer to allow said microprocessor to adjust said absolute position based on said screen resolution.

46. A method as recited in claim 40 further comprising receiving a current screen aspect ratio of said display screen of said host computer to allow said microprocessor to adjust said absolute position based on said screen aspect ratio.

47. A method as recited in claim 39 wherein said scaled position of said manipulandum is a ballistic position in a ballistic frame, said ballistic frame having a resolution that is a multiple of a resolution of said screen frame.

48. A method as recited in claim 39 wherein said scaled position is determined according to a rate control method for indexing when said manipulandum is moved within a predetermined distance of a limit to said device frame.

49. A method as recited in claim 39 further comprising outputting said force on said manipulandum.

50. A method as recited in claim 39 wherein said delta position is determined from a current manipulandum position and a previous manipulandum position, wherein said current and previous manipulandum positions are determined from measured angles of joints of said force feedback device.

51. A method as recited in claim 39 wherein said host computer is not enabled to receive said absolute position when said host computer has first detected said force feedback interface device, and wherein said host computer is enabled to receive said absolute position after force-feedback related driver software has been loaded into memory of said host computer.

52. A force feedback interface device coupled to a host computer displaying a graphical environment and a user-controlled graphical object within said graphical environment on a display device, the interface device comprising:

a user manipulandum physically contacted by a user and movable in a planar workspace in two degrees of freedom;

a sensor that provides raw data describing a device position of said manipulandum in said workspace;

a plurality of actuators coupled to said manipulandum that provide a force on said manipulandum in said planar workspace; and

a local microprocessor, separate from said host computer and coupled to said sensor and to said actuators, said local microprocessor;

determining a delta position of said manipulandum that includes the change in position of said manipulandum from a previous position, said microprocessor reporting said delta position to said host computer when said host computer is not enabled to receive an absolute position of said user-controlled graphical object to allow said host computer to determine a position at which to display said user-controlled graphical object; and

when said host computer is enabled to receive said absolute position of said user-controlled graphical object, determining said absolute position of said user-controlled graphical object from said delta position, reporting said absolute position to said host computer to allow said host computer to display said user-controlled graphical object at a position derived from said absolute position, and using said absolute position in determination of said force output by said actuators if said force is based at least partially on an interaction of said user-controlled graphical object with another graphical object in said graphical environment.

53. A force feedback interface device as recited in claim 52 wherein said absolute position is determined from a ballistic position, said ballistic position being determined from said delta position by using a ballistic algorithm, and wherein said force is determined using said ballistic position.

54. A force feedback interface device as recited in claim 52 wherein said local microprocessor receives a current screen size of said display screen of said host computer to allow said microprocessor to adjust said absolute position based on said screen size.

55. A force feedback interface device as recited in claim 53 wherein said local microprocessor receives ballistic parameters from said host computer which characterize said ballistic algorithm.

56. A force feedback interface device as recited in claim 52 wherein said local microprocessor modifies said absolute position for indexing when said manipulandum and said user-controlled graphical object position have become offset and when said manipulandum is moved within a predetermined distance of a limit to said device workspace.

57. A force feedback interface device as recited in claim 52 wherein said manipulandum is a mouse.

58. A force feedback interface device as recited in claim 52 wherein said host computer is not enabled to receive said absolute position when said host computer has first detected said force feedback interface device, and wherein said host computer is enabled to receive said absolute position after force-feedback related driver software has been loaded into memory of said host computer.

59. A method for controlling a force feedback interface device coupled to a host computer displaying a graphical environment and a cursor within said graphical environment on a display screen, the method comprising:

reading sensor data and determining a position of a user manipulandum in a workspace of said force feedback interface device from said sensor data, said manipulandum being physically contacted and moveable by a user;

determining and reporting a position of said manipulandum to said host computer, said position used by said host computer to display said cursor in said graphical environment; and

receiving information from said host computer describing a current screen size of said display screen of said host computer, wherein said current screen size is included in said determination of said position reported to said host computer.

60. A method as recited in claim 59 wherein said position is a change in position of said manipulandum in said workspace.

61. A method as recited in claim 59 wherein said position is an absolute position at which to display said cursor.

62. A method as recited in claim 59 wherein said screen size is a current screen resolution of said display screen.

63. A method as recited in claim 59 wherein said screen size includes information describing a current aspect ratio of said display screen.

64. A method as recited in claim 59 further comprising determining on said force feedback interface device a force to be output by at least one actuator of said force feedback interface device, wherein said force is determined based at least in part on said position reported to said host computer.

65. A method as recited in claim 64 wherein said position reported to said host computer is a screen position at which to display said cursor, and wherein said determining of said position includes scaling a manipulandum position in accordance with said current screen size to determine said screen position for said current screen size.

66. A method as recited in claim 65 wherein said scaling said device position includes scaling a ballistic position based on said manipulandum position and on a velocity of said manipulandum in said workspace, and wherein said screen position is determined from said ballistic position.

67. A method of interfacing a tactile feedback mouse with a software application running on a host computer, said method comprising:

providing a physical object contacted by a user and moveable by said user on a planar surface;

providing a sensor that tracks motion of said physical object with respect to said planar surface, wherein motion data provided by said sensor is transmitted to said host computer for updating a status of a graphical object on a graphical display device, said motion data transmitted over a communication bus;

providing an actuator that delivers a tactile sensation to said user when said user is in contact with said physical object, said actuator delivering said tactile sensation in response to tactile signals received from said host computer over said communication bus; and

enabling two modes of operation, said two modes being a relative mode wherein said motion data is reported to said host computer as relative data, and an absolute mode wherein said motion is reported to said host computer as absolute data.

68. A method as recited in claim 67 further comprising providing a switch on said tactile feedback mouse, said switch able to select each of said two modes.

69. A method as recited in claim 67 wherein said host computer can select each of said two modes.

70. A method as recited in claim 67 wherein said relative mode is a mouse mode for reporting data when said force feedback device is a mouse, and wherein said absolute mode is a joystick mode for reporting data when said force feedback device is a joystick.

71. A method as recited in claim 67 wherein one of said modes is active at any one time, said active mode being said relative mode when said software application is a general application program running on said host computer and controlling results of interactions between said graphical object and other displayed objects, and said active mode being said absolute mode when said software application is a force feedback game running on said host computer and controlling said graphical object.

72. A method as recited in claim 71 wherein said force feedback games run on said host computer under a Microsoft®

DirectX®

API.

73. A method as recited in claim 67 wherein said relative data is used to provide position control input within said software application and said absolute data is used to provide rate control input within said software application.

74. A method as recited in claim 67 wherein said relative data is interpreted by said software application as mouse motion and said absolute data is interpreted by said software application as joystick motion.

75. A method of interfacing a tactile feedback mouse with a software application running on a host computer, said method comprising:

providing a physical object contacted by a user and moveable in a planar workspace on a planar surface;

providing a sensor that tracks motion of said physical object with respect to said planar surface, wherein motion data provided by said sensor is transmitted to said host computer for updating a location of a graphical object on a graphical display device, said motion data transmitted over a communication bus;

providing an actuator that delivers a tactile sensation to a palm of said user when said palm is in contact with said physical object, said actuator delivering said tactile sensation in response to tactile signals received from said host computer over said communication bus; and

providing said tactile feedback mouse with capability of reporting said motion data to said host computer as relative data and reporting said motion data to said host computer as absolute data, said relative data representing an incremental change in position of said physical object in said planar workspace, and said absolute data indicating an absolute position of said physical object in said planar workspace.

76. A method as recited in claim 75 wherein said capability of reporting said motion data includes two modes of operation such that one of said modes is active on said tactile feedback mouse, said two modes being a relative mode wherein said motion data is reported to said host computer as relative data, and an absolute mode wherein said motion is reported to said host computer as absolute data.

77. A tactile feedback mouse device interfacing with a software application running on a host computer, said mouse device comprising:

a physical object contacted by a user and moveable by said user on a planar surface;

at least one sensor that tracks motion of said physical object with respect to said planar surface, wherein motion data provided by said sensor is transmitted to said host computer for updating a status of a graphical object on a graphical display device, said motion data transmitted over a communication bus;

an actuator operative to output a tactile sensation received by said user when said user is in contact with said physical object, said actuator outputting said tactile sensation in response to tactile signals received by said mouse device from said host computer over said communication bus; and

a local processor electrically coupled to said at least one sensor and to said actuator, said processor providing a reporting mode on said mouse device, wherein said reporting mode can be either of two modes, said two modes including a relative mode wherein said motion data is reported to said software application as relative data, and an absolute mode wherein said motion is reported to said software application as absolute data.

78. A tactile feedback mouse device as recited in claim 77 further comprising a switch on said tactile feedback mouse, said switch able to select which of said reporting modes is active.

79. A tactile feedback mouse device as recited in claim 77 wherein said host computer can select said relative mode or said absolute mode.

80. A tactile feedback mouse device as recited in claim 77 wherein said reporting mode is said relative mode when said software application is a general application program running on said host computer, and wherein said reporting mode is said absolute mode when said software application is a force feedback game running on said host computer.

81. A tactile feedback mouse device as recited in claim 77 wherein said relative data is interpreted by said software application as mouse motion and said absolute data is interpreted by said software application as joystick motion.