Providing force feedback to a user of an interface device based on interactions of a user-controlled cursor in a graphical user interface

• US 20010002126A1
• Filed: 01/08/2001
• Published: 05/31/2001
• Est. Priority Date: 12/01/1995
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Brave, Scott B., Rosenberg, Louis B.
• Granted Patent: US 7,199,790 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: H01H 2003/008 : with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

A method and apparatus for providing force feedback to a user operating a human/computer interface device in conjunction with a graphical user interface (GUI) displayed by a host computer system. A physical object, such as a joystick or a mouse, controls a graphical object, such as a cursor, within the GUI. The GUI allows the user to interface with operating system functions implemented by the computer system. A signal is output from the host computer to the interface device to apply a force sensation to the physical object using one or more actuators. This desired force sensation is associated with at least one of the graphical objects and operating system functions of the graphical user interface and is determined by a location of the cursor in the GUI with respect to targets that are associated with the graphical objects. The graphical objects include icons, windows, pull-down menus and menu items, scroll bars (“sliders”), and buttons. The force sensation assists the user to select a desired operating system function or physically informs the user of the graphical objects encountered by the cursor within the GUI. A microprocessor local to the interface apparatus and separate from the host computer can be used to control forces on the physical object.

Claims

1. A method for providing force feedback to users interacting with a graphical user interface environment of a computer system, the method comprising steps of:

receiving an indication of movement of a physical object that is manipulated by a user, said physical object being included in a human interface device that outputs said indication to said computer system;

moving a user-controlled graphical object within a graphical user interface based on said indication of said movement of said physical object, wherein said user-controlled graphical object and said graphical user interface are displayed on a display screen connected to said computer system, and wherein said graphical user interface allows said user to interface with operating system functions implemented by said computer system;

outputting an signal from said computer system to said interface device to command said interface device to apply a force sensation to said physical object, wherein said force is associated with at least one of said operating system functions of said graphical user interface.

2. A method as recited in

claim 1

wherein said user-controlled graphical object is a cursor.

3. A method as recited in

claim 2

wherein said force sensation applied to said physical object is determined, at least in part, by a location of said cursor in said graphical user interface with respect to targets located in said graphical user interface.

4. A method as recited in

claim 2

wherein said force sensation applied to said physical object is determined, at least in part, by a velocity of said cursor moving within said graphical user interface.

5. A method as recited in

claim 2

wherein said force sensation applied to said physical object is determined, at least in part, by an acceleration of said cursor moving within said graphical user interface.

6. A method as recited in

claim 2

wherein said force sensation applied to said physical object is determined, at least in part, by a history of previous locations of said cursor within said graphical user interface with respect to targets located in said graphical user interface.

7. A method as recited in

claim 3

wherein said force sensation on said physical object assists said user to select said at least one operating system function associated with said force sensation.

8. A method as recited in

claim 3

wherein said force sensation on said physical object informs said user of other graphical objects in said GUI which can be manipulated to perform said at least one operating system function.

9. A method as recited in

claim 7

further comprising a step of performing one of said operating system functions as indicated by said location of said cursor and as indicated by a command from said user, wherein said force sensation applied to said physical object assists said user in selecting said operating system function.

10. A method as recited in

claim 9

wherein said performed operating system function is associated with a particular type of target of said graphical user interface.

11. A method as recited in

claim 9

wherein a magnitude of said force sensation applied to said physical object depends on a particular target of said graphical user interface into which said cursor is moved.

12. A method as recited in

claim 9

wherein a type of said force sensation applied to said physical object depends on a particular target of said graphical user interface into which said cursor is moved.

13. A method as recited in

claim 9

wherein said target region includes an icon, and wherein said command from said user includes selecting a button on the interface device.

14. A method as recited in

claim 13

wherein said operating system function includes executing a program that is independent from said operating system, said program being associated with said icon selected by said cursor.

15. A method as recited in

claim 13

wherein said force sensation is an attractive force between said icon and said cursor, said attractive force being applied to said user object when said cursor is within a predetermined distance of said icon to assist said user in selecting said icon.

16. A method as recited in

claim 3

wherein said force sensation is an obstructive force positioned at a boundary of a target to prevent said user from easily moving said cursor out of said target.

17. A method as recited in

claim 7

wherein said force sensation is an inertia force applied to said physical object when a target region is moved by said cursor in said graphical user interface.

18. A method as recited in

claim 17

wherein said inertial force is a damping force that damps movement of said user object according to a damping constant and the velocity of said cursor.

19. A method as recited in

claim 1

wherein said force sensation is a damping force applied to said physical object as said cursor is moved on said display screen.

20. A method as recited in

claim 1

wherein said physical object is a joystick handle.

21. A method as recited in

claim 1

wherein said physical object and said interface device are included in a mouse.

22. A method as recited in

claim 1

wherein said force sensation is applied to said physical object by an electrically controlled actuator of said interface device.

23. A method as recited in

claim 2

wherein said signal output to said interface device is a high level host command sent to microprocessor local to said interface device and separate from said computer system,.

24. A method as recited in

claim 23

further comprising a step of providing layout data to said microprocessor, said layout data indicating a spatial layout of targets in said GUI such that said microprocessor can check said spatial layout to determine when force sensations associated with said targets may be applied to said user object.

25. A including program instructions for performing steps for a process of providing force feedback to the user of a graphical user interface displayed by a computer system, the steps comprising:

determining a location of a user-controlled cursor within a graphical user interface displayed on a display screen of a computer system, said cursor being controlled by said user by manipulating a physical object of an interface device;

determining which targets displayed within said graphical user interface are associated with target forces that affect said physical object based on said location of said user-controlled cursor, wherein said targets allow said user to interface with operating system functions implemented by said graphical user interface;

providing a signal to apply a resulting force to said physical object based on said forces associated with said targets which affect forces on said physical object.

26. A computer readable medium as recited in

claim 25

wherein each of said targets are associated with at least two different target force sensations depending on said location of said cursor with respect to each of said targets.

27. A computer readable medium as recited in

claim 26

wherein a position control paradigm is implemented such that said location of said cursor in said graphical user interface approximately corresponds to a location of said physical object with reference to an origin of said interface device.

28. A computer readable medium as recited in

claim 27

wherein said two different target force sensations include an internal target force sensation and an external target force sensation, wherein said internal target force sensation is applied to said physical object when said cursor is located within said target, and said external target force sensation is applied to said physical object when said cursor is located outside said target.

29. A computer readable medium as recited in

claim 28

wherein said internal target force sensation includes a capture force sensation when said cursor is located near a boundary of said target, and wherein said internal target force sensation includes a dead region force sensation when said cursor is located near a center of said target.

30. A computer readable medium as recited in

claim 28

wherein said targets of said GUI are ordered in a hierarchy, and wherein said step of determining which targets may affect force sensations on said physical object includes determining a cursor target of lowest hierarchy in which said cursor is located.

31. A computer readable medium as recited in

claim 30

wherein an internal target force sensation associated with said cursor target affects said physical object and wherein no other internal force sensations of other targets affect said physical object.

32. A computer readable medium as recited in

claim 31

wherein other targets at a hierarchical level lower than said cursor target have external force sensations that affect said physical object, wherein said cursor is not located in said other targets.

33. A computer readable medium as recited in

claim 32

wherein said resulting force sensation applied to said physical object is a sum of said internal target force sensation of said cursor target and said external target force sensations of said other targets at said hierarchy lower than said lowest hierarchical level.

34. A computer readable medium as recited in

claim 33

wherein said other targets are included within said cursor target and are at a lower hierarchical level than said cursor target.

35. A computer readable medium as recited in

claim 32

wherein when said cursor is located in at least two of said cursor targets, only the target displayed on top of said other targets of said at least two targets is selected as said cursor target.

36. A computer readable medium as recited in

claim 25

further comprising a step of mapping force sensations to targets provided by said graphical user interface.

37. A computer readable medium as recited in

claim 36

wherein said step of mapping force sensations includes assigning ranges to said targets, said ranges defining external regions around said targets which affect said external force sensation on physical object when said cursor is located within said external regions.

38. A computer readable medium as recited in

claim 37

wherein said ranges are assigned a shape.

39. A computer readable medium as recited in

claim 38

wherein said step of mapping force sensations includes mapping internal and external target force sensations to said targets.

40. A computer readable medium as recited in

claim 29

wherein said capture force sensation includes a snap-over force sensation, said snap-over force sensation providing a resistance to said movement of said cursor when said cursor from within said target to outside said target.

41. A computer readable medium as recited in

claim 29

wherein said external target force sensation includes a force field, said force field providing an attractive force sensation to said physical object to draw said cursor toward said target.

42. A computer readable medium as recited in

claim 29

wherein said external target force sensation includes a force field, said force field providing a repulsive force sensation to said physical object to push said cursor away from said target.

43. A computer readable medium as recited in

claim 29

wherein said external target force sensation includes two intersecting groove force sensations, said groove force sensations biasing said cursor to move toward said target and locking said cursor on said target.

44. A computer readable medium as recited in

claim 43

further comprising a step of removing said groove force sensations after said cursor is locked on said target when said target is selected by said user.

45. A computer readable medium as recited in

claim 36

wherein said target is an icon.

46. A computer readable medium as recited in

claim 36

wherein said target is a button.

47. A computer readable medium as recited in

claim 36

wherein said target is a slider.

48. A computer readable medium as recited in

claim 47

wherein said slider includes a thumb movable on a guide, and wherein an attractive force attracts said cursor to said thumb.

49. A computer readable medium as recited in

claim 36

wherein said target is a menu item on a pull-down menu.

50. A computer readable medium as recited in

claim 36

wherein said target is a window.

51. A computer system for providing force feedback to a user of a graphical user interface displayed by said computer system, comprising:

means for determining a location of a user-controlled cursor within a graphical user interface displayed on a display screen of a computer system, said cursor being controlled by said user by manipulating a physical object of an interface device, and wherein targets are provided in said graphical user interface which are associated with operating system functions implemented by said graphical user interface, said cursor being movable to select said targets to select said associated operating system function;

means for determining which targets displayed within said graphical user interface are associated with target forces affecting said physical object based on said location of said user-controlled cursor;

means for providing a signal to apply a resulting force sensation to said physical object based on said forces associated with said targets which affect forces on said physical object.

52. A computer system as recited in

claim 51

wherein said means for determining said targets includes means for determining a cursor target of lowest hierarchy in which said cursor is located.

53. A computer system as recited in

claim 52

wherein said cursor target is a menu item in a pull-down menu.

54. A computer system as recited in

claim 53

wherein said menu item is associated with a snap-over force at a boundary of said menu item, said snap-over force providing a obstructive force to said movement of said cursor when said cursor is moved from one menu item to another menu item.

55. A computer system as recited in

claim 53

wherein said menu item is associated with a snap-to force at a center area of said menu item, said snap-to force providing an attractive force that keeps said cursor at said center area of said menu item.

56. A computer system as recited in

claim 52

wherein said cursor target is a slider having a thumb that can be moved within a linear guide.

57. A computer system as recited in

claim 56

wherein said slider is associated with a snap-to force provided at a middle line along a length and halfway across a width of said guide of said slider, said snap-to force providing an attractive force that keeps said cursor at said middle line of said guide.

58. A computer system as recited in

claim 52

wherein said cursor target is an icon.

59. A computer system as recited in

claim 56

wherein said icon is associated with an attractive force associated with a surrounding region having a predetermined size, said attractive force pulling said physical object and said cursor when said cursor is moved within said region.

60. A computer system as recited in

claim 59

wherein said icon is associated with an capture force associated with a boundary of said icon, said capture force providing an obstructive force to said physical object when said cursor is moved out of said icon.

61. A computer system as recited in

claim 59

wherein said icon is associated with a dead region force associated with an internal region of said icon, said dead region force providing a predetermined damping force to said physical object when said cursor is moved in said internal region.

62. A computer system as recited in

claim 52

wherein said target is a graphical button.

63. A system for providing force feedback to a user manipulating an interface device, the system comprising:

a host computer system for receiving an input control signal describing a location of a user manipulable object in a degree of freedom and for providing a host output control signal, wherein said host computer system updates the location of a user-controlled cursor within a graphical user interface displayed on a display screen of said host computer system, said cursor being updated based on said input control signal;

a processor local to said interface device and separate from said host computer system for receiving said host output control signal from said host computer system and providing a processor output control signal;

an actuator for receiving said processor output control signal and providing a force along said degree of freedom in accordance with said processor output control signal to said user manipulable object coupled to said actuator; and

a sensor for detecting motion of said user manipulable object along said degree of freedom and outputting said input control signal including information representative of the position and motion of said object.

64. A system as recited in

claim 63

wherein said sensor outputs said input control signal to said processor, and wherein said processor provides said input control signal to said host computer.

65. A system as recited in

claim 63

wherein said host computer system displays a number of targets in said graphical user interface and determines which targets are associated with target forces that affect said physical object based on said location of said user-controlled cursor, wherein said targets allow said user to interface with operating system functions implemented by said graphical user interface.

66. A system as recited in

claim 65

wherein said host output control signal commands resulting force to said physical object based on said forces associated with said targets which affect forces on said physical object.

67. A system as recited in

claim 66

wherein said processor is operative in a reflex process to provide said processor output control signal to said actuator in response to said position and motion of said object independently of said host output control signal.

68. A system as recited in

claim 67

wherein said host output signal is a high level command from said host computer system, and wherein said processor implements one of a plurality of local processes selected in accordance with said high level command to implement said reflex process.

69. A system as recited in

claim 65

wherein said user manipulable object can be moved by said user in a plurality of degrees of freedom, and wherein said system further comprises an actuator for providing a force along a degree of freedom of said object, and a sensor for detecting motion of said object in said degree of freedom for each of said plurality of degrees of freedom.

70. A system as recited in

claim 65

further comprising a serial interface for outputting said host output control signal from said host computer system and for receiving said input control signal to said host computer system.

71. A system as recited in

claim 70

wherein said serial interface includes a Universal Serial Bus.

72. A system as recited in

claim 65

further comprising a clock coupled to said processor, wherein said processor accesses said clock to determine, in part, said force provided by said actuator.

73. A system as recited in

claim 65

further comprising a memory device accessible by said local processor, wherein said local processor stores a spatial representation of said targets in said memory device such that said processor may detect when said target forces affect said physical object.

74. A system as recited in

claim 73

wherein said memory device includes a permanent memory storage for storing predetermined target forces associated with said targets.

75. A method for providing force feedback for graphical objects in a game implemented on a computer system, the method comprising the steps of:

displaying a user-controlled first graphical object on a display screen of a computer system, said graphical object moving on said display screen during a game in response to manipulations of a physical object of an interface device by a user, said interface device being coupled to said computer system;

displaying a second graphical object on said display screen;

when said first graphical object collides with said second graphical object on said screen;

a) displaying a compression of said first object where said second object contacts said first object, wherein said first object has a predetermined simulated compliance and said second object has a predetermined simulated mass;

b) outputting a force command to said interface device to apply a force to said physical object manipulated by said user in at least one degree of freedom provided by said interface device, said force being applied in the direction of said compression and having a magnitude in accordance with said simulated masses of said first and second graphical objects.

76. A method as recited in

claim 75

wherein said force has a magnitude also in accordance with a simulated compliance of said first graphical object.

77. A method as recited in

claim 76

wherein said force has a magnitude also in accordance with simulated velocities of said first graphical object and said second graphical object.

78. A method as recited in

claim 76

wherein said force has a magnitude also in accordance with a simulated gravity acting on said objects.

79. A method as recited in

claim 76

wherein said first graphical object is a linear segment, and wherein said second graphical object is a circular object.

80. A method as recited in

claim 76

wherein said second graphical object moves on said display screen during a game in response to manipulations of a second physical object of an second interface device by a second user, said second interface device being coupled to said computer system.

81. A method as recited in

claim 80

wherein said first and second interface devices input sensor signals indicating motion of said first physical object and said second physical object, respectively, wherein said user is a first user and said force command to said first interface device is based on sensor signals output from said second interface device, and wherein a force command output to said second interface device is based on sensor signals output from said first interface device.

82. A method as recited in

claim 76

wherein said second graphical object moves on said display screen during a game in response to manipulations of a second physical object of an second interface device by a second user, said second interface device being coupled to a second computer system coupled to said computer system through a network interface.

83. A method as recited in

claim 76

further comprising a step of changing said simulated compliance of said first graphical object when an input command is received from said user operating an input device on said interface device.

84. A method as recited in

claim 76

further comprising displaying a goal object on said display screen and wherein said user moves said first graphical object to block said second graphical object from moving into said goal object.

85. A method as recited in

claim 79

further comprising changing a displayed orientation of said linear segment according to input data received from said user operating said input device.

86. A method as recited in

claim 85

wherein said physical object is a joystick handle, and wherein said input data received from said user includes a spin of said joystick handle in a rotary degree of freedom.

87. A method as recited in

claim 86

wherein said interface device provides two linear degrees of freedom to said joystick handle in addition to said rotary degree of freedom.