Method for providing force feedback to a user of an interface device based on interactions of a controlled cursor with graphical elements in a graphical user interface

• US 6,219,032 B1
• Filed: 12/13/1995
• Issued: 04/17/2001
• Est. Priority Date: 12/01/1995
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Brave, Scott B., Rosenberg, Louis B.
• Primary Examiner(s): Saras, Steven J.
• Assistant Examiner(s): Bell, Paul A.
• Application Number:
  US08/571,606
• Time in Patent Office: 1,952 Days
• Field of Search: 345/145, 345/146, 345/156, 345/157, 345/158, 345/161, 345/163, 395/326, 395/339, 395/340, 395/348, 395/349, 395/352, 395/353, 395/99, 463/30, 463/31, 463/36, 463/37, 463/38, 318/568.11, 318/568.25, 364/578, 340/407.1, 340/407.2, 744/71, 273/148.B, 434/45, 244/223
• US Class Current: 345/157
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: H01H 2003/008 : with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

A method and apparatus for providing force feedback to a user operating a human/computer interface device in conjunction with a graphical user interface (GUI) displayed by a host computer system. A physical object, such as a joystick or a mouse, controls a graphical object, such as a cursor, within the GUI. The GUI allows the user to interface with operating system functions implemented by the computer system. A signal is output from the host computer to the interface device to apply a force sensation to the physical object using one or more actuators. This desired force sensation is associated with at least one of the graphical objects and operating system functions of the graphical user interface and is determined by a location of the cursor in the GUI with respect to targets that are associated with the graphical objects. The graphical objects include icons, windows, pull-down menus and menu items, scroll bars (“sliders”), and buttons. The force sensation assists the user to select a desired operating system function or physically informs the user of the graphical objects encountered by the cursor within the GUI. A microprocessor local to the interface apparatus and separate from the host computer can be used to control forces on the physical object.

Claims

1. A method for providing force feedback to the user of a graphical user interface displayed by a computer system and for aiding a user in manipulating a user-controlled cursor and in targeting graphical elements in said graphical user interface, said graphical elements interfacing said user to functions of said computer system, the method comprising:

receiving data representing a displayed location of said user-controlled cursor within said graphical user interface displayed on a display screen of said computer system, a determination of said displayed location based upon signals received from a user interface device that represent the motion of a manipulatable physical object in at least one degree of freedom, said interface device coupled to said computer system and including an actuator for applying electronically modulated forces to be felt by said user, wherein said forces are modulated as a function of the location of said physical object in said at least one degree of freedom;

selecting a force sensation to be output to said user based at least in part on said data representing said displayed location, wherein a plurality of targets displayed within said graphical user interface are associated with target force sensations that are output to said user, wherein said targets allow said user to interface with operating system functions implemented by said computer system, and wherein a particular target is associated with at least three different ones of said target force sensations, said force sensations being different such that said actuator changes its force output based on said force sensation to be output, a first one of said different force sensations selected to be output to said user when said cursor is moved from a position outside a boundary of said particular target to a position inside said boundary, a second one of said different force sensations selected to be output to said user when said cursor is moved from a position inside said boundary to a position outside said boundary, and a third one of said different force sensations selected to be output to said user when said cursor is moved within said particular target inside said boundary; and

providing a signal to cause said selected force sensation to be output as forces to said user by said actuator.

8. A method as recited in claim 1 wherein a history of sensor readings is used to determine if said cursor is moving into or out of said boundary.

9. A method as recited in claim 1 wherein said second one of said different force sensations is not output as forces by said actuator if said particular target has been selected by said user prior to moving said cursor from inside to outside said boundary.

10. A method as recited in claim 1 wherein said third one of said different force sensations is no longer output as forces by said actuator after said particular target has been selected by said user, provided that said cursor has not exited and re-entered said boundary.

2. A method as recited in 1 wherein said first one of said different force sensations provides an attractive force that assists said user in bringing said cursor into said target boundary.

3. A method as recited in 2 wherein said second one of said different force sensations provides a barrier force that resists motion of said user object in moving said cursor out of said boundary.

4. A method as recited in claim 3 wherein said particular target is a menu item in a pull down menu and wherein said barrier force helps said user from overshooting a particular menu selection in said menu item with said cursor.

5. A method as recited in 1 wherein said force sensations provide different force magnitudes.

6. A method as recited in 1 wherein said third one of said different force sensations is one of a vibration sensation and a texture sensation.

7. A method as recited in 1 wherein said first one of said different force sensations provides an attractive force that assists said user in bringing said cursor upon said target, and wherein said second one of said different force sensations is a barrier force that resists said user in removing said cursor from said target such that a magnitude of said attractive entry force is different than a magnitude of said resistive exit force.

11. A method of assisting a user in manipulating a computer displayed cursor in a graphical user interface displayed on a display screen of a computer system and in targeting graphical elements in said graphical user interface, the method comprising:

receiving a signal from an interface device that includes a user manipulatable object, wherein said signal represents motion of said user manipulatable object in a degree of freedom, said interface device including an actuator for applying electronically modulated forces to said user who is contacting said interface device;

moving a displayed location of said computer-displayed cursor within said graphical user interface based upon signals received from said interface device that represent motion of said user manipulatable object in said degree of freedom; and

causing the application of a force sensation to said user by said actuator, said force sensation based on a position of said user manipulatable object in said degree of freedom which causes said cursor to interact with a particular one of said graphical elements, wherein a magnitude of said force sensation is scaled in proportion to a size of a collection of data represented by said particular graphical element.

12. A method as recited in claim 11 said force sensation attracts said cursor towards said particular graphical element, wherein said particular graphical element is a graphically displayed icon that represents said collection of data, said collection of data being a file in an operating system, wherein a magnitude of said attractive force sensation is sealed with respect to a size of said file that is represented by said icon.

13. A method as recited in claim 12 wherein said attractive force is output when said cursor is on or outside a boundary of said particular graphical element.

14. A method as recited in claim 11 wherein said force sensation is a first force sensation, and wherein a dead region is defined within a boundary of said particular graphical element such that a second force sensation different from said first force sensation is applied by said actuator if said cursor is within said dead region.

15. A method as recited in claim 14 wherein said second force sensation is one of a damping force sensation and a texture force sensation.

16. A method as recited in claim 11 wherein said force sensation is an inertia sensation and wherein said particular graphical element is an icon such that said user feels a weight of said icon when performing a drag operation on said icon with said cursor, wherein a mass value used in a computation of said inertia sensation is scaled with respect to a size of said collection of data represented by said icon.

17. A method as recited in claim 11 wherein said force sensation is a damping force sensation and said particular graphical element is an icon such that said user feels resistance to moving said icon when performing a drag operation on said icon with said cursor, wherein a damping coefficient used in a computation of said damping sensation is scaled with respect to said size of said collection of data represented by said icon.

18. method as recited in claim 11 wherein said force sensation is an inertia sensation and wherein said graphical element is a thumb of a scroll bar such that said user feels a weight of said thumb when performing a drag operation on said thumb of said scroll bar with said cursor, wherein a mass used in a computation of said inertia sensation is scaled with respect to said size of said collection of data represented being scrolled by dragging said thumb of said scroll bar.

19. A method as recited in claim 11 wherein said force sensation is a damping sensation and wherein said graphical element is a thumb of a scroll bar such that said user feels resistance to moving said thumb when performing a drag operation on said thumb of said scroll bar with said cursor, wherein a damping coefficient used in a computation of said damping sensation is scaled with respect to said size of said collection of data represented being scrolled by dragging said thumb of said scroll bar.

20. A method for providing force feedback to the user of a graphical user interface displayed by a computer system, said user using an interface device including a physical object contacted and moved by said user in a plurality of degrees of freedom, an actuator for imparting an electronically modulated force to said user, and a sensor apparatus for providing a locative signal responsive to and corresponding with manipulation of said object in said degrees of freedom, wherein a displayed cursor has a location correlated to said physical object as indicated by said locative signal, the method comprising:

receiving an indication that said cursor is interacting with at least one graphical object displayed in said graphical user interface;

determining a total force to be output by said actuator of said interface device, said total force based on a combination of a plurality of force sensations associated with said at least one graphical object;

scaling a magnitude of said total force, wherein said scaling is based on a current velocity of said cursor in said graphical user interface, said scaling being performed after said total force is determined from said combination of associated force sensations; and

outputting said scaled total force to said physical object by said actuator of said interface device.

21. A method as recited in claim 20 wherein said cursor is assigned a mass and said mass influences said magnitude that is scaled.

22. A method as recited in claim 20 wherein said at least one force sensation is associated with all graphical objects having a particular type, wherein said at least one graphical object has said particular type.

23. A method as recited in claim 22 wherein said graphical user interface displays graphical objects of a plurality of types, said types including icons, windows, and menu items.

24. A method of creating a tactile user interface performed on a computer system, wherein a user feels force sensations when a graphically displayed cursor interacts with a graphical object associated with a function of said computer system and displayed in a graphical user interface on a display device, and a location of said cursor on said display device is updated based on a received indication of movement of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system, said method comprising:

associating a graphical object with an entry force sensation, said entry force sensation represented by entry sensation parameters;

associating said graphical object with an exit force sensation different from said entry force sensation, said exit force sensation represented by exit sensation parameters having different values from said entry sensation parameters;

if said cursor has entered a region associated with said graphical object based on movement of said physical object, sending a first signal to said human interface device, said first signal representing said entry force sensation, wherein said human interface device outputs said entry force sensation to said user using at least one electronically-modulated actuator; and

if said cursor has exited said region associated with said graphical object based on movement of said physical object, sending a second signal to said human interface device, said second signal representing said exit force sensation, wherein said human interface device outputs said exit force sensation to said user using said at least one electronically-modulated actuator.

25. A method as recited in claim 24 wherein said sensation parameters include a duration parameter that indicates the time duration for which said first entry force sensation should execute.

26. A method as recited in claim 24 wherein a magnitude of said entry force sensation is different from than a magnitude of said exit force sensation.

27. A method as recited in claim 24 wherein said entry force sensation is a vibration sensation.

28. A method as recited in claim 27 wherein said entry sensation parameters and exit sensation parameters include magnitude, frequency, and duration parameters.

29. A method as recited in claim 24 wherein said entry force sensation is a texture sensation.

30. A method as recited in claim 24 wherein said graphical object is an icon.

31. A method as recited in claim 24 wherein said graphical object is a menu item.

32. A method as recited in claim 24 wherein said graphical object is a hyperlink on a web page.

33. A method as recited in claim 24 wherein said first signal or said second signal is a high-level command including said entry sensation parameters or exit sensation parameters, respectively.

34. A method as recited in claim 33 wherein said parameters include a magnitude and a duration.

35. A method of creating a tactile user interface on a computer system wherein a user feels force sensations when a graphically displayed cursor interacts with a particular graphical object displayed in a graphical user interface on a display device, and a location of said cursor on said display device is updated based on a received indication of movement of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system, said method comprising:

associating on said computer system a first type of graphical object with a first entry force sensation, said first type of graphical object associated with a first user interface function, said first entry force sensation represented by first sensation parameters;

associating on said computer system a second type of graphical object with a second entry force sensation, said second type of graphical object associated with a second user interface function, said second entry force sensation represented by second sensation parameters having different values from said first sensation parameters;

determining on said computer system if said particular graphical object is of said first type or of said second type;

if said cursor has entered a region associated with said particular graphical object based on movement of said physical object, and if said particular graphical object is of said first type, sending a signal from said computer system to said human interface device, said signal representing said first entry force sensation, wherein said first entry force sensation is output to said user using an electronically-modulated actuator of said human interface device; and

if said cursor has entered said region associated with said particular graphical object based on movement of said physical object, and if said particular graphical object is of said second type, sending a signal from said computer system to said human interface device, said signal representing said second entry force sensation, wherein said second entry force sensation is output to said user using an electronically-modulated actuator of said human interface device.

36. A method as recited in claim 35 wherein said sensation parameters include a duration parameter that indicates the time duration for which said first entry force sensation should execute.

37. A method as recited in claim 35 wherein a magnitude of said first entry force sensation is different from a magnitude of said second entry force sensation.

38. A method as recited in claim 35 wherein said first entry force sensation is a vibration, wherein said parameters representing said first force sensation include magnitude, frequency, and duration parameters.

39. A method as recited in claim 35 wherein said first entry force sensation is a texture.

40. A method as recited in claim 35 wherein said first type of graphical object is an icon and said second type of graphical object is a menu item.

41. A method as recited in claim 35 wherein said first type of graphical object is a menu heading and said second type of graphical object is a menu item.

42. A method as recited in claim 35 wherein said first type of graphical object is a hyperlink on a web page.

43. A method as recited in claim 35 wherein said signal representing said first entry force sensation or said signal representing said second entry force sensation is sent to a local microprocessor included in said interface device and separate from said computer system, said local microprocessor receiving said signal as a high level command and executing a local routine corresponding to said command to output said force sensation.

44. A method of creating a tactile user interface using a computer system wherein a user feels forces when a graphically displayed cursor interacts with a particular graphical object displayed in a graphical user interface on a display device, said graphical object associated with a function of said computer system, and a location of said cursor on said display device is updated based on an indication of movement, received over a communication bus, of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system over said communication bus, said method comprising:

creating a mapping with said computer system that associates each of a plurality of types of graphical objects in said graphical user interface with at least one of a plurality of force sensations using said computer system, at least two of said assigned force sensations being different and represented by sensation parameters including a magnitude and a duration;

if said cursor has entered a boundary of a region associated with said particular graphical object, selecting with said computer system an appropriate one of said force sensations based on said mapping and said type of said particular graphical object; and

sending a signal from said computer system to said human interface device over said communication bus, said signal representing said selected force sensation, wherein said selected force sensation is output to said user using an electronically-modulated actuator of said human interface device.

45. A method as recited in claim 44 wherein said types of graphical objects include at least one icon type, menu item type, and window type.

46. A method as recited in claim 44 wherein said signal sent over said communication bus is a high level host command that includes at least one of said sensation parameters.

47. A method as recited in claim 44 wherein said sensation parameters include a frequency.

48. A method as recited in claim 44 wherein said magnitude of said selected force sensation is determined in part by a velocity of said cursor.

49. A method as recited in claim 44 wherein said selected force sensation is a jolt sensation.

50. A method as recited in claim 44 wherein said selected force sensation is a vibration sensation.

51. A method as recited in claim 44 wherein said selected force sensation is a texture sensation.

52. A method as recited in claim 44 wherein said signal representing said selected force sensation is sent to a local microprocessor included in said interface device and separate from said computer system, said local microprocessor receiving said signal as a high level command and executing a local routine corresponding to said high level command to output said selected force sensation.

53. A method of assisting a user in manipulating a computer displayed cursor in a graphical user interface displayed on a display screen of a computer system and in targeting graphical elements in said graphical user interface, the method comprising:

determining if a particular graphical element in said graphical user interface with which said cursor interacts is associated with a force sensation, wherein a displayed location of said cursor is based upon signals received from said interface device that represent motion of said user manipulatable object, and wherein said interface device includes an actuator for outputting electronically modulated forces to said user; and

causing the application of a force sensation to said user by said actuator, said force sensation based on a position or movement of said user manipulatable object which causes said cursor to interact with said particular one of said graphical elements, wherein a magnitude of said force sensation is scaled in proportion to a size of a collection of data represented by said particular graphical element.

54. A method for providing force feedback to the user of a graphical user interface displayed by a computer system and for aiding a user in manipulating a user-controlled cursor and in targeting graphical elements in said graphical user interface, said graphical elements interfacing said user to functions of said computer system, the method comprising:

receiving data representing a displayed location of said user-controlled cursor within said graphical user interface displayed on a display screen of said computer system, a determination of said displayed location based upon signals received from a user interface device that represent the motion of a manipulatable physical object in at least one degree of freedom, said interface device coupled to said computer system and including an actuator for applying electronically modulated forces to be felt by said user, wherein said forces are modulated as a function of the location of said physical object in said at least one degree of freedom;

selecting a force sensation to be output to said user based at least in part on said data representing said displayed location, wherein a plurality of targets displayed within said graphical user interface are associated with target force sensations that are output to said user, wherein said targets allow said user to interface with operating system functions implemented by said computer system, and wherein a particular target is associated with at least two different ones of said target force sensations, said force sensations being different such that said actuator outputs forces based on parameters having different values for each of said force sensations, a first one of said different force sensations selected to be output to said user when said cursor is moved from a position outside a boundary of said particular target to a position inside said boundary, and a second one of said different force sensations selected to be output to said user when said cursor is moved from a position inside said boundary to a position outside said boundary; and

providing a signal to cause said selected force sensation to be output as forces to said user by said actuator.

55. A method of creating a tactile user interface performed on a computer system, wherein a user feels force sensations when a graphically displayed cursor interacts with a graphical object associated with a function of said computer system and displayed in a graphical user interface on a display device, and a location of said cursor on said display device is updated based on a received indication of movement of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system, said method comprising:

enabling an association of a graphical object with an entry force sensation, said entry force sensation represented by entry sensation parameters;

enabling an association of said graphical object with an exit force sensation different from said entry force sensation, said exit force sensation represented by exit sensation parameters having different values from said entry sensation parameters;

if said cursor has entered a region associated with said graphical object based on movement of said physical object, enabling a sending of a first signal to said human interface device, said first signal representing said entry force sensation, wherein said human interface device outputs said entry force sensation to said user using at least one electronically-modulated actuator; and

if said cursor has exited said region associated with said graphical object based on movement of said physical object, enabling a sending of a second signal to said human interface device, said second signal representing said exit force sensation, wherein said human interface device outputs said exit force sensation to said user using said at least one electronically-modulated actuator.

56. A method of creating a tactile user interface on a computer system wherein a user feels force sensations when a graphically displayed cursor interacts with a particular graphical object displayed in a graphical user interface on a display device, and a location of said cursor on said display device is updated based on a received indication of movement of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system, said method comprising:

enabling an association on said computer system of a first type of graphical object with a first entry force sensation, said first type of graphical object associated with a first user interface function, said first entry force sensation represented by at least one first sensation parameter;

enabling an association on said computer system of a second type of graphical object with a second entry force sensation, said second type of graphical object associated with a second user interface function, said second entry force sensation represented by at least one second sensation parameter having different values from said at least one first sensation parameter;

enabling a determination on said computer system of a type of said particular graphical object, said type being one of said first type or said second type;

if said cursor has entered a region associated with said particular graphical object based on movement of said physical object, and if said particular graphical object is of said first type, enabling a sending of a signal from said computer system to said human interface device, said signal representing said first entry force sensation, wherein said first entry force sensation is then output to said user using an electronically-modulated actuator of said human interface device; and

if said cursor has entered said region associated with said particular graphical object based on movement of said physical object, and if said particular graphical object is of said second type, enabling a sending of a signal from said computer system to said human interface device, said signal representing said second entry force sensation, wherein said second entry force sensation is then output to said user using an electronically-modulated actuator of said human interface device.

57. A method of creating a tactile user interface using a computer system wherein a user feels forces when a graphically displayed cursor interacts with a particular graphical object displayed in a graphical user interface on a display device, said graphical object associated with a function of said computer system, and a location of said cursor on said display device is updated based on an indication of movement, received over a communication bus, of a physical object that is manipulated by said user, said physical object being included in a human interface device that outputs said indication to said computer system over said communication bus, said method comprising:

enabling a creation of a mapping with said computer system that associates each of a plurality of types of graphical objects in said graphical user interface with at least one of a plurality of force sensations using said computer system, at least two of said assigned force sensations being different and represented by sensation parameters including a magnitude and a duration;

if said cursor has entered a boundary of a region associated with said particular graphical object, enabling a selection with said computer system of an appropriate one of said force sensations based on said mapping and said type of said particular graphical object; and

enabling a sending of a signal from said computer system to said human interface device over said communication bus, said signal representing said selected force sensation, wherein said selected force sensation is then output to said user using an electronically-modulated actuator of said human interface device.

58. A method as recited in claim 57 wherein said signal representing said selected force sensation is sent to a local microprocessor included in said interface device and separate from said computer system, said local microprocessor receiving said signal as a high level command and executing a local routine corresponding to said high level command to output said selected force sensation.