Providing force feedback to a user of an interface device based on interactions of a user-controlled cursor in a graphical user interface

• US 7,199,790 B2
• Filed: 01/03/2001
• Issued: 04/03/2007
• Est. Priority Date: 12/01/1995
• Status: Expired due to Fees

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Rosenberg, Louis B., Brave, Scott B.
• Primary Examiner(s): Lefkowitz; Sumati
• Assistant Examiner(s): Kumar; Srilakshmi K
• Application Number:
  US09/756,617
• Publication Number:
  US 20010002126A1
• Time in Patent Office: 2,281 Days
• Field of Search: 345/156, 345/157, 345/161, 345/163, 345/169, 345/701, 345/702, 345/856, 463/30, 463/36, 463/37, 463/38, 273/148.B, 434/45
• US Class Current: 345/175
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: H01H 2003/008 : with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

A method and apparatus for providing force feedback to a user operating a human/computer interface device in conjunction with a graphical user interface (GUI) displayed by a host computer system. A physical object, such as a joystick or a mouse, controls a graphical object, such as a cursor, within the GUI. The GUI allows the user to interface with operating system functions implemented by the computer system. A signal is output from the host computer to the interface device to apply a force sensation to the physical object using one or more actuators. This desired force sensation is associated with at least one of the graphical objects and operating system functions of the graphical user interface and is determined by a location of the cursor in the GUI with respect to targets that are associated with the graphical objects. The graphical objects include icons, windows, pull-down menus and menu items, scroll bars (“sliders”), and buttons. The force sensation assists the user to select a desired operating system function or physically informs the user of the graphical objects encountered by the cursor within the GUI. A microprocessor local to the interface apparatus and separate from the host computer can be used to control forces on the physical object.

Claims

1. A method, comprising:

receiving data associated with a displayed location of a cursor within a graphical user interface, the graphical user interface being associated with a target, the displayed location of the cursor being associated with motion of a manipulandum;

generating a first force feedback when the cursor is moved from a position outside a boundary of the target to a position inside the boundary of the target; and

generating a second force feedback, the second force feedback being one of a vibrational force feedback and a texture force feedback, the first force feedback being the other of the vibrational force feedback and the texture force feedback, the generating the second force feedback occurring when the cursor is moved from a position inside the boundary of the target to a position outside the boundary of the target.

2. The method of claim 1, comprising:

generating a third force feedback, the third force feedback being a different type from the first force feedback and the second force feedback, the generating the third force feedback being based on movement of the cursor within the boundary of the target.

3. The method of claim 2, wherein the third force feedback is one of a vibrational force feedback and a texture force feedback.

4. The method of claim 1, wherein the first force feedback is an attractive force feedback.

5. The method of claim 4, wherein the second force feedback is a barrier force feedback.

6. The method of claim 5, wherein the target is a menu item in a pull-down menu.

7. The method of claim 1, wherein the first force feedback and the second force feedback have different magnitudes.

8. The method of claim 1, wherein the first force feedback is an attractive force feedback, the second force feedback is a barrier force feedback, the first force feedback having a different magnitude than the second force feedback.

9. The method of claim 1, further comprising:

maintaining a history of the position of the cursor; and

determining whether the cursor is moving into or out of the target based on the history of the position of the cursor.

10. The method of claim 1, further comprising:

determining if a signal associated with a selection of the target has been received; and

preventing the generating of the second force feedback if the target has been selected.

11. The method of claim 1, further comprising:

determining if a signal associated with a selection of the target has been received; and

preventing the generating of a third force feedback if the target has been selected while the cursor remains within the boundary of the target.

12. A method, comprising:

receiving an indication of a simulated interaction between a cursor and at least one graphical object displayed in a graphical user interface, the graphical user interface being displayed by a computer;

determining a simulated collision force to be output by an actuator coupled to an interface device, the simulated collision force being based on a force feedback associated with the simulated interaction of the cursor with the at least one graphical object;

scaling a magnitude of the simulated collision force, the scaling being based on a velocity of the cursor in the graphical user interface, the scaling being performed after the simulated collision force is determined; and

outputting the scaled magnitude of the simulated collision force to the interface device.

13. The method of claim 12, wherein the scaling of the magnitude is further based on a simulated mass of the at least one graphical object.

14. The method of claim 12, wherein the at least one graphical object includes a plurality of graphical objects of different types, wherein the determining the simulated collision force is based on the force feedback is associated with all graphical objects of a particular type.

15. The method of claim 12, wherein the graphical user interface is configured to display graphical objects each being a type from a plurality of types, the plurality of types including at least two of icons, windows, and item menus.

16. A method, comprising:

defining at least one graphical object located within a graphical user interface as one of a solid object and a pass-though object;

associating the at least one graphical object defined as a solid object with a force feedback, the force feedback including sensation parameters, the sensation parameters including a duration parameter, a magnitude parameter, and a frequency parameter;

receiving data associated with a displayed location of a cursor within the graphical user interface, the at least one graphical object being associated with a target, the displayed location of the cursor being based on motion of a manipulandum; and

outputting the force feedback associated with the graphical object defined as the solid object when the cursor interacts with the graphical object, whereby the force feedback is output when the cursor interacts with the solid object, and whereby no force feedback is output when the cursor interacts with pass-through objects.

17. The method of claim 16, wherein the force feedback is a vibration.

18. The method of claim 16, wherein the force feedback is a texture sensation.

19. The method of claim 16, wherein the graphical object is an icon.

20. The method of claim 16, wherein the graphical object is a menu item.

21. The method of claim 16, wherein the graphical object is a hyperlink on a web-page.

22. The method of claim 16, wherein the force feedback is associated with a high-level command including the sensation parameters.

23. A method, comprising:

associating a first type of graphical object with a first force feedback, the first force feedback being one of a vibrational force feedback and a texture force feedback, the first type of graphical object being associated with a first interface function;

associating a second type of graphical object with a second force feedback, the second force feedback being the other of the vibrational force feedback and the texture force feedback, the second type of graphical object being associated with a second interface function, the first force feedback and the second force feedback having a sensation parameter, the sensation parameter including a duration parameter;

receiving data associated with a displayed location of a cursor within a graphical user interface, the graphical user interface having a graphical object, the displayed location of the cursor being based on motion of a manipulandum;

determining if the graphical object is of the first type of graphical object or the second type of graphical object based on the data associated with the displayed location of the cursor; and

generating one of the first force feedback and the second force feedback based on the determining.

24. The method of claim 23, the first force feedback having sensation parameters, the sensation parameters including a duration parameter, the method further comprising,

limiting the time duration of an execution of the first force feedback based on the duration parameter.

25. The method of claim 23, the generating one of the first force feedback and the second force feedback further comprising:

generating one of the first force feedback having a first magnitude and the second force feedback having a second magnitude, the first magnitude being different than the second magnitude.

26. The method of claim 23, wherein the first type of graphical object is an icon and the second type of graphical object is a menu item.

27. The method of claim 23, wherein the first type of graphical object is a menu heading and the second type of graphical object is a menu item.

28. The method of claim 23, wherein the first type of graphical object is a hyperlink on a webpage.

29. The method of claim 23, further comprising:

receiving a signal associated with one of the first force feedback and the second force feedback at a microprocessor local to the manipulandum, the signal including a high-level command;

executing a local routine corresponding to the high-level command; and

generating one of the first force feedback and the second force feedback based on the local routine.

30. A method, comprising:

creating a mapping that associates each of a plurality of types of graphical objects with at least one of a plurality of types of force feedback, each type of force feedback including sensation parameters, the sensation parameters having at least a magnitude parameter, a duration parameter, and a frequency parameter;

receiving data associated with a displayed location of a cursor within a graphical user interface, the displayed location of the cursor being based on motion of a manipulandum;

determining when a cursor has entered a boundary region of one of the plurality of graphical objects, the determining being based on the mapping; and

generating a force feedback when the data associated with the displayed location of the cursor is located within a boundary region of at least one graphical object from the plurality of graphical objects, the generating being based on the mapping.

31. The method of claim 30, wherein the plurality of types of graphical objects includes at least one icon, menu, and window.

32. The method of claim 30, wherein the generating further includes:

generating a signal configured to be received by an interface device, the interface device including the manipulandum, the signal including a high-level host command having at least one of the sensation parameters.

33. The method of claim 30, wherein the generating the force feedback further includes:

determining a value of the magnitude parameter for the force feedback based on the velocity of the cursor.

34. The method of claim 30, wherein the selected force feedback is a jolt sensation.

35. The method of claim 30, wherein the selected force feedback is a vibration sensation.

36. The method of claim 30, wherein the selected force feedback is a texture section.

37. The method of claim 30, comprising:

receiving a signal associated with the selected force feedback at a microprocessor local to the manipulandum, the signal including a high-level command;

executing a local routine corresponding to the high-level command;

generating one of the first force feedback and the second force feedback based on the local routine.

38. A method, comprising:

receiving data associated with a location of a displayed cursor within a graphical user interface, the graphical user interface being associated with a target, the data associated with the location of the displayed cursor being associated with motion of a manipulandum;

generating a first force feedback when the cursor is moved from a position outside a boundary of the target to a position inside the boundary of the target;

generating a second force feedback when the cursor is moved from a position inside the boundary of the target to a position outside the boundary of the target, the second force feedback being of a different type of force feedback than the first force feedback; and

generating a third force feedback based on movement of the cursor within the boundary of the target, the third force feedback being a different type from the first force feedback and the second force feedback.

39. A method, comprising:

receiving data associated with a location of a displayed cursor within a graphical user interface, the graphical user interface being associated with a target, the data associated with the location of the displayed cursor being associated with motion of a manipulandum;

generating a first force feedback when the cursor is moved from a position outside a boundary of the target to a position inside the boundary of the target;

generating a second force feedback when the cursor is moved from a position inside the boundary of the target to a position outside the boundary of the target, the second force feedback being of a different type of force feedback than the first force feedback;

determining if a signal associated with a selection of the target has been received; and

generating a third force feedback when the cursor is moved from a position inside the boundary of the target to a position outside the boundary of the target after the signal associated with the selection of the target is received, the third force feedback being of a different type of force feedback than the first force feedback and the second force feedback.

40. A method, comprising:

receiving data associated with a location of a displayed cursor within a graphical user interface, the graphical user interface being associated with a target, the data associated with the location of the displayed cursor being associated with motion of a manipulandum;

generating a first force feedback when the cursor is moved from a position outside a boundary of the target to a position inside the boundary of the target;

generating a second force feedback when the cursor is moved from a position inside the boundary of the target to a position outside the boundary of the target, the second force feedback being of a different type of force feedback than the first force feedback;

determining if a signal associated with a selection of the target has been received; and

discontinuing the generating of the second force feedback when the target has been selected.