Method and apparatus for providing simulated physical interactions within computer generated environments

• US 6,028,593 A
• Filed: 06/14/1996
• Issued: 02/22/2000
• Est. Priority Date: 12/01/1995
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Rosenberg, Louis B., Brave, Scott B.
• Primary Examiner(s): Brier, Jeffery
• Assistant Examiner(s): Bell, Paul A
• Application Number:
  US08/664,086
• Time in Patent Office: 1,348 Days
• Field of Search: 345/145, 345/146, 345/156, 345/158, 345/161, 345/163, 345/326, 345/339, 345/348, 345/349, 345/352, 345/353, 345/949, 345/952, 345/959, 345/967, 463/30, 463/36, 463/37, 463/38, 318/568.11, 318/568.25, 318/561, 482/4, 273/148 B
• US Class Current: 345/156
• CPC Class Codes: A63B 24/00 Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
• CPC Class Codes: A63F 11/0051 Indicators of values, e.g. score counters
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/573 using trajectories of game objects, e.g. of a golf ball according to the point of impact
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0383 : Signal control means within the pointing device
• CPC Class Codes: H01H 2003/008 : with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

A method and apparatus for providing force feedback to a user operating a human/computer interface device and interacting with a computer-generated simulation. In one aspect, a computer-implemented method simulates the interaction of simulated objects displayed to a user who controls one of the simulated objects manipulating a physical object of an interface device. The position of the simulated object, as provided within the simulation and as displayed, is mapped directly to the physical position of the user object. This mapping is broken under conditions that are effective to provide force feedback to the user which imparts a physical sensation corresponding to the interaction of the simulated objects. In another aspect, a ball simulated ball object interacts with a user-controlled simulated object in a simulation to allow the user to utilize a wide range of physical skill and dexterity in interacting with the simulation. In another aspect, a simulation apparatus provides a display device such as one or more display screens or a projection device, and which also provides an intuitive mechanical interface device for the user to skillfully and dexterously manipulate objects within a computer-generated simulation.

Claims

1. A computer-implemented method for simulating the spatial interaction of a user-manipulated displayed first simulated object with a displayed second simulated object in a computer-simulated spatial environment, such that the user is provided with force feedback that represents the interaction between the first simulated object and second simulated object, the method comprising:

a) generating within a computer-simulated spatial environment a first simulated object on a display device, said first simulated object having a simulated location within said simulated spatial environment;

b) moving under computer control said first simulated object in response to motion of a physical object of an interface device controlled by a user, wherein said physical object has a physical position in a physical environment, and wherein a mapping between said simulated location of said first simulated object and said physical position of said physical object is created such that the physical position of the physical object in the physical environment has a spatial correspondence to the location of said first simulated object in said simulated spatial environment;

c) generating within said computer-simulated spatial environment a second simulated object on said display device, said second simulated object defining a set of boundaries in said motion simulation and on said display device;

d) determining whether said first simulated object is interacting with said second simulated object by examining a current location of said first simulated object and a previous location of said first simulated object to determine a path of said first simulated object; and

e) breaking said mapping between said simulated location of first simulated object and said physical position of said physical object when said first simulated object interacts with said second simulated object such that said physical position of said physical object is no longer spatially correlated with said simulated location of said first simulated object, said breaking occurring under conditions effective to provide visual feedback and force feedback to said user which are effective to impart to said user a physical sensation that corresponds to a realistic, simulated physical interaction of said first simulated object with said second simulated object.

2. The method of claim 1, wherein said physical position of said physical object is mapped to said simulated location of said first simulated object by a position control mapping.

3. The method of claim 2, wherein said spatial correspondence includes a scale factor.

4. The method of claim 1, wherein said simulated location of said first simulated object includes a displayed location on said display device that is correlated with said simulated location.

5. The method of claim 4, wherein said first simulated object interacts with said second simulated object after moving along a trajectory which intersects boundaries of said second simulated object.

6. The method of claim 4, wherein said force feedback corresponds to a restoring force having a magnitude that is proportional to a degree that said mapping is broken.

7. The method of claim 6, wherein said amount of said breaking of said mapping is proportional to the degree to which said physical position of said physical object exceeds the result of said interaction between said first simulated object and said second simulated object as determined by said simulation.

8. The method of claim 7, wherein said restoring force is a spring force having the mathematical form:

space="preserve" listing-type="equation">F=kx

where F is said restoring force, x is a magnitude of a deviation of said spatial correlation including a deviation between the current location of the first simulated object and a location of said first simulated object had said mapping not been broken, and k is a spring constant parameter.

9. The method of claim 8, wherein said restoring force includes a damping force and said restoring force has the mathematical form:

space="preserve" listing-type="equation">F=kx+bv

where F is said restoring force, x is a magnitude of a deviation of said spatial correspondence including a deviation between the current location of the first simulated object and a location of said first simulated object had said mapping not been broken, v is a function of a velocity of said physical object, and k and b are constant parameters.

10. The method of claim 8, wherein said restoring force includes an intertial force and said restoring force has the mathematical form:

space="preserve" listing-type="equation">F=kx+bv+ma

where F is said restoring force, x is a magnitude of a deviation of said spatial correspondence including a deviation between the current location of the first simulated object and a location of said first simulated object had said mapping not been broken, v is a function of a velocity of said physical object, a is a function of an acceleration of said physical object, and k, b and m are constant parameters.

11. The method of claim 10, wherein said restoring force includes a component resulting from friction between said simulated object and said simulated spatial environment.

12. The method of claim 1, wherein said second simulated object can be independently moved within said simulation and on said display device.

13. The method of claim 12, wherein said second simulated object is controlled by said computer within said simulation.

14. The method of claim 12, wherein user is a first user manipulating a first physical object, and wherein said second simulated object is controlled by a second user manipulating a second physical object of a second interface device.

15. The method of claim 12, wherein said force feedback corresponds to a restoring force having a magnitude that is proportional to a degree that said mapping is broken.

16. The method of claim 15, wherein said restoring force is a spring force which is proportional to the degree to which the trajectories of said first and second simulated objects exceed the result of said interaction between said simulated objects as determined by said simulation.

17. The method of claim 16, wherein said restoring force is a spring force having the mathematical form:

space="preserve" listing-type="equation">F=k(x.sub.x +x.sub.2)

where F is said restoring force, x₁ is a magnitude of a deviation of said spatial correlation including a deviation between the current location of the first simulated object and a location of said first simulated object had said mapping not been broken, x₂ is a magnitude of a deviation of said spatial correlation including a deviation between the current location of the second simulated object and a location of said second simulated object had said mapping not been broken, and k is a spring constant parameter.

18. The method of claim 17, wherein said restoring force includes a damping force having the mathematical form:

space="preserve" listing-type="equation">F=k(x.sub.1 +x.sub.2)+b(v.sub.1 +v.sub.2)

where F is said restoring force, v₁ is a function of the velocity of said first physical object, v₂ is a function of the velocity of said second physical object, and k and b are constant parameters.

19. The method of claim 17, wherein said restoring force is determined, at least in part, utilizing a weighting factor for influencing the location of said first and second simulated objects resulting from said interaction.

20. The method of claim 19, wherein the location on said display, L of the simulated objects shown on said display is determined by the equation:

##EQU4##

21. The method of claim 15, wherein said first simulated object and said second simulated object are displayed at an equilibrium position determined, at least in part, by a position of said first physical object and a position of said second physical object.

22. The method of claim 21, wherein said equilibrium position is determined, at least in part, by a position of said first physical object, by a position of said second physical object, and by a weighting factor.

23. The method of claim 15, further comprising a step of generating a plurality of simulated objects on said display device and within said simulation, wherein each of said plurality of simulated objects is controlled by one of a plurality of users each using an associated interface device.

24. The method of claim 1, wherein said second simulated object is an obstruction object, wherein a location of said obstruction object cannot change within said simulation.

25. The method of claim 1, wherein said first simulated object is a paddle and said second simulated object is a ball object.

26. The method of claim 1, wherein said simulated environment is a sporting environment including a playfield.

27. A computer system for simulating the spatial interaction of a displayed first simulated object with a displayed second simulated object in a computer-simulated spatial environment such that the user is provided with a force feedback that realistically represents said interaction, comprising:

a processor for executing a simulation including a first simulated object, said simulation being configured to implement the motion of said first simulated object in response to motion of a physical object of an interface device controlled by a user, wherein said physical object has a physical position in a physical environment, and wherein a position control mapping between said simulated location of said first simulated object and said physical position of said physical object is created, said simulation being further configured to generate a second simulated object having boundaries such that said second simulated object impedes the simulated motion of said first simulated object when the trajectory of said first simulated object intersects said boundaries of said second simulated object;

a display device for viewing the location and motion of said first simulated object and said second simulated object; and

a force feedback mechanism configured to impart to a user of said computer system a physical sensation that corresponds to the simulated physical interaction of said first simulated object with said second simulated object when the trajectory of said first simulated object intersects the boundaries of said second simulated object, wherein said interaction is determined by examining a current location and a previous location of said first simulated object to determine said trajectory, and wherein said position control mapping between said physical object and said first simulated object is broken during said interaction when providing said physical sensation.

28. The computer system of claim 27, wherein said force feedback mechanism is configured to generate a restoring force that is proportional to the magnitude by which said mapping is broken when said first simulated object intersects said boundaries of said second simulated object.

29. The computer system of claim 28, wherein said restoring force is a spring force having the mathematical form:

space="preserve" listing-type="equation">F=kx

where F is said restoring force, x is a magnitude of a deviation of said spatial correlation including a deviation between the current location of the first simulated object and a location of said first simulated object had said mapping not been broken, and k is a spring constant parameter.

30. The method of claim 28, wherein said restoring force includes a component resulting from friction between said simulated object and said simulated spatial environment.

31. The computer system of claim 28, wherein said second simulated object can be moved within said simulation and on said display device.

32. The computer system of claim 31, wherein said second simulated object moves on said display device during said simulation in response to manipulations of a second physical object of a second interface device by said second user, said second interface device being coupled to a second computer system coupled to said computer system through a network interface.

33. The computer system of claim 31, wherein said restoring force includes a weighting factor such that the location L on said display device of the simulated objects shown on said display device is determined by the equation:

##EQU5##

34. The computer system of claim 27, wherein said user interface device is operated manually, and said display device at least partially conceals the interface device from the view of said user such that said user cannot view the user'"'"'s hands when operating said interface device.

35. A method for providing force feedback for interacting simulated objects in a simulation implemented on a computer system, the method comprising:

displaying a user-controlled first simulated object and a second simulated object on a display device of a computer system, said first simulated object moving on said display device during a simulation in response to manipulations of a physical object of an interface device by a user according to a position control paradigm, said interface device being coupled to said computer system;

determining when said first simulated object engages said second simulated object within said simulation;

displaying said determined engagement of said first simulated object with said second simulated object, wherein said first simulated object has a simulated compliance of a predetermined amount and said second object has a simulated mass of a predetermined amount wherein an elongation of said first graphical object is displayed in accordance with said predetermined simulated compliance and said predetermined simulated mass at a point of impact with said second simulated object; and

outputting a force command to said interface device to apply a force to said physical object manipulated by said user in at least one degree of freedom provided by said interface device, said force being applied in the direction of said engagement of said second simulated object with said first simulated object and having a magnitude in accordance with said simulated mass of said second simulated object.

36. A method as recited in claim 35 wherein a simulated compliance of said first simulated object additionally affects said magnitude of said force.

37. A method as recited in claim 36 wherein simulated velocities of said first simulated object and said second simulated object additionally affect said magnitude of said force.

38. A method as recited in claim 36 wherein a simulated gravity acting on said simulated objects additionally affects said magnitude of said force.

39. A method as recited in claim 35 further comprising detecting whether said user has input a command to trap said second object such that said second simulated object remains engaged with said first simulated object, thereby allowing said user to move said first simulated object and said second simulated object.

40. A method as recited in claim 39 wherein said command to trap is input by said user pressing a button of said interface device, wherein said ball is released from said engagement with said paddle when said button is released.

41. A method as recited in claim 39 wherein said first simulated object is a paddle having two endpoints and said second simulated object is a ball object, wherein said paddle and said ball object can be moved by said user when said ball object is trapped as if said paddle is a flexible sling.

42. A method as recited in claim 41 wherein when said trap command is removed while said ball object is moved by said user in an approximately circular path, said ball object is disengaged and moved tangential to said circular path away from said paddle.

43. A method as recited in claim 41 wherein when said trap command is removed while said second simulated object is moving toward said two endpoints of said paddle, said ball object remains engaged with said paddle until said ball object is moved past said two endpoints, and is then forced away from said paddle.

44. A method as recited in claim 41 wherein when said trap command is removed while said ball object is moving away from said two endpoints of said paddle, said ball object is disengaged from said paddle and is moved away from said paddle.

45. A method as recited in claim 39 wherein when said command to trap is received, said ball object and said paddle are held in a configuration that was present when said command to trap was received, and wherein forces associated with said ball object interacting with said paddle are turned off and said paddle is modeled as a rigid object.

46. A method as recited in claim 45 wherein when said command to trap is removed, a reaction force is provided to said physical object which is opposite in direction from the force that would have been provided by said simulation in the absence of said command to trap.

47. A method as recited in claim 39 wherein when said command to trap is received, forces associated with said ball object interacting with said paddle remain active and said paddle remains compliant.

48. A method as recited in claim 39 further comprising displaying a third simulated object in said simulation, said third simulated object being controlled by a second user by a second physical object and being operative to trap said second graphical object when said second user inputs a command to trap said second simulated object, and wherein said first and third simulated objects can concurrently trap said second simulated object, such that force feedback based on movement of said first simulated object is output to said second physical object and force feedback based on movement of said third simulated object is output to said first physical object when both said simulated objects trap said ball.

49. A method as recited in claim 35 further comprising displaying a simulated goal object on said display screen, wherein said user moves said first simulated object to block said second simulated object from moving into said simulated goal object.

50. A method as recited in claim 35 further comprising a displaying a third simulated object on said display device, said third simulated object moving on said display device during a simulation in response to manipulations of a second physical object of a second interface device by a second user.

51. A method as recited in claim 50 wherein said second interface device is coupled to said computer system.

52. A method as recited in claim 50 wherein said computer system is a first computer system, and wherein said second interface device is coupled to a second computer system that is linked with said first computer system.

53. A method as recited in claim 35 further comprising detecting whether said user has input a command to trap said second object such that said second simulated object remains engaged with said first simulated object, thereby allowing said user to move said first simulated object and said second simulated object, and wherein said first simulated object and said second simulated object can be moved by said user when said second simulated object is trapped as if said first simulated object is a flexible sling, wherein said second simulated object, while so trapped, is engaged to said first simulated object by a simulated spring that modifies said force on said physical object such that said magnitude of said force increases the further said second simulated object is positioned from a point on said first simulated object, and wherein when said trap command is removed, said second simulated object is disengaged from said first simulated object.

54. A method for providing an interaction between displayed objects in a graphical environment implemented by a host computer, wherein a user interfaces with said graphical environment using a force feedback device coupled to said host computer, the method comprising:

moving a first graphical object in response to movement of a user manipulatable object of said force feedback device by said user, said movement of said first graphical object provided according to a position control mapping to said movement of said user manipulatable object;

determining whether said first graphical object has engaged a second graphical object by examining a path of said first graphical object in said graphical environment, said path determined by examining a current location of said first graphical object and a previous location of said first graphical object;

breaking said position control mapping and providing an illusion of rigidity of said second graphical object by displaying said first graphical object as remaining engaged with said second graphical object when said path of said first graphical object has been determined to move through said second graphical object if said position control mapping were maintained;

outputting an opposing force on said user manipulatable object by at least one actuator in said force feedback device in a direction approximately opposite to said path of said first graphical object while said first graphical object is engaged with said second graphical object.

55. A method as recited in claim 54 wherein said opposing force is a restoring spring force.

56. A method as recited in claim 54 wherein said second graphical object is fixed in location within said graphical environment.

57. A method as recited in claim 54 wherein said user is a first user, and wherein said second graphical object is moveable according to input from a second user of a second force feedback device coupled to said host computer.

58. A method as recited in claim 57 wherein said host computer is a first host computer, and wherein said second force feedback device is coupled to a second host computer which is coupled to said first host computer via a network.

59. A method as recited in claim 58 wherein said network is the World Wide Web.

60. A method as recited in claim 54 wherein a friction force is output on said user manipulatable object when said user manipulatable object is moved in a direction corresponding to a direction approximately perpendicular to said path of engagement of said first graphical object while said first and second graphical objects are engaged.

61. A method as recited in claim 60 wherein said friction force has a magnitude that is a function of said magnitude of said opposing spring force.

62. A method as recited in claim 54 wherein said second graphical object may be moved due to said engagement with said first graphical object, wherein said second graphical object has a simulated mass and may be pushed by said first graphical object.

63. A method as recited in claim 62 wherein said opposing force is reduced in magnitude when said second graphical object is moved approximately in a direction of movement of said first graphical object.

64. A method as recited in claim 54 wherein said first and second graphical objects are objects displayed on a web page.

66. A method as recited in claim 54 wherein said second physical object is coupled to a second force feedback device coupled to said host computer.

67. A method as recited in claim 66 wherein said second force feedback device is directly coupled to said host computer.

68. A method as recited in claim 66 wherein said host computer is a first host computer, and wherein said second force feedback device is coupled to a second host computer, wherein said first host computer and said second host computer are connected via a communication link.

69. A method as recited in claim 66 wherein an opposing force is output on said second physical object by at least one actuator in said second force feedback device, and wherein said position control mapping for said second physical object is broken during said engagement.

65. A method for providing an interaction between displayed objects in a graphical environment implemented by a host computer, wherein a user interfaces with said graphical environment by manipulating a force feedback device coupled to said host computer, the method comprising:

moving a displayed first graphical object in response to movement of a first physical object of said force feedback device, said movement of said first graphical object provided according to a position control mapping to said movement of said first physical object;

moving a displayed second graphical object in response to movement of a second physical object, said movement of said second graphical object provided according to a position control mapping to said movement of said second physical object;

determining whether said first graphical object has engaged said second graphical object;

breaking said position control mapping for said first physical object and providing an illusion of rigidity of said engagement by displaying said first graphical object as remaining engaged with said second graphical object when said first graphical object and second graphical objects have been determined to move through each other to continue their motion if said position control mapping with said physical objects were maintained; and

outputting an opposing force on said first physical object by at least one actuator in said force feedback device in a direction opposite to a direction of said motion of said first graphical object while said first graphical object is engaged with said second graphical object.

70. A method as recited in claim 65 wherein said engagement of said first and second graphical objects is determined by examining a path of said first graphical object and a path od said graphical object in said graphical environment.

71. A method as recited in claim 70 wherein said paths are determined by examining a current location and a previous location of said graphical objects.