Networked applications including haptic feedback

• US 8,508,469 B1
• Filed: 09/16/1998
• Issued: 08/13/2013
• Est. Priority Date: 12/01/1995
• Status: Expired due to Fees

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Rosenberg, Louis B., Brave, Scott B., Tan, Sian W.
• Primary Examiner(s): Liang, Regina
• Application Number:
  US09/153,781
• Time in Patent Office: 5,445 Days
• Field of Search: 345156-158, 345/161, 345/701, 345/702, 273/148.B, 709/203, 709/217, 709/227, 709/250, 709/218, 709/230, 715/701, 715/702, 463 30- 32, 463 36- 42
• US Class Current: 345/156
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G05B 2219/40132 Haptic joystick with force feedback based on accelerometer included in joystick
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 2009/04777 with additional push or pull action on the handle
• CPC Class Codes: G05G 9/047 the controlling member being movable by hand about orthogonal axes, e.g. joysticks for switches H01H25/04
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: H01H 2003/008 : with a haptic or a tactile feedback controlled by electrical means, e.g. a motor or magnetofriction

Abstract

Method and apparatus for providing peer-to-peer force feedback over a computer network. A network force feedback system includes a network, a first computer coupled to the network, and a second computer coupled to the network. The first and second computers each include a visual display and a force feedback interface device. Each computer provides a force feedback signal to its force feedback device based on information received from the other, remote computer and in some cases also based on input from the local force feedback device. Positional information of each force feedback device and/or feel sensation information can be transmitted between the computers over the network. A graphical environment can be displayed to enhance the interaction between users. The present invention therefore permits two computer users to interact using force feedback provided over a network on a peer-to-peer basis.

Claims

1. A system comprising:

a first electronic device coupled to a network; and

a second electronic device coupled to said network, said second electronic device remote from said first electronic device, said second electronic device configured to produce a graphical environment, wherein said graphical environment is based, at least in part, on information transferred from said first electronic device to said second electronic device over said network, and

a human/computer interface, wherein said human/computer interface comprises an actuator means, said second electronic device further configured to interpret force information repeatedly received from said first electronic device over said network, update said graphical environment based, at least in part, on said information, and cause said actuator to generate a physical feel sensation at said human/computer interface based, at least in part, on said force information.

2. A system as recited in claim 1 wherein said second electronic device input comprises at least one of a position input from said human/computer interface device, or a button click input.

3. A system as recited in claim 1 wherein said human/computer interface coupled to said second electronic device includes a local controller that communicates with said second electronic device, and a plurality of said actuator means for providing said physical feel sensations.

4. A system as recited in claim 3 wherein said second electronic device sends a force feedback command to said local controller that can be parsed by said local controller such that said local controller can control said actuator means in response to said force feedback command in a control loop.

5. A method for providing haptic feedback, comprising:

receiving a remote computer information from a first computer at a second computer over a network, wherein said first computer is remote from said second computer, and wherein said remote computer information comprises information representing force information,

generating a graphical environment;

receiving an input information at said second computer from a haptic feedback device; and

causing a force signal to be provided to said haptic feedback device from said second computer, said force signal being based, at least in part, on said remote computer information and said input information, wherein said force signal causes said haptic feedback device to output a force.

6. A method as recited in claim 5 further comprising sending second computer information from said second computer to said first computer over said network, said second computer information comprising information representing force information.

7. A method as recited in claim 6 wherein said second computer information includes said input information from said second haptic feedback device and a tactile sensation to be output by said first haptic feedback device.

8. A method as recited in claim 5 wherein said graphical environment includes a first graphical object controlled by a user of said first haptic feedback device, and a second graphical object controlled by a user of said second haptic feedback device.

9. A method as recited in claim 8 wherein said first and second graphical objects are paddles.

10. A method as recited in claim 8 wherein said first and second graphical objects are displayed in a web page.

11. A method as recited in claim 5 wherein said second haptic feedback device includes a local controller that communicates with said second computer, wherein said local controller parses a haptic feedback command sent by said second computer such that said local controller can control said actuator in response to said haptic feedback command in a control loop with at least one sensor of said second haptic feedback device.

12. A method for providing haptic feedback between a first computer and a second computer comprising:

sending a first computer information to said second computer over a network, wherein said first computer information comprises force information, wherein said first computer is remote to said second computer;

causing a first haptic feedback signal to be sent to a second haptic feedback device from said second computer, said first haptic feedback signal being based, at least in part, on said force information, wherein said first haptic feedback signal causes said second haptic feedback device to output a haptic sensation;

sending a second computer information to said first computer over said network, wherein said second computer information comprises a position of a manipulandum of a second haptic feedback device; and

causing a second haptic feedback signal to be sent to said first haptic feedback device from said first computer, said second haptic feedback signal being based, at least in part, on said second computer information, wherein said haptic feedback signal causes said first haptic feedback device to output a haptic sensation.

13. A method as recited in claim 12 wherein said first computer receives a first input information from said first haptic feedback device in response to a manipulation of said first haptic feedback device, and wherein said second computer receives a second input information from said second haptic feedback device in response to a manipulation of said second haptic feedback device.

14. A method as recited in claim 13 wherein said force information from said first computer and said second computer is based, at least in part, on said input information from said first haptic feedback device and said second haptic feedback device, respectively.

15. A method as recited in claim 12 wherein said first computer information includes force information indicating a tactile sensation to be output by said second haptic feedback device, and wherein said second computer information includes force information indicating a tactile sensation to be output by said second haptic feedback device.

16. A method as recited in claim 15 wherein said first computer and said second computer each produce a graphical environment having a first graphical object controlled by a first user and a second graphical object controlled by a second user.

17. A method as recited in claim 12 further comprising accessing a server computer with at least one of said first and second computers and downloading a feel sensation information from said server computer, said feel sensation information to be included in said first computer information or said second computer information.

18. A method as recited in claim 17 wherein said server computer provides a web page to said respective computer accessing said server, said web page including an embedded feel sensation information.

19. A method for providing haptic feedback comprising:

receiving a first computer information from a first computer at a server computer over a network;

providing said first computer information to a second computer, wherein said first computer is remote from said second computer, wherein said first computer information comprises first force information and information operable to update a simulated graphical object in a graphical environment output by said second computer, and wherein at least one of said second computer and said server computer uses said first computer information to update a game program running on at least one of said second computer and said server computer, and wherein said second computer provides a second haptic feedback signal based at least in part on said first force information to a second haptic feedback device;

receiving a second computer information from said second computer over said network; and

providing said second computer information to said first computer, wherein said second computer information comprises second force information and information operable to update a simulated graphical object in a graphical environment output by said first computer, and wherein at least one of said first computer and said server computer uses said second computer information to update a game program running on at least one of said first computer and said server computer, and wherein said first computer provides a first haptic feedback signal based at least in part on said second force information to a first haptic feedback device.

20. A method as recited in claim 19 wherein said first force information describes a tactile sensation, wherein said tactile sensation is output by said second haptic feedback device.

21. A method as recited in claim 19 further comprising sending tactile sensation data stored on said server computer to said first computer.

22. A method as recited in claim 19 wherein said first computer information comprises position data allowing said second computer to display a graphical object in said graphical environment output by said second computer.

23. A method as recited in claim 19 wherein said server computer runs a web page.

24. A method as recited in claim 19 wherein updating said game program running on said first computer includes updating a location of a displayed player graphical object based at least in part on said second computer information.

25. A method as recited in claim 24 wherein said displayed player graphical object represents a sporting object.

26. A method as recited in claim 25 wherein said displayed player graphical object includes a weapon.

27. A method as recited in claim 24 wherein a collision between said player graphical object and a different graphical object is detected, and wherein said first haptic feedback signal is based at least in part on said detected collision.

28. A method as recited in claim 27 wherein said different graphical object is a projectile.

29. A method as recited in claim 27 wherein said different graphical object is an obstruction in said game environment.

30. A method as recited in claim 19 wherein said updating of said game program running on said first computer includes updating a location of a projectile.

31. A method as recited in claim 30 wherein said projectile is a ball or a puck.

32. A method for providing haptic feedback over a computer network comprising:

receiving a first information from a remote computer over a network, said first information comprising force information and position information for a graphical object displayed by said remote computer;

using said first information to update a visual display

providing a haptic feedback signal based at least in part on said force information to a haptic feedback device, wherein said haptic feedback device outputs a tactile sensation based, at least in part, on said haptic feedback signal and correlated with said updated visual display; and

sending a second information to said remote computer over said network.

33. A method as recited in claim 32 wherein said first computer is a client computer and said second computer is a server computer.

34. A method as recited in claim 32 wherein said first computer and said second computer are client computers.

35. A method as recited in claim 34 wherein said haptic feedback device is a first haptic feedback device, and wherein said remote computer includes a second haptic feedback device providing computer-controlled physical tactile sensations to a user of said second haptic feedback device.

36. A method as recited in claim 32 wherein said first information received from said second computer includes web page information.

37. A method as recited in claim 32 wherein said visual display is updated by moving a graphical object within a graphical game environment based, at least in part, on position data received from said haptic feedback device, where a collision between said graphical object and a different graphical object can be detected to cause said tactile sensation to be output.

38. A method as recited in claim 32 wherein said first information comprises an indication of a gaming event, and further comprising synchronizing said visual display associated with said gaming event with said tactile sensation that is associated with said gaming event.

39. A method as recited in claim 38 wherein said gaming event is a collision.

40. A method as recited in claim 38 wherein said gaming event is an explosion.

41. A method as recited in claim 38 wherein said visual display is updated at a rate substantially faster than said tactile sensation.

42. A method comprising:

executing a first local model of a computer-gaming simulation on a first computer in communication with a network;

executing, substantially simultaneously with said first local model, a second local model of said computer-gaming simulation on a second computer in communication with said first computer over said network, said second computer remote from said first computer;

updating a location of a first graphical object of said first local model based at least in part on position data output by a sensor in communication with a haptic input device in communication with said first computer, said haptic input device comprising an actuator configured to output haptic feedback to said haptic input device;

updating a location of a second graphical object based at least in part on information received over said Internet from said second network interface of said second computer, said information comprising a gaming event; and

determining, by said first computer, whether said first graphical object and said second graphical object interact, and, if so;

determining a haptic effect to be output based at least in part on force information received from said second computer, and

outputting said haptic effect to said haptic input device, said haptic effect configured to be substantially synchronized with said gaming event.

43. A method as recited in claim 42 wherein said local model of said particular client computer also receives button data from said associated haptic feedback device, said button data describing a state of at least one button on said associated haptic feedback device.

44. A method as recited in claim 42 wherein said first graphical object is a representation of sporting equipment.

45. A method as recited in claim 44 wherein said second graphical object is a representation of a ball or puck.

46. A method as recited in claim 42 wherein said first graphical object includes a representation of a weapon.

47. A method as recited in claim 42 wherein each of said local models of said computer-gaming simulation of said multiple client computers displays a graphical object having a location influenced by position data received from an associated interface device in communication with each client computer.

48. A method as recited in claim 42 wherein a sound is associated with an event occurring in said computer-gaming simulation, wherein said computer synchronizes an output of said sound with said tactile sensation that is associated with said event.

49. A method as recited in claim 48 wherein said event is a collision in said computer-gaming simulation.

50. A method as recited in claim 48 wherein said event is an explosion in said computer-gaming simulation.

51. A system comprising:

a first computer, said first computer comprising;

a first processor in communication with a network,

a first memory coupled to said first processor,

a first force feedback device in communication with said first processor, said first force feedback device configured to provide a first input signal, said first force feedback device coupled to a first actuator, said first actuator configured to provide tactile sensations in response to a first haptic feedback signal, and

wherein said first processor is configured to;

produce a first image, and

provide said first haptic feedback signal to said first force feedback device, said first image and said first haptic feedback signal based at least in part on a first force information received from a second computer over said network and based at least in part on said first input signal; and

said second computer remote from said first computer, said second computer comprising;

a second processor,

in communication with a network,

a second memory coupled to said second processor,

a second force feedback device coupled to said second processor, said second force feedback device configured to provide a second input signal, said second force feedback device coupled to a second actuator, said second actuator configured to provide a tactile sensation in response to a second haptic feedback signal, and

wherein said second processor is configured to;

produce a second image, and

provide said second haptic feedback signal to said second interface device, said second image and said second haptic feedback signal based at least in part on a second force information received from said first computer over said network and based at least in part on said second input signal.

52. A system comprising:

a first computer, comprising;

a first processor capable of generating a first image signal, said first processor in communication with a network,

a first force feedback device capable of providing a first input signal, comprising;

a first actuator configured to provide tactile sensations in response to a first haptic feedback signal, and

said first image signal and said first haptic feedback signal based at least in part on a first force information received from a second computer over said network and based at least in part on said first input signal; and

said second computer remote from said first computer, said second computer, comprising;

a second processor capable of generating a second image signal, said second processor in communication with said network, a second force feedback device capable of providing a second input signal to said second processor, comprising;

a second actuator configured to provide tactile sensations in response to a second haptic feedback signal, and

said second image and said second haptic feedback signal based at least in part on a second force information received from said first computer over said network and based at least in part on said second input signal.

53. A system as recited in claim 52 wherein said first force feedback device is coupled to a manipulandum configured to move in two degrees of freedom.

54. A system as recited in claim 53 wherein said first force feedback device is coupled to a third processor, said third processor in communication with said first processor, and said first force feedback device includes a local controller that communicates with said first computer, a plurality of actuators for providing said tactile sensations, and at least one sensor for sensing positions of said manipulandum.

55. A system as recited in claim 54 wherein said haptic feedback signal includes a haptic feedback command that can be parsed by said local controller such that said controller can control said actuators in response to said haptic feedback command in a control loop.

56. A system as recited in claim 53 wherein said manipulandum is manipulable by a finger of a user.

57. A system as recited in claim 52 wherein said first computer and said second computer communicate with at least one server computer over said network, wherein said second force information received from said first computer and said first force information received from said second computer are communicated via said server.

58. A system as recited in claim 52 wherein said first image includes a graphical object that can interact with a projectile.

59. A system as recited in claim 52 wherein said first image comprises a first graphical object and a second graphical object, said first graphical object having a location based, at least in part, on a position information received from said first force feedback device, said first graphical object able to collide with said second graphical object said second graphical object having a location based at least in part on first information received from said second computer.

60. A system as recited in claim 52 wherein said first image includes a graphical object having a location based, at least in part, on position information received from first second force feedback device, said graphical object able to collide with an obstruction displayed in said first image.

61. A device comprising:

a processor configured to;

receive a first information from a remote processor over a network, said first information comprising force information and position information for a graphical object to be displayed,

update a visual display coupled to said processor based at least in part on said first information;

generate a haptic feedback signal based at least in part on said force information, said haptic feedback signal configured to cause a haptic feedback device to output a tactile sensation based at least in part on said haptic feedback signal and correlated with said updated visual display; and

transmit a second information over said computer network.

62. A device as recited in claim 61 further comprising a visual display coupled to said processor, said visual display configured to display a first graphical object based at least in part on said first information.

63. A device as recited in claim 61 wherein said first information is received from a server computer over said network.

64. A device as recited in claim 63 wherein said server computer and said processor communicate over said network using TCP/IP protocols.

65. A device as recited in claim 61 wherein said first information is received from a client machine over said network.

66. A non-transitory computer-readable medium comprising program code to cause a processor to perform the steps of:

receive a first information from a remote processor over a network, said first information comprising force information and position information for a graphical object displayed by said second computer;

update a visual display based at least in part on said first information;

provide a haptic feedback signal based at least in part on said force information to a haptic feedback device, wherein said haptic feedback device outputs a tactile sensation based, at least in part, on said haptic feedback signal and correlated with said updated visual display; and

send a second information over said network.