Systems and methods for providing haptic feedback for remote interactions

• US 10,613,627 B2
• Filed: 05/12/2014
• Issued: 04/07/2020
• Est. Priority Date: 05/12/2014
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Levesque, Vincent, Modarres, Ali, Grant, Danny A., Dionne, Jean Francois, Birnbaum, David M.
• Primary Examiner(s): Mishler, Robin J
• Application Number:
  US14/275,216
• Publication Number:
  US 20150323993A1
• Time in Patent Office: 2,157 Days
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system includes a first electronic device and a second electronic device. The first electronic device includes a sensor configured to sense or property experienced by the first electronic device, and a transmitter configured to transmit a signal based on output from the sensor. The second electronic device is in signal communication with the first electronic device. The second electronic device includes a receiver configured to receive the transmitted signal, a detector configured to determine an object that a user of the second device is focusing on, a processor configured to generate a haptic signal representative of the transmitted signal if it is determined that the object the user is focusing on corresponds with a location of the first electronic device, and a haptic output device configured to receive the haptic signal and generate a haptic effect to the user.

Claims

1. A system comprising:

an electronic device comprising;

an eye-gaze sensor;

a wireless receiver;

one or more processors configured to;

receive one or more eye-gaze sensor signals from the eye-gaze sensor;

determine a focus object based on the one or more eye-gaze sensor signals, the focus object being a real-world object that a user of the electronic device is determined to be focused on;

receive, using the wireless receiver, one or more wireless signals from a source electronic device indicating a sensed condition or a sensed property of the focus object, the source electronic device corresponding to the focus object and configured to sense the sensed condition or the sensed property of the focus object;

generate a haptic signal based on the condition or property of the focus object; and

a haptic output device configured to receive the haptic signal and generate a haptic effect to the user.

2. The system according to claim 1, wherein the eye-gaze sensor comprises a camera configured to capture an image of an eye of the user, and an image processor configured to determine a direction of an eye gaze of the user based on the image.

3. The system according to claim 1, wherein the source electronic device is configured to sense vibrations or deformations of the focus object and wherein the sensed condition or the sensed property indicate the sensed vibrations or deformations.

4. The system according to claim 3, wherein the haptic effect simulates the sensed vibrations or deformations.

5. The system according to claim 1, wherein the source electronic device further comprises a camera configured to output a video stream, the video stream being part of the signal, and wherein the electronic device further comprises a display configured to display the video stream.

6. The system according to claim 1, wherein the electronic device is a head-mounted display.

7. The system according to claim 6, wherein the eye-gaze sensor comprises a camera mounted on the head-mounted display, the camera having a field of view aligned with a direction the head-mounted display is pointing.

8. A method comprising:

receiving, by an electronic device and from an eye-gaze sensor, one or more eye-gaze sensor signals from the eye-gaze sensor;

determining, by the electronic device, a focus object based on the one or more eye-gaze sensor signals, the focus object being a real-world object that a user of the electronic device is determined to be focused on;

receiving, by the electronic device and using a wireless receiver, one or more wireless signals indicating a sensed condition or sensed property experienced by the focus object from a source electronic device, the source electronic device corresponding to the focus object and configured to sense the sensed condition or the sensed property of the focus object and to transmit the one or more wireless signals;

generating, by the electronic device, a haptic signal based on the sensed condition or the sensed property of the focus object; and

transmitting the haptic signal to a haptic output device to output a haptic effect.

9. The method according to claim 8, wherein the eye-gaze sensor comprises a camera and wherein determining the object that the user of the electronic device is focusing on comprises:

capturing, by the camera, an image of an eye of the user; and

determining a direction of an eye gaze of the user based on the image.

10. The method according to claim 8, wherein the source electronic device is configured to sense vibrations or deformations of the focus object, and wherein the sensed condition or the sensed property indicate the sensed vibrations or deformations.

11. The method according to claim 10, wherein the haptic effect simulates the sensed vibrations or deformations.

12. The method according to claim 8, further comprising:

receiving, from a camera of the source electronic device, a video stream, wherein the one or more wireless signals include the video stream; and

displaying, by a display of the electronic device, the video stream.

13. The method according to claim 8, wherein the electronic device is a head-mounted display.

14. The method according to claim 13, wherein the eye-gaze sensor comprises a camera mounted on the head-mounted display, the camera having a field of view aligned with a direction the head-mounted display is pointing.

15. A non-transitory computer-readable medium storing program code executable by a processor, the program code configured to cause the processor to:

receive, by an electronic device and from an eye-gaze sensor, one or more eye-gaze sensor signals from the eye-gaze sensor;

determine a focus object based on the one or more eye-gaze sensor signals, the focus object being a real-world object that a user of the electronic device is determined to be focused on;

receive, by the electronic device and using a wireless receiver, one or more wireless signals indicating a sensed condition or sensed property experienced by the focus object from a source electronic device, the source electronic device corresponding to the focus object and configured to sense the sensed condition or the sensed property of the focus object and to transmit the one or more wireless signals;

generate a haptic signal based on the sensed condition or the sensed property of the focus object; and

transmit the haptic signal to a haptic output device to output a haptic effect.

16. The non-transitory computer-readable medium according to claim 15, wherein the eye-gaze sensor comprises a camera and wherein the program code is further configured to cause the processor to:

capture, using the camera, an image of an eye of the user; and

determine a direction of an eye gaze of the user based on the image.

17. The non-transitory computer-readable medium according to claim 15, wherein the source electronic device is configured to sense vibrations or deformations of the focus object, and wherein the sensed condition or the sensed property indicate the sensed vibrations or deformations.

18. The non-transitory computer-readable medium according to claim 17, wherein the haptic effect is configured to simulate the sensed vibrations or deformations.

19. The non-transitory computer-readable medium according to claim 15, wherein the program code is further configured to cause the processor to:

receive a video stream from the source electronic device, wherein the one or more wireless signals include the video stream; and

cause a display of the video stream.

20. The non-transitory computer-readable medium according to claim 15, wherein the electronic device is a head-mounted display.

21. The non-transitory computer-readable medium according to claim 20, wherein the eye-gaze sensor comprises a camera mounted on the head-mounted display, the camera having a field of view aligned with a direction the head-mounted display is pointing.