Systems and methods to synchronize visual effects and haptic feedback for interactive experiences

• US 10,444,840 B2
• Filed: 08/30/2017
• Issued: 10/15/2019
• Est. Priority Date: 08/30/2017
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Nocon, Nathan
• Primary Examiner(s): Pham, Viet D
• Application Number:
  US15/691,307
• Publication Number:
  US 20190064924A1
• Time in Patent Office: 776 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2370/10 Use of a protocol of communication by packets in interfaces along the display data pipeline
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information

Abstract

This disclosure presents systems and methods to synchronize visual effects and haptic feedback for interactive experiences. Visual circuit control signals may dictate control of a presentation device to present images forming virtual content. Haptic circuit control signals may dictate control of a haptic feedback device to provide haptic output. Timing information conveying one or both of latency in the execution of the visual circuit control signals by the presentation device and/or latency in the execution of the haptic circuit control signals by the haptic feedback device may be determined. Based on the timing information, transmission of the visual circuit control signals to the presentation device and/or transmission of the haptic circuit control signals to the haptic feedback device may be effectuated so that the presentation of images and provision of haptic output occur within a threshold time span.

Claims

1. A system configured to synchronize visual effects and haptic feedback for an interactive experience, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain visual circuit control signals, the visual circuit control signals dictating control of a presentation device to present images forming virtual content, wherein the presentation device is arranged on a headset such that when the headset is installed on a user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images, wherein execution of the visual circuit control signals by the presentation device causes the presentation device to present the images at a first point in time;

obtain haptic circuit control signals, the haptic circuit control signals dictating control of a haptic feedback device to provide haptic output, wherein execution of the haptic circuit control signals by the haptic feedback device causes the haptic feedback device to provide the haptic output at a second point in time;

determine timing information conveying one or both of latency in the execution of the visual circuit control signals by the presentation device and/or latency in the execution of the haptic circuit control signals by the haptic feedback device;

based on the timing information, effectuate transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the first point in time and the second point in time occur within a threshold time span;

wherein determining the timing information comprises;

determining a prior relative time delay between prior executions of the visual circuit control signals by the presentation device and prior executions of the haptic circuit control signals by the haptic feedback device;

based on the prior relative time delay, effectuate current transmissions of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that a current relative time delay between a current execution of the visual circuit control signals by the presentation device and a current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span;

based on the prior relative time delay being greater than the threshold time span and indicating the prior executions of the visual circuit control signals by the presentation device occur relatively sooner than the prior executions of the haptic circuit control signals by the haptic feedback device,

delay the transmission of the current transmissions of the visual circuit control signals to the presentation device so that the current relative time delay between the current execution of the visual circuit control signals by the presentation device and the current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span; and

based on the prior relative time delay being greater than the threshold time span and indicating the prior executions of the visual circuit control signals by the presentation device occur relatively later than the prior executions of the haptic circuit control signals by the haptic feedback device,

delay the transmission of the current transmissions of the haptic circuit control signals to the haptic feedback device so that the current relative time delay between the current execution of the visual circuit control signals by the presentation device and the current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span.

2. The system of claim 1, wherein the prior relative time delay is an immediately prior relative time delay.

3. The system of claim 2, wherein the prior relative time delay is an aggregate prior time delay.

4. The system of claim 1, wherein the presentation device is arranged on the headset such that when the headset is installed on the user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images and the real world to create an augmented reality environment.

5. A system configured to synchronize visual effects and haptic feedback for an interactive experience, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain visual circuit control signals, the visual circuit control signals dictating control of a presentation device to present images forming virtual content, wherein the presentation device is arranged on a headset such that when the headset is installed on a user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images, wherein execution of the visual circuit control signals by the presentation device causes the presentation device to present the images at a first point in time;

obtain haptic circuit control signals, the haptic circuit control signals dictating control of a haptic feedback device to provide haptic output, wherein execution of the haptic circuit control signals by the haptic feedback device causes the haptic feedback device to provide the haptic output at a second point in time;

determine timing information conveying one or both of latency in the execution of the visual circuit control signals by the presentation device and/or latency in the execution of the haptic circuit control signals by the haptic feedback device;

based on the timing information, effectuate transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the first point in time and the second point in time occur within a threshold time span;

wherein determining the timing information comprises;

determining a value of a visual circuit latency parameter, the value of the visual circuit latency parameter specifying a first time delay between points in time when a transmission of the visual circuit control signals is initiated and points in time when the visual circuit control signals are executed by the presentation device; and

determining a value of a haptic circuit latency parameter, the value of the haptic circuit latency parameter specifying a second time delay between points in time when a transmission of the haptic circuit control signals is initiated and points in time when the haptic circuit control signals are executed by the haptic feedback device;

compare the value of the visual circuit latency parameter with the value of the haptic circuit latency parameter;

based on the comparison, effectuate transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the points in time when the visual circuit control signals are executed by the presentation device and the points in time when the haptic circuit control signals are executed by the haptic feedback device occur within the threshold time span;

determine the value of the visual circuit latency parameter and the value of the haptic circuit latency parameter associated with a current transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device; and

for a subsequent transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device;

based on the comparison indicating that the first time delay in the current transmission is greater than the second time delay, delay the subsequent transmission of the haptic circuit control signals to the haptic feedback device a first amount of time so that the second time delay is substantially the same as the first time delay; and

based on the comparison indicating that the second time delay in the current transmission is greater than the first time delay, delay the subsequent transmissions of the visual circuit control signals to the presentation device a second amount of time so that the first time delay is substantially the same as the second time delay.

6. The system of claim 5, wherein the one or more physical processors are further configured by the machine-readable instructions to:

determine the first amount of time based on a difference between the value of the visual circuit latency parameter for the current transmission and the value of the haptic circuit latency parameter associated for the current transmission; and

determine the second amount of time based on a difference between the value of the haptic circuit parameter for the current transmission and the value of the latency visual circuit latency parameter associated for the current transmission.

7. A method to synchronize visual effects and haptic feedback for an interactive experience, the method comprising:

obtaining visual circuit control signals, the visual circuit control signals dictating control of a presentation device to present images forming virtual content, wherein the presentation device is arranged on a headset such that when the headset is installed on a user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images, wherein execution of the visual circuit control signals by the presentation device causes the presentation device to present the images at a first point in time;

obtaining haptic circuit control signals, the haptic circuit control signals dictating control of a haptic feedback device to provide haptic output, wherein execution of the haptic circuit control signals by the haptic feedback device causes the haptic feedback device to provide the haptic output at a second point in time;

determining timing information conveying one or both of latency in the execution of the visual circuit control signals by the presentation device and/or latency in the execution of the haptic circuit control signals by the haptic feedback device;

based on the timing information, effectuating transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the first point in time and the second point in time occur within a threshold time span;

wherein determining the timing information comprises;

determining a prior relative time delay between prior executions of the visual circuit control signals by the presentation device and prior executions of the haptic circuit control signals by the haptic feedback device;

based on the prior relative time delay, effectuating current transmissions of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that a current relative time delay between a current execution of the visual circuit control signals by the presentation device and a current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span;

based on the prior relative time delay being greater than the threshold time span and indicating the prior executions of the visual circuit control signals by the presentation device occur relatively sooner than the prior executions of the haptic circuit control signals by the haptic feedback device,

delaying the transmission of the current transmissions of the visual circuit control signals to the presentation device so that the current relative time delay between the current execution of the visual circuit control signals by the presentation device and the current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span; and

based on the prior relative time delay being greater than the threshold time span and indicating the prior executions of the visual circuit control signals by the presentation device occur relatively later than the prior executions of the haptic circuit control signals by the haptic feedback device,

delaying the transmission of the current transmissions of the haptic circuit control signals to the haptic feedback device so that the current relative time delay between the current execution of the visual circuit control signals by the presentation device and the current execution of the haptic circuit control signals by the haptic feedback device is less than or equal to the threshold time span.

8. The method of claim 7, wherein the prior relative time delay is an immediately prior relative time delay.

9. The method of claim 8, wherein the prior relative time delay is an aggregate prior time delay.

10. The method of claim 7, wherein the presentation device is arranged on the headset such that when the headset is installed on the user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images and the real world to create an augmented reality environment.

11. A method to synchronize visual effects and haptic feedback for an interactive experience, the method comprising:

obtaining visual circuit control signals, the visual circuit control signals dictating control of a presentation device to present images forming virtual content, wherein the presentation device is arranged on a headset such that when the headset is installed on a user'"'"'s head, the user'"'"'s gaze is directed towards the presentation device to view the images, wherein execution of the visual circuit control signals by the presentation device causes the presentation device to present the images at a first point in time;

obtaining haptic circuit control signals, the haptic circuit control signals dictating control of a haptic feedback device to provide haptic output, wherein execution of the haptic circuit control signals by the haptic feedback device causes the haptic feedback device to provide the haptic output at a second point in time;

determining timing information conveying one or both of latency in the execution of the visual circuit control signals by the presentation device and/or latency in the execution of the haptic circuit control signals by the haptic feedback device;

based on the timing information, effectuating transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the first point in time and the second point in time occur within a threshold time span;

determining the timing information comprises;

determining a value of a visual circuit latency parameter, the value of the visual circuit latency parameter specifying a first time delay between points in time when a transmission of the visual circuit control signals is initiated and points in time when the visual circuit control signals are executed by the presentation device; and

determining a value of a haptic circuit latency parameter, the value of the haptic circuit latency parameter specifying a second time delay between points in time when a transmission of the haptic circuit control signals is initiated and points in time when the haptic circuit control signals are executed by the haptic feedback device; and

comparing the value of the visual circuit latency parameter with the value of the haptic circuit latency parameter;

based on the comparison, effectuating transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device so that the points in time when the visual circuit control signals are executed by the presentation device and the points in time when the haptic circuit control signals are executed by the haptic feedback device occur within the threshold time span;

determining the value of the visual circuit latency parameter and the value of the haptic circuit latency parameter associated with a current transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device; and

for a subsequent transmission of the visual circuit control signals to the presentation device and the haptic circuit control signals to the haptic feedback device;

based on the comparison indicating that the first time delay in the current transmission is greater than the second time delay, delaying the subsequent transmission of the haptic circuit control signals to the haptic feedback device a first amount of time so that the second time delay is substantially the same as the first time delay; and

based on the comparison indicating that the second time delay in the current transmission is greater than the first time delay, delaying the subsequent transmissions of the visual circuit control signals to the presentation device a second amount of time so that the first time delay is substantially the same as the second time delay.

12. The method of claim 11, further comprising:

determining the first amount of time based on a difference between the value of the visual circuit latency parameter for the current transmission and the value of the haptic circuit latency parameter associated for the current transmission; and

determining the second amount of time based on a difference between the value of the haptic circuit parameter for the current transmission and the value of the latency visual circuit latency parameter associated for the current transmission.