Apparatus, system, and method for mitigating motion-to-photon latency in head-mounted displays

• US 10,706,813 B1
• Filed: 06/12/2018
• Issued: 07/07/2020
• Est. Priority Date: 02/03/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Fruchter, Vlad, Meitav, Ohad, Vakrat, David
• Primary Examiner(s): Wu, Sing-Wai
• Application Number:
  US16/006,438
• Time in Patent Office: 756 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G09G 2310/08 Details of timing specific for flat panels, other than clock recovery
• CPC Class Codes: G09G 2320/0252 Improving the response speed
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/397 Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay G09G5/02 takes precedence

Abstract

A special-purpose hardware device for achieving intraframe image processing in head-mounted displays may include an image-correction component that receives a single image frame destined to be displayed for presentation to a user wearing a head-mounted-display system. The image-correction component may also receive user-motion data indicating that the user wearing the head-mounted-display system has made at least one movement since generation of the single image frame. The image-correction component may further perform hardware-accelerated intraframe processing operations on the single image frame in accordance with the user-motion data to compensate for the movement made by the user on a line-by-line basis. In addition, the special-purpose hardware device may include a frame-output interface that feeds the single image frame to a display device of the head-mounted-display system to facilitate displaying the single image frame for presentation to the user. Various other apparatuses, systems, and methods are also disclosed.

Claims

1. An Application Specific Integrated Circuit (ASIC) comprising:

a hardware-accelerated image-correction component that;

receives an image frame to be displayed on a head-mounted-display system,

wherein the image frame comprises;

a subset of lines; and

an additional subset of lines;

 

receives, from a motion-detection sensor operating in conjunction with the head-mounted-display system, motion data indicating that the head-mounted-display system has moved since generation of the image frame, wherein the motion data is received after the subset of lines have completed processing by the hardware-accelerated image-correction component and before the additional subset of lines have completed processing by the hardware-accelerated image-correction component;

 

performs one or more hardware-accelerated intraframe processing operations on the additional subset of lines by modifying the additional subset of lines to compensate for the movement of the head-mounted-display system in accordance with the motion data even though the subset of lines are not eligible for the hardware-accelerated intraframe processing operations due at least in part to having completed processing when the motion data is received; and

a hardware frame-output interface communicatively coupled to the hardware-accelerated image-correction component, wherein the hardware frame-output interface, upon completion of the hardware-accelerated intraframe processing operations, feeds the additional subset of lines to a display device of the head-mounted-display system to facilitate displaying the additional subset of lines.

2. The ASIC of claim 1, further comprising a display imaging pipeline that:

incorporates the hardware-accelerated image-correction component; and

operates on a clock signal that is;

generated by a clock generator included in the head-mounted-display system; and

common to both the display imaging pipeline and the display device of the head-mounted-display system;

wherein the clock signal synchronizes the display imaging pipeline and the display device with respect to one another.

3. The ASIC of claim 2, wherein the clock signal enables the display imaging pipeline to forgo at least one handshake-based synchronization operation with the display device.

4. The ASIC of claim 3, wherein the clock signal enables the hardware-accelerated image-correction component to perform the hardware-accelerated intraframe processing operations on the image frame on a line-by-line basis due at least in part to the display imaging pipeline forgoing the handshake-based synchronization operation with the display device.

5. The ASIC of claim 1, wherein the hardware frame-output interface forwards the additional subset of lines to the display device to facilitate displaying the additional subset of lines together with the subset of lines.

6. The ASIC of claim 5, wherein:

the subset of lines have not been modified to compensate for the movement of the head-mounted-display system since the generation of the image frame; and

the display device simultaneously displays, during at least one point in time, the subset of modified lines and the additional subset of lines.

7. The ASIC of claim 1, wherein the image frame comprises at least one of:

a photographic frame captured by a camera device operating in conjunction with the head-mounted-display system;

a virtual-reality frame that includes computer-generated imagery; and

a mixed-reality frame that includes both photographic content and computer-generated imagery.

8. The ASIC of claim 1, wherein the hardware-accelerated intraframe processing operations comprise at least one of:

asynchronous time warping; and

asynchronous space warping.

9. The ASIC of claim 1, wherein the image frame comprises a mixed-reality frame that includes both photographic content and computer-generated imagery;

and

further comprising a blending component communicatively coupled to the hardware-accelerated image-correction component, wherein the blending component generates the mixed-reality frame by overlaying computer-generated imagery onto a photographic frame captured by a camera device operating in conjunction with the head-mounted-display system.

10. The ASIC of claim 9, further comprising a processing component communicatively coupled to the blending component by way of a direct communication path that enables the photographic frame captured by the camera device to be fed directly to the blending component without being buffered in memory.

11. The ASIC of claim 10, wherein the processing component comprises at least one of:

the hardware-accelerated image-correction component; and

an image signal processor that performs one or more image processing operations on the photographic frame prior to being fed to the blending component for blending.

12. A head-mounted-display system comprising:

an Application Specific Integrated Circuit (ASIC) that;

 

receives an image frame to be displayed on a head-mounted-display system,

wherein the image frame comprises;

a subset of lines; and

an additional subset of lines;

 

receives, from a motion-detection sensor operating in conjunction with the head-mounted-display system, motion data indicating that the head-mounted-display system has moved since generation of the image frame, wherein the motion data is received after the subset of lines have completed processing by the ASIC and before the additional subset of lines have completed processing by the ASIC; and

 

performs one or more hardware-accelerated intraframe processing operations on the additional subset of lines by modifying the additional subset of lines to compensate for the movement of the head-mounted-display system in accordance with the motion data even though the subset of lines are not eligible for the hardware-accelerated intraframe processing operations due at least in part to having completed processing when the motion data is received; and

a display device communicatively coupled to the ASIC, wherein the display device;

 

receives the additional subset of lines image frame from the ASIC special purpose hardware device; and

 

displays the additional subset of lines image frame.

13. The head-mounted-display system of claim 12, further comprising a clock generator that:

generates a clock signal; and

synchronizes the ASIC and the display device to one another by feeding the clock signal to both the ASIC and the display device.

14. The head-mounted-display system of claim 13, wherein the clock signal enables the ASIC to forgo at least one handshake-based synchronization operation with the display device.

15. The head-mounted-display system of claim 14, wherein the clock signal enables the ASIC to perform the hardware-accelerated intraframe processing operations on the image frame on a line-by-line basis due at least in part to the ASIC forgoing the handshake-based synchronization operation with the display device.

16. The head-mounted-display system of claim 15, further comprising a hardware frame-output interface forwards the additional subset of lines to the display device to facilitate displaying the additional subset of lines together with the subset of lines.

17. The head-mounted-display system of claim 16, wherein:

the subset of lines have not been modified to compensate for the movement of the head-mounted-display system since the generation of the image frame; and

the display device simultaneously displays, during at least one point in time, the subset of modified lines and the additional subset of lines.

18. The head-mounted-display system of claim 12, wherein the image frame comprises at least one of:

a photographic frame captured by a camera device operating in conjunction with the head-mounted-display system;

a virtual-reality frame that includes computer-generated imagery; and

a mixed-reality frame that includes both photographic content and computer-generated imagery.

19. The head-mounted-display system of claim 12, wherein the hardware-accelerated intraframe processing operations comprise at least one of:

asynchronous time warping; and

asynchronous space warping.

20. A method comprising:

receiving, by an Application Specific Integrated Circuit (ASIC), an image frame to be displayed on a head-mounted-display system, wherein the image frame comprises;

a subset of lines; and

an additional subset of lines;

receiving, by the ASIC, motion data indicating that the head-mounted-display system has moved since generation of the image frame, wherein the motion data is received after the subset of lines have completed processing by the ASIC and before the additional subset of lines have completed processing by the ASIC;

performing, by the ASIC, one or more hardware-accelerated intraframe processing operations on the additional subset of lines by modifying the additional subset of lines to compensate for the movement of the head-mounted-display system in accordance with the motion data even though the subset of lines are not eligible for the hardware-accelerated intraframe processing operations due at least in part to having completed processing when the motion data is received; and

feeding, by the ASIC upon completion of the hardware-accelerated intraframe processing operations, the additional subset of lines to a display device of the head-mounted-display system to facilitate displaying the additional subset of lines.