Adaptive parameters in image regions based on eye tracking information

• US 10,373,592 B2
• Filed: 07/28/2017
• Issued: 08/06/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Tall, Martin Henrik, San Agustin Lopez, Javier, Dahl, Rasmus
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US15/662,582
• Publication Number:
  US 20180033405A1
• Time in Patent Office: 739 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/37 Details of the operation on graphic patterns G09G5/38 takes precedence
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object

Abstract

A display system divides a screen into regions and applies a different set of rendering/encoding parameters to each region. The system applies a first set of parameters to a first region that is being viewed by a fovea of an eye of a user. The system may also apply a second set of parameters to a second region that is being viewed by a parafovea of the eye, and apply a third set of parameters to a third region that is being viewed by the area of the eye outside of the parafovea. The first set of parameters are selected to yield relatively high image quality, while the second set of parameters are yield intermediate quality, and the third set of parameters yield lower quality. As a result, the second region and the third region can be rendered, encoded, and transmitted with less computing power and less bandwidth.

Claims

1. A method comprising:

receiving eye tracking information comprising at least one image of an eye of a user viewing a screen of a display device;

determining, based on the eye tracking information, a first screen region corresponding to a region of the screen, the first screen region containing a point of regard of the user, the point of regard representing a point on the screen at which the user is looking at the time the eye tracking information was captured;

determining, based on the eye tracking information, a second screen region corresponding to a region of the screen that is separate from the first screen region;

determining, based on the eye tracking information, a user'"'"'s depth of focus;

encoding an image for display on the screen, the image comprising an object with at least, a portion of the object to be displayed within the first screen region, the encoding comprising;

encoding a first image region of the image based on a first set of encoding parameters, the first image region comprising a portion of the image to be displayed in the first screen region,

encoding a second image region of the image based on a second set of encoding parameters, the second image region separate from the first image region and to be displayed in the second screen region, the second set of encoding parameters resulting in a lower quality than the first set of encoding parameters, and

responsive to a difference between a depth of the object in the image and the user'"'"'s depth of focus exceeding a threshold value, encoding the object, including the portion to be displayed within the first screen region, with the second set of encoding parameters; and

transmitting the encoded image to the display device to be displayed on the screen.

2. The method of claim 1, wherein the first screen region corresponds to a region of the screen viewed by a fovea of the eye.

3. The method of claim 1, wherein the second screen region corresponds to a region of the screen viewed by a portion of the retina of the eye outside of the fovea.

4. The method of claim 1, wherein the first set of encoding parameters includes a first image resolution, the second set of encoding parameters includes a second image resolution, and the second image resolution is lower than the first image resolution.

5. The method of claim 4, wherein encoding the first image region comprises generating a first encoded image region, wherein encoding the second image region comprises generating a second encoded image region, and further comprising:

generating a packed image comprising the first encoded image region and the second encoded image region.

6. The method of claim 1, wherein the first set of encoding parameters includes a first framerate, the second set of encoding parameters includes a second framerate, and the second framerate is lower than the first framerate.

7. The method of claim 1, further comprising:

determining, based on the eye tracking information, a third screen region corresponding to a region of the screen that is separate from the first screen region and the second screen region,

wherein encoding the image for display on the screen further comprises;

encoding a third image region of the image based on a third set of encoding parameters, the third image region separate from the first image region and the second image region and to be displayed in the third screen region, the third set of encoding parameters resulting in a lower quality than the first set of encoding parameters and a higher quality than the second set of encoding parameters.

8. The method of claim 7, wherein the third screen region corresponds to a region of the screen viewed by a parafovea of the eye.

9. The method of claim 1, wherein the display device is a virtual reality headset.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving eye tracking information comprising at least one image of an eye of a user viewing a screen of a display device;

determining, based on the eye tracking information, a first screen region corresponding to a region of the screen, the first screen region containing a point of regard of the user, the point of regard representing a point on the screen at which the user is looking at the time the eye tracking information was captured;

determining, based on the eye tracking information, a second screen region corresponding to a region of the screen that is separate from the first screen region;

determining, based on the eye tracking information, a user'"'"'s depth of focus,

encoding an image for display on the screen, the image comprising an object with at least a portion of the object to be displayed within the first screen region, the encoding comprising;

encoding a first image region of the image based on a first set of encoding parameters, the first image region comprising a portion of the image to be displayed in the first screen region,

encoding a second image region of the image based on a second set of encoding parameters, the second image region separate from the first image region and to be displayed in the second screen region, the second set of encoding parameters resulting in a lower quality than the first set of encoding parameters, and

responsive to a difference between a depth of the object in the image and the user'"'"'s depth of focus exceeding a threshold value, encoding the object, including the portion to be displayed within the first screen region, with the second set of encoding parameters; and

transmitting the encoded image to the display device to be displayed on the screen.

11. The computer-readable storage medium of claim 10, wherein the first screen region corresponds to a region of the screen viewed by a fovea of the eye.

12. The computer-readable storage medium of claim 10, wherein the second screen region corresponds to a region of the screen viewed by a portion of the retina of the eye outside of the fovea.

13. The computer-readable storage medium of claim 10, wherein the first set of encoding parameters includes a first image resolution, the second set of encoding parameters includes a second image resolution, and the second image resolution is lower than the first image resolution.

14. The computer-readable storage medium of claim 10, wherein encoding the first image region comprises generating a first encoded image region, wherein encoding the second image region comprises generating a second encoded image region, and further comprising:

generating a packed image comprising the first encoded image region and the second encoded image region.

15. The computer-readable storage medium of claim 10, wherein the first set of encoding parameters includes a first framerate, the second set of encoding parameters includes a second framerate, and the second framerate is lower than the first framerate.

16. The computer-readable storage medium of claim 10, the operations further comprising:

determining, based on the eye tracking information, a third screen region corresponding to a region of the screen that is separate from the first screen region and the second screen region,

wherein encoding the image for display on the screen further comprises;

encoding a third image region of the image based on a third set of encoding parameters, the third image region separate from the first image region and the second image region and to be displayed in the third screen region, the third set of encoding parameters resulting in a lower quality than the first set of encoding parameters and a higher quality than the second set of encoding parameters.

17. The computer-readable storage medium of claim 16, wherein the third screen region corresponds to a region of the screen viewed by a parafovea of the eye.

18. The computer-readable storage medium of claim 10, wherein the display device is a virtual reality headset.

19. A system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving eye tracking information comprising at least one image of an eye of a user viewing a screen of a display device,

determining, based on the eye tracking information, a first screen region corresponding to a region of the screen, the first screen region containing a point of regard of the user, the point of regard representing a point on the screen at which the user is looking at the time the eye tracking information was captured,

determining, based on the eye tracking information, a second screen region corresponding to a region of the screen that is separate from the first screen region,

determining, based on the eye tracking information, a user'"'"'s depth of focus,

encoding an image for display on the screen, the image comprising an object with at least a portion of the object to be displayed within the first screen region, the encoding comprising;

encoding a first image region of the image based on a first set of encoding parameters, the first image region comprising a portion of the image to be displayed in the first screen region,

encoding a second image region of the image based on a second set of encoding parameters, the second image region separate from the first image region and to be displayed in the second screen region, the second set of encoding parameters resulting in a lower quality than the first set of encoding parameters, and

responsive to a difference between a depth of the object in the image and the user'"'"'s depth of focus exceeding a threshold value, encoding the object, including the portion to be displayed within the first screen region, with the second set of encoding parameters, and

transmitting the encoded image to the display device to be displayed on the screen.

20. The system of claim 19, the operations further comprising:

determining, based on the eye tracking information, a third screen region corresponding to a region of the screen that is separate from the first screen region and the second screen region,

wherein encoding the image for display on the screen further comprises;

encoding a third image region of the image based on a third set of encoding parameters, the third image region separate from the first image region and the second image region and to be displayed in the third screen region, the third set of encoding parameters resulting in a lower quality than the first set of encoding parameters and a higher quality than the second set of encoding parameters.