Use of eye tracking to adjust region-of-interest (ROI) for compressing images for transmission

• US 10,684,685 B2
• Filed: 08/06/2019
• Issued: 06/16/2020
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Young, Andrew, Stafford, Jeffrey Roger
• Primary Examiner(s): Nguyen, Vu
• Application Number:
  US16/533,659
• Publication Number:
  US 20190361526A1
• Time in Patent Office: 315 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2310/04 Partial updating of the display screen
• CPC Class Codes: G09G 2340/0428 Gradation resolution change
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]

Abstract

Gaze tracking data may representing a user'"'"'s gaze with respect to one or more images transmitted to a user are analyzed to determine one or more regions of interest.

The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

Claims

1. A method comprising:

obtaining gaze tracking data representing a user'"'"'s gaze with respect to one or more images transmitted to a user;

analyzing the gaze tracking data to determine one or more regions of interest; and

selectively compressing the one or more images to reduce the number of bits needed to transmit data corresponding to the one or more images to the user so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest wherein selectively compressing the one or more images includes ignoring packet loss for transmission of one or more parts of the image outside the one or more regions of interest of the one or more transmitted images.

2. The method of claim 1, wherein selectively compressing the one or more images includes selectively compressing the image so that the one or, more regions of interest undergo less compression than portions of the image outside the one or more regions of interest.

3. The method of claim 1, wherein selectively compressing the one or more images includes using a higher quantization parameter for portions of the image that are outside the regions of interest than for portions of the image within the one or more regions of interest.

4. The method of claim 1, further comprising reducing transmission power for transmitting the one or more transmitted images for a period of time during which the user blinks or the user'"'"'s vision undergoes a saccade.

5. The method of claim 1, further comprising ceasing transmission of the one or more images for time period during which the user blinks.

6. The method of claim 1, further comprising ceasing transmission of the one or more images for a time period during which a user'"'"'s vision undergoes a saccade.

7. The method of claim 1, wherein said analyzing the gaze tracking data includes analyzing a rotational velocity and/or rotational acceleration of the user'"'"'s eye.

8. The method of claim 1, wherein said analyzing the gaze tracking data includes analyzing information related to movement of the user'"'"'s eyelid.

9. The method of claim 1, wherein said analyzing the gaze tracking data to determine the onset and duration of a vision tracking event includes analyzing previous gaze tracking data to improve detection and estimates of duration estimation for vision interrupting events.

10. The method of claim 9, further comprising determining one or more gaze tracking parameters from the gaze tracking data;

generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters;

generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and

presenting the one or more foveated images with the foveated image data to the user.

11. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes.

12. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining an error in a fixation of the user'"'"'s eye on a region of interest of the one or more regions of interest.

13. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining whether the user'"'"'s eye is moving in smooth pursuit.

14. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a confidence interval regarding the current gaze position.

15. The method of claim 10, wherein, determining one or more gaze tracking error parameters from the gaze tracking data includes determining the blink cycle of a user.

16. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining the saccade cycle of a user.

17. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a transition in the gaze direction of a user as a result of a change in the depth of field of the presented image.

18. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining whether a user is color blind.

19. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining the level of gaze stability of the user'"'"'s eye.

20. The method of claim 10, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining whether the user'"'"'s eye movement is a precursor to the user'"'"'s head movement.

21. The method of claim 10, wherein the adjusted foveation data includes data characterizing a particular region of interest of the one or more regions of interest, wherein the data characterizing the particular region of interest includes a foveation region and a transition region, wherein the foveation region is characterized by a higher level of detail than the transition region and the transition region is characterized by a higher level of detail than regions of a corresponding image outside the particular region of interest.

22. The method of claim 21, wherein generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest includes adjusting a size of the transition region.

23. The method of claim 10, wherein the adjusted foveation data includes data representing vertex density in the one or more regions of interest.

24. The method of claim 10, wherein the adjusted foveation data includes data representing pixel resolution in the one or more regions of interest.

25. The method of claim 10, further comprising creating a standard 2D compliant image from the foveated image data for presentation on one or more additional displays.

26. The method of claim 1, further comprising reducing transmission power for transmitting the one or more transmitted images for a period of time during which the user blinks or the user'"'"'s vision undergoes a saccade and subsequently increasing transmission power.

27. The method of claim 1, further comprising generating the one or more transmitted images by generating foveated image data representing one or more foveated images using foveation data representing one or more regions of interest of the image determined from the gaze tracking data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest.

28. The method of claim 27, wherein the foveation data includes data representing vertex density in the one or more regions of interest.

29. The method of claim 27, wherein the foveation data includes data representing pixel resolution in the one or more regions of interest.

30. The method of claim 27, further comprising creating a standard 2D compliant image from the foveated image data for presentation on one or more additional displays.

31. The method of claim 1, wherein selectively compressing the one or more images includes omitting error packet transmission for portions of the one or more images outside the one or more regions interest but not for portions of the one or more images within the one or more regions of interest.

32. A system, comprising:

a processor;

a memory; and

computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a method when executed, the method comprising;

obtaining gaze tracking data representing a user'"'"'s gaze with respect to one or more images transmitted to a user;

analyzing the gaze tracking data to determine one or more regions of interest; and

selectively compressing the one or more images to reduce the number of bits needed to transmit data corresponding to the one or more images to the user so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest wherein selectively compressing the one or more images includes ignoring packet loss for transmission of one or more parts of the image outside the one or more regions of interest of the one or more transmitted images.

33. A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a method when executed, the method comprising:

obtaining gaze tracking data representing a user'"'"'s gaze with respect to one or more images transmitted to a user;

analyzing the gaze tracking data to determine one or more, regions of interest;

and selectively compressing the one or more images to reduce the number of bits needed to transmit data corresponding to the one or more images to the user so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest wherein selectively compressing the one or more images includes ignoring packet loss for transmission of one or more parts of the image outside the one or more regions of interest of the one or more transmitted images.