Selective peripheral vision filtering in a foveated rendering system

• US 10,169,846 B2
• Filed: 03/31/2016
• Issued: 01/01/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Young, Andrew
• Primary Examiner(s): Demeter, Hilina K
• Application Number:
  US15/087,629
• Publication Number:
  US 20170287112A1
• Time in Patent Office: 1,006 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

Gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to the viewer is used to generate foveated image data representing one or more foveated images characterized by a higher level of detail within one or more regions of interest and a lower level of detail outside the regions of interest. The image data for portions outside the one or more regions of interest is selectively filtered to reduce visual artifacts due to contrast resulting from the lower level of detail before compositing foveated images for presentation.

Claims

1. A graphics processing method comprising:

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to a viewer;

generating foveated digital image data representing one or more foveated images using the gaze tracking data, wherein the one or more foveated images are generated with a higher level of detail within one or more regions of interest than a lower level of detail for one or more portions of the foveated images outside the one or more regions of interest;

filtering the digital image data for the at least the one or more portions of the foveated images outside the one or more regions of interest to reduce visual artifacts due to contrast resulting from the lower level of detail, wherein one or more objects within the one or more foveated images but outside the regions of interest are tagged with information reflecting a relative importance of the one or more objects by automatically tagging objects in the one or more foveated images that are moving as important based on motion vectors computed from the digital image data; and

compositing the one or more filtered foveated images; and

displaying the one or more filtered foveated images to the viewer on a display device.

2. The method of claim 1, wherein filtering the digital image data for the at least the one or more portions of the one or more foveated images outside the one or more regions of interest to reduce visual artifacts due to contrast resulting from the lower level of detail includes selectively filtering of the one or more foveated images outside the one or more regions of interest that correspond to selected objects to maintain or increase the contrast.

3. The method of claim 1, wherein important objects outside the one or more regions of interest are subjected to less filtering to reduce artifacts than other portions of the one or more images outside the one or more regions of interest.

4. The method of claim 1, wherein tagging the one or more objects includes associating metadata with the one or more objects.

5. The method of claim 4, wherein associating the metadata with the one or more objects is done subjectively through an application programming interface (API).

6. The method of claim 4, wherein associating the metadata with the one or more objects is done objectively by an algorithm.

7. The method of claim 1, wherein filtering the digital image data includes inserting filtering information into the digital image data.

8. The method of claim 7, wherein filtering the digital image data includes inserting the filtering information into the digital image data by inserting filtering data into a color buffer via the alpha channel, depth buffer, stencil buffer, or an auxiliary buffer for each pixel in the one or more foveated images.

9. The graphics processing method of claim 1 wherein filtering the image data comprises applying an anti-aliasing filter to one or more portions of the one or more foveated images outside the one or more regions of interest.

10. A system, comprising:

a processor;

a memory; and

computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising;

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to a viewer;

generating digital image data representing one or more foveated images using the gaze tracking data, wherein the one or more foveated images are generated with a higher level of detail within one or more regions of interest than a lower level of detail for one or more portions of the foveated images outside the one or more regions of interest;

generating filtered digital image data by filtering the digital image data for the at least the one or more portions of the foveated images outside the one or more regions of interest to reduce visual artifacts due to contrast resulting from the lower level of detail, wherein one or more objects within the one or more foveated images but outside the regions of interest are tagged with information reflecting a relative importance of the one or more objects by automatically tagging objects in the one or more foveated images that are moving as important based on motion vectors computed from the digital image data; and

displaying a final composited image comprised of one or more filtered foveated images to the viewer on a display device.

11. A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to a viewer;

generating digital image data representing one or more foveated images using the gaze tracking data, wherein the one or more foveated images are generated with a higher level of detail within one or more regions of interest than a lower level of detail for one or more portions of the foveated images outside the one or more regions of interest;

generating filtered digital image data by filtering the digital image data for the at least the one or more portions of the foveated images outside the one or more regions of interest to reduce visual artifacts due to contrast resulting from the lower level of detail, wherein one or more objects within the one or more foveated images but outside the regions of interest are tagged with information reflecting a relative importance of the one or more objects by automatically tagging objects in the one or more foveated images that are moving as important based on motion vectors computed from the digital image data; and

displaying a final composited image comprised of one or more filtered foveated images to the viewer on a display device.