Glare correction via pixel processing

• US 9,911,395 B1
• Filed: 12/23/2014
• Issued: 03/06/2018
• Est. Priority Date: 12/23/2014
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Townsend, Matthew Alan, Khatri, Huma Dawood, Halberg, Gabrielle, Harris, Bryan, Dietrich, Douglas Sim
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US14/581,111
• Time in Patent Office: 1,169 Days
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G09G 2320/0233 Improving the luminance or brightness uniformity across the screen
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/0666 for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 3/3406 Control of illumination source illumination devices structurally associated with liquid crystal cells G02F1/1336
• CPC Class Codes: G09G 5/02 characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 5/10 Intensity circuits

Abstract

A system and method for generating a corrected input to a display to offset glare is provided. The system determines a location of a user'"'"'s head and calculates a reflection vector based on a head vector from a device to a point on the user'"'"'s head. The system identifies light sources using a camera and determines display pixels on a display of the device affected by the reflection of the light sources based on head detection and locations of the light sources. The system dynamically adjusts pixel brightness to account for a detected glare, thus improving the user'"'"'s experience operating the UI.

Claims

1. A computer-implemented method comprising:

determining a first head vector describing at least a direction from a device to a location of a user'"'"'s head relative to the device;

determining a distance from the user'"'"'s head to the device;

determining a second head vector for a first display pixel of a plurality of display pixels, wherein the determining the second head vector is based on the first head vector, the distance and a location of the first display pixel within the plurality of display pixels of a display of the device;

calculating, based on the second head vector, a reflection vector describing at least a direction from the device to an area reflected on the display;

generating an environment map, the environment map representing a three-dimensional environment of the device;

identifying a region in the environment map corresponding to a light source;

determining, using the reflection vector and the environment map, that the first display pixel corresponds to a reflection location on the display, the reflection location corresponding to a reflection of the light source; and

rendering a corrected user interface (UI) for the display, the corrected UI including a UI portion corresponding to the first display pixel, wherein the UI portion is displayed based at least in part on determining that the first display pixel corresponds to the reflection location.

2. The computer-implemented method of claim 1, further comprising:

identifying a first color value associated with a selected pixel from the region; and

inverting the first color value to generate a first inverted color value, the first inverted color value being a negative to the first color value, wherein the UI portion is displayed at the reduced brightness and includes the selected pixel displayed at the first inverted color value.

3. The computer-implemented method of claim 1, further comprising:

determining a reflection pixel in the environment map corresponding to the first display pixel using the reflection vector;

determining a first intensity of the reflection pixel; and

reducing an intensity of the first display pixel by the first intensity.

4. The computer-implemented method of claim 3, further comprising:

determining that the first intensity exceeds a threshold;

reducing the intensity of the first display pixel to zero;

rendering content to be displayed to avoid the first display pixel, the content to be displayed flowing around the first display pixel; and

displaying the rendered content.

5. The computer-implemented method of claim 1, wherein:

the determining the first head vector further comprises;

acquiring a first image that includes the user'"'"'s head;

acquiring a second image that includes the user'"'"'s head;

detecting the user'"'"'s head in the first image using facial detection;

detecting the user'"'"'s head in the second image using facial detection; and

determining the first head vector based on the first image and the second image, and

the determining the second head vector further comprises;

determining the location of the first display pixel on the device;

determining an offset from the location of the first display pixel to a location on the device associated with a first end of the first head vector; and

generating the second head vector based at least in part on the first head vector, the offset and the distance.

6. The computer-implemented method of claim 1, wherein the calculating further comprises:

determining a normal vector of the device;

calculating a dot product between the normal vector and the second head vector;

multiplying the dot product by twice the normal vector to generate a first vector; and

subtracting the second head vector from the first vector to generate a second vector, wherein the second vector is the reflection vector.

7. The computer-implemented method of claim 1, wherein the generating the environment map further comprises:

obtaining an image;

generating a first cubemap from the image, the first cubemap including image data;

detecting a rotation of the device; and

modifying the image data within the first cubemap based on the rotation of the device to generate a second cubemap, wherein the second cubemap is the environment map.

8. The computer-implemented method of claim 1, wherein:

the generating the environment map further comprises;

obtaining an image; and

generating, based on the image, a cubemap that surrounds a representation of the device, the cubemap being the environment map, and

the determining that the first display pixel corresponds to the reflection location further comprises;

determining a first point in the cubemap based on a first orientation of the device, wherein the reflection vector extends from the representation of the device to the first point;

detecting movement of the device;

determining a second orientation of the device based on the movement; and

determining a second point in the cubemap based on the second orientation of the device, wherein the reflection vector extends from the representation of the device to the second point.

17. The computer-implemented method of claim 1, wherein the generating the environment map further comprises:

receiving image data corresponding to the environment of the device;

generating, based on a first portion of the image data, a first side of the environment map; and

generating, based on a second portion of the image data, a second side of the environment map, the second side orthogonal to the first side in at least one dimension.

18. The computer-implemented method of claim 1, wherein the generating the environment map further comprises:

receiving image data corresponding to the environment of the device;

determining an orientation of the device; and

generating, based on the image data and the orientation of the device, the environment map such that at least a portion of the image data surrounds a representation of the device.

19. The computer-implemented method of claim 1, further comprising:

determining, at a second time, a second distance from the user'"'"'s head to the device;

determining a third head vector for the first display pixel, wherein the determining the third head vector is based on the first head vector, the second distance and the location of the first display pixel within the plurality of display pixels; and

calculating, based on the third head vector, a second reflection vector describing at least a direction from the device to a second area reflected on the display, wherein a size of the second area relative to a size of the first area corresponds to a difference between the distance and the second distance.

20. The computer-implemented method of claim 1, further comprising:

determining, at a second time, a second distance from the user'"'"'s head to the device;

determining, based at least in part on the second distance, a third head vector for the first display pixel; and

calculating, based on the third head vector, a second reflection vector describing at least a direction from the device to a second area reflected on the display,

wherein;

the area corresponds to a first portion of the environment map,

the second area corresponds to a second portion of the environment map,

the second distance is smaller than the distance, and

the second portion is larger than the first portion.

9. A system, comprising:

one or more processors; and

a memory including instructions operable to be executed by the one or more processors to perform a set of actions to configure the system to;

determine a first head vector describing at least a direction from a device to a location of a user'"'"'s head relative to the device;

determine a distance from the user'"'"'s head to the device;

determine a second head vector for a first display pixel of a plurality of display pixels, wherein the determining the second head vector is based on the first head vector, the distance and a location of the first display pixel within the plurality of display pixels of a display of the device;

calculate, based on the second head vector, a reflection vector describing at least a direction from the device to an area reflected on the display;

generate an environment map, the environment map representing a three-dimensional environment of the device;

identify a region in the environment map corresponding to a light source;

determine, using the reflection vector and the environment map, that the first display pixel corresponds to a reflection location on the display, the reflection location corresponding to a reflection of the light source; and

render a corrected user interface (UI) for the display, the corrected UI including a UI portion corresponding to the first display pixel, wherein the UI portion is displayed based at least in part on determining that the first display pixel corresponds to the reflection location.

10. The system of claim 9, wherein the instructions further configure the system to:

identify a first color value associated with a selected pixel from the region; and

invert the first color value to generate a first inverted color value, the first inverted color value being a negative of the first color value, wherein the UI portion is displayed at the reduced brightness and includes the selected pixel displayed at the first inverted color value.

11. The system of claim 9, wherein the instructions further configure the system to:

determine a reflection pixel in the environment map corresponding to the first display pixel using the reflection vector;

determine a first intensity of the reflection pixel; and

reduce an intensity of the first display pixel by the first intensity.

12. The system of claim 11, wherein the instructions further configure the system to:

determine that the first intensity exceeds a threshold;

reduce the intensity of the first display pixel to zero;

render content to be displayed to avoid the first display pixel, the content to be displayed flowing around the first display pixel; and

display the rendered content.

13. The system of claim 9, wherein the instructions further configure the system to:

acquire a first image that includes the user'"'"'s head;

acquire a second image that includes the user'"'"'s head;

detect the user'"'"'s head in the first image using facial detection;

detect the user'"'"'s head in the second image using facial detection;

determine the first head vector based on the first image and the second image determine the location of the first display pixel on the device;

determine an offset from the location of the first display pixel to a location on the device associated with the first head vector; and

generate the second head vector based at least in part on the first head vector, the offset and the distance.

14. The system of claim 9, wherein the instructions further configure the system to:

determine a normal vector of the device;

calculate a dot product between the normal vector and the second head vector;

multiply the dot product by twice the normal vector to generate a first vector; and

subtract the second head vector from the first vector to generate a second vector, wherein the second vector is the reflection vector.

15. The system of claim 9, wherein the instructions further configure the system to:

obtain an image;

generate a first cubemap from the image, the first cubemap including image data;

detect a rotation of the device; and

modify the image data within the first cubemap based on the rotation of the device to generate a second cubemap, wherein the second cubemap is the environment map.

16. The system of claim 9, wherein the instructions further configure the system to:

obtain an image;

generate, based on the image, a cubemap that surrounds a representation of the device, the cubemap being the environment map;

determine a first point in the cubemap based on a first orientation of the device, wherein the reflection vector extends from the representation of the device to the first point;

detect movement of the device;

determine a second orientation of the device based on the movement; and

determine a second point in the cubemap based on the second orientation of the device, wherein the reflection vector extends from the representation of the device to the second point.