Correcting for parallax in electronic displays

• US 10,019,107 B2
• Filed: 11/10/2014
• Issued: 07/10/2018
• Est. Priority Date: 01/26/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Baldwin, Leo B.
• Primary Examiner(s): SOTO LOPEZ, JOSE R
• Application Number:
  US14/537,821
• Publication Number:
  US 20150138163A1
• Time in Patent Office: 1,338 Days
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

The ability of a user to provide input when using a touch screen or other such element of a computing device can be improved by correcting for parallax errors. A camera of the computing device can determine the relative position of the user'"'"'s head or eyes with respect to the device, and determine a viewing perspective of the user with respect to content displayed on the touch screen. Using the perspective direction and information about the dimensions of the touch screen, the device can determine an amount and direction of parallax error, and can make adjustments to remove or minimize that error. The device can cause the position of the displayed graphical elements to shift and/or can adjust the mappings of the touch sensitive regions that correspond to the graphical elements. Such an approach enables the graphical elements and touch mappings to align from the current perspective of the user.

Claims

1. A computer-implemented method, comprising:

displaying content on a touch screen of a computing device, the touch screen having a first surface for displaying the content and a second surface for receiving input to the computing device;

activating gaze detection on the computing device for a first portion of a first period of time;

determining, for the first portion of the first period of time, a range of angles of incidence between a first gaze direction and the touch screen;

associating, for the first portion of the first period of time, a first location of the second surface to a first element displayed on the first surface;

determining, for the first portion of the first period of time, that a number of times that inputs are edited from a second element, associated with a second location of the second surface, to the first element exceeds a threshold number of times;

deactivating the gaze detection on the computing device after the first period of time;

activating the gaze detection on the computing device for a second portion of a second period of time;

determining, for at least one first portion of time within the second period of time, that an angle of incidence between a second gaze direction and the touch screen is within the range of angles of incidence;

associating, for the at least one first portion of time within the second period of time, the second location to the first element;

receiving, within the at least one first portion of time within the second period of time, an input to the second location; and

determining, within the at least one first portion of time within the second period of time, an action to perform on the computing device corresponding to a selection of the first element.

2. The computer-implemented method of claim 1, further comprising:

capturing image information using a camera of the computing device;

analyzing the image information to locate a representation of at least one feature of a user; and

determining the first gaze direction based at least in part upon a position of the representation in the image information.

3. The computer-implemented method of claim 1, further comprising:

determining a change in an orientation of the computing device within at least one second portion of time within the second period of time;

determining, for the at least one second portion of time within the second period of time, that a second angle of incidence between a third gaze direction and the touch screen is outside the range of angles of incidence; and

associating, for the at least one second portion of time within the second period of time, a third location of the second surface to the first element.

4. The computer-implemented method of claim 1, further comprising:

displaying the first element as a first key of a virtual keyboard;

displaying the second element as a second key of the virtual keyboard; and

monitoring the number of times that the inputs are edited as a number of times that the second key is deleted and the first key is selected.

5. The computer-implemented method of claim 1, further comprising:

displaying the first element as a first graphical user interface element;

displaying the second element as a second graphical user interface element; and

monitoring the number of times that the inputs are edited as a number of times that the second graphical user interface element is deselected and the first graphical user interface element is selected.

6. The computer-implemented method of claim 1, further comprising:

determining a dominant eye of a user; and

determining the first gaze direction based at least in part on the dominant eye.

19. The computer-implemented method of claim 1, further comprising:

updating a scale of at least one of the first element, the first location, or the second location to account for foreshortening.

7. A computing device, comprising:

a processor;

a touch screen having a first surface for displaying content and a second surface for receiving input to the computing device; and

non-transitory memory including instructions that, upon being executed by the processor, cause the computing device to;

display content on the touch screen;

activate gaze detection on the computing device for a first portion of a first period of time;

determine, for the first portion of the first period of time, a range of angles of incidence between a first gaze direction and the touch screen;

associate, for the first portion of the first period of time, a first location of the second surface to a first element displayed on the first surface;

determine, for the first portion of the first period of time, that a number of times that inputs are edited from a second element, associated with a second location of the second surface, to the first element exceeds a threshold number of times;

deactivate the gaze detection on the computing device after the first period of time;

activate the gaze detection on the computing device for a second portion of a second period of time;

determine, for at least one portion of time within the second period of time, that an angle of incidence between a second gaze direction and the touch screen is within the range of angles of incidence;

associate, for the at least one portion of time, the second location to the first element;

receive, within the at least one portion of time, an input to the second location; and

determine, within the at least one portion of time, an action to perform on the computing device corresponding to a selection of the first element.

8. The computing device of claim 7, further comprising:

a camera,

wherein the instructions, upon being executed, further cause the computing device to;

capture image information using the camera;

analyze the image information to locate a representation of at least one feature of a user; and

determine the first gaze direction based at least in part upon a position of the representation in the image information.

9. The computing device of claim 8, wherein the instructions, upon being executed, further cause the computing device to:

analyze the image information using at least one of an image recognition algorithm, a contour matching algorithm, a retro-reflection detecting algorithm, or a color matching algorithm.

10. The computing device of claim 7, wherein the instructions, upon being executed, further cause the computing device to:

display the first element as a first key of a virtual keyboard;

display the second element as a second key of the virtual keyboard; and

monitor the number of times that the inputs are edited as a number of times that the second key is deleted and the first key is selected.

11. The computing device of claim 7, wherein instructions, upon being executed, further cause the computing device to:

display the first element as a first graphical user interface element;

display the second element as a second graphical user interface element; and

monitor the number of times that the inputs are edited as a number of times that the second graphical user interface element is deselected and the first graphical user interface element is selected.

12. The computing device of claim 7, further comprising:

an illumination element,

wherein the instructions, upon being executed, further cause the computing device to;

determine a threshold amount of illumination needed to locate a feature of a user; and

activate an illumination element when a current amount of illumination is less than the threshold amount of illumination.

13. The computing device of claim 7, wherein the instructions, upon being executed, further cause the computing device to:

update a scale of at least one of the first element, the first location, or the second location to account for foreshortening.

14. A non-transitory computer-readable storage medium storing instructions that, upon being executed by at least one processor of a computing device, cause the computing device to:

display content on a touch screen of the computing device, the touch screen having a first surface for displaying the content and a second surface for receiving input to the computing device;

activate gaze detection on the computing device for a first portion of a first period of time;

determine, for the first portion of the first period of time, a range of angles of incidence between a first gaze direction and the touch screen;

associate, for the first portion of the first period of time, a first location of the second surface to a first element displayed on the first surface based at least in part upon a first angle of incidence between the first gaze direction and the touch screen;

determine, for the first portion of the first period of time, that a number of times that inputs are edited from a second element, associated with a second location of the second surface, to the first element exceeds a threshold number of times;

deactivate the gaze detection on the computing device after the first period of time;

activate the gaze detection on the computing device for a second portion of a second period of time;

determine, for at least one first portion of time within the second period of time, that an angle of incidence between a second gaze direction and the touch screen is within the range of angles of incidence;

associate, for the at least one first portion of time within the second period of time, the second location to the first element;

receive, within the at least one first portion of time within the second period of time, an input to the second location; and

determine, within the at least one first portion of time within the second period of time, an action to perform on the computing device corresponding to a selection of the first element.

15. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, upon being executed, further cause the computing device to:

determine a change in orientation of the computing device within a second portion of the second period of time;

determine, for the second portion of the second period of time, that a second angle of incidence between a third gaze direction and the touch screen is outside the range of angles of incidence; and

associate, for the second portion of the second period of time, a third location of the second surface to the first element.

16. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, upon being executed, further cause the computing device to:

display the first element as a first key of a virtual keyboard;

display the second element as a second key of the virtual keyboard; and

monitor the number of times that the inputs are edited as a number of times that the second key is deleted and the first key is selected.

17. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, upon being executed, further cause the computing device to:

display the first element as a first graphical user interface element;

display the second element as a second graphical user interface element; and

monitor the number of times that the inputs are edited as a number of times that the second graphical user interface element is deselected and the first graphical user interface element is selected.

18. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, upon being executed, further cause the computing device to:

update a scale of at least one of the first element, the first location, or the second location to account for foreshortening.

20. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, upon being executed, further cause the computing device to:

determine a dominant eye of a user; and

determine the first gaze direction based at least in part on the dominant eye.