Action based device control

• US 8,913,004 B1
• Filed: 03/05/2010
• Issued: 12/16/2014
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bozarth, Bradley J., Karakotsios, Kenneth M., Bezos, Jeffrey P.
• Primary Examiner(s): FRY, MATTHEW A
• Application Number:
  US12/718,788
• Time in Patent Office: 1,747 Days
• Field of Search: 345/156, 345/173, 345/157, 345/211, 382/103, 382117-118, 348/78, 715/700, 715/767, 715/802, 351/209, 713/300, 396/51, 700 12- 15
• US Class Current: 345/156
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3234 Power saving characterised by the action undertaken
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The determination of a gaze direction or field of view of a user with respect to a computing device can control aspects of the device, such as to reduce power or resource consumption. A computing device can include an imaging element and software for locating aspects of a user'"'"'s facial features relative to the device, such that an orientation of the user'"'"'s features relative to the device can be determined. Various actions on the device can be executed based at least in part upon a determined gaze direction of the user. In some embodiments, a display element of the device can turn off, and one or more inputs can be disabled, when the user is not looking at the device.

Claims

1. A method of controlling a computing device, comprising:

obtaining image data using an imaging element of the computing device;

determining that a user interacting with the computing device appears in at least one portion of the image data;

analyzing the image data, using the computing device, to determine a gaze direction of the user in response to determining that the user appears in the at least one portion of the image data;

determining a first region of a display element of the computing device, a second region that is outside a first boundary of the first region and within a second boundary, and a third region that is outside the second boundary, wherein at least one of the first boundary or the second boundary is adjusted over a period of time, the first region corresponding to a first user state in which the user is at least substantially gazing at the display element, the second region corresponding to a second user state in which the user is gazing away from the display element and at least substantially facing the computing device, and the third region corresponding to a third user state in which the user is gazing away from the display element and facing away from the computing device;

causing the display element to be in a first power state and activating at least one of a plurality of input elements of the computing device, based at least in part upon the gaze direction being determined to be directed towards the first region;

causing the display element to be in a second power state and altering a functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the second region, the second power state having a lower power consumption rate than the first power state; and

causing the display element to be in a third power state and deactivating functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the third region, the third power state having a lower power consumption rate than the second power state.

2. The method according to claim 1, wherein the first power state is an on state and the third power state is an off state.

3. The method according to claim 1, wherein:

the first power state comprises at least one of a first contrast, a first brightness, and a first resolution;

the second power state is one of a plurality of power states each comprising at least one of a second contrast, a second brightness, and a second resolution; and

the third power state comprises at least one of a third contrast, a third bright, and a third resolution.

4. The method according to claim 1, wherein analyzing the image data to determine the gaze direction of the user comprises analyzing the at least one portion of the image data to determine at least one of a state, or a change in a set of measurable aspects, corresponding to facial features of the user of the computing device.

5. The method according to claim 4, wherein the set of measurable aspects includes at least one of a dimension of at least one eye of the user, a dimension of at least one pupil of the user, a dimension of at least one retina of the user, a separation of features on a face of the user, a relative location of a feature on the face of the user, or a relative orientation between at least two features on the face of the user.

6. The method according to claim 1, wherein analyzing the image data to determine the gaze direction of the user further comprises analyzing the image data to determine a change in relative orientation between the user and the computing device.

7. The method according to claim 1, wherein the imaging element includes at least one of a digital camera element or an infrared radiation detector.

8. The method according to claim 1, wherein the plurality of input elements includes at least one of a touch-sensitive element, a keyboard, a keypad, a roller button, a second imaging element, an audio-capturing element, a power button, a power charging element, or a physical switch.

9. A method of controlling a computing device, comprising:

obtaining image data using an imaging element of the computing device;

determining that a user interacting with the computing device appears in at least one portion of the image data;

obtaining environmental data from an environment-determining element of the computing device;

analyzing, using the computing device, the image data to determine a gaze direction of the user and the environmental data to determine an environmental state in response to determining that the user appears in the at least one portion of the image data;

determining a first zone of a display element of the computing device, a second zone that is outside a first boundary of the first zone and within a second boundary, and a third zone that is outside the second boundary, wherein at least one of the first boundary or the second boundary is adjusted over a period of time, the first zone corresponding to a first user state in which the user is at least substantially gazing at the display element, the second zone corresponding to a second user state in which the user is gazing away from the display element and at least substantially facing the computing device, and the third zone corresponding to a third user state in which the user is gazing away from the display element and facing away from the computing device;

causing the display element to be in a first power state and activating at least one of a plurality of input elements of the computing device, based at least in part upon the gaze direction of the user being determined to be directed towards the first zone;

causing the display element to be in a second power state and altering a functionality of at least one of the plurality of input elements based at least in part upon the environmental state and the gaze direction of the user being determined to be directed towards the second zone; and

causing the display element to be in a third power state and deactivating functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction of the user being determined to be directed towards the third zone.

10. The method according to claim 9, wherein:

the first power state is an on state, and causing the display element to be in the first power state is in further response to determining that the user is holding the computing device, as determined from the environmental data;

the second power state is one of a plurality of power states each comprising at least one of a respective contrast, a respective brightness, or a respective resolution; and

the third power state is an off state, and causing the display element to be in the second power state is in further response to determining that the user is not holding the computing device, as determined from the environmental data.

11. The method according to claim 9 wherein the at least one user input element includes at least one of a touch-sensitive element, a keyboard, a keypad, a roller button, a second imaging element, an audio-capturing element, a power button, a power charging element, or a physical switch.

12. The method according to claim 9, wherein the at least one environment-determining element includes at least one of a touch-sensitive element, an orientation determining element, an audio recording element, or a light sensing element.

13. The method according to claim 12, wherein the orientation determining element is at least one of an accelerometer, a motion detector, an electronic compass, a global-positioning system receiver, or an electronic level.

14. The method according to claim 9, further comprising:

utilizing calibration information to analyze the image data and the environmental data.

15. The method according to claim 9, wherein causing the display element to be in the first power state, causing the display element to be in the second power state, and causing the display element to be in the third power state are each further based at least in part upon at least one application executing on the computing device.

16. A computing device, comprising:

a processor;

a display element for displaying a graphical user interface;

an imaging element;

a plurality of input elements;

a non-transitory memory device including instructions operable to be executed by the processor to perform a set of actions, the instructions, upon being executed by the processor, causing the computing device to;

obtain image data using the imaging element;

determine that a user interacting with the computing device appears in at least one portion of the image data;

analyze the image data to determine a gaze direction of the user in response to determining that the user appears in the at least one portion of the image data;

determine a first zone that includes at least one portion of the display element, a second zone that is outside a first boundary of the first zone and within a second boundary, and a third zone that is outside the second boundary, wherein at least one of the first boundary or the second boundary is adjusted over a period of time, the first zone corresponding to a first user state in which the user is at least substantially gazing at the display element, the second zone corresponding to a second user state in which the user is gazing away from the display element and at least substantially facing the computing device, and the third zone corresponding to a third user state in which the user is gazing away from the display element and facing away from the computing device;

cause the display element to be in a first power state and activate at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the first zone;

cause the display element to be in a second power state and alter a functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the second zone; and

cause the display element to be in a third power state and deactivate functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the third zone.

17. The computing device according to claim 16, wherein:

the first power state is at least one of an on power state or a state with at least one of a first contrast, a first brightness, or a first resolution;

the second power state is one of a plurality of states each comprising at least one of a second contrast, a second brightness, or a second resolution; and

the third power state is at least one of an off power state or a state with at least one of a third contrast, a third brightness, or a third resolution.

18. The computing device according to claim 16, wherein the imaging element includes at least one of a digital camera element or an infrared radiation detector.

19. The computing device according to claim 16, wherein the plurality of input elements includes at least one of a touch-sensitive element, a keyboard, a keypad, a roller button, a second imaging element, an audio-capturing element, a power button, a power charging element, or a physical switch.

20. A non-transitory computer-readable storage medium storing processor-executable instructions for controlling a computing device, the processor-executable instructions, upon being executed by a processor of the computing device, causing the computing device to perform:

obtaining image data using an imaging element;

determining that a user interacting with the computing device appears in at least one portion of the image data;

analyzing the image data to determine a gaze direction of a user in response to determining that the user appears in the at least one portion of the image data;

determining a first zone of a display element of the computing device, a second zone that is outside a first boundary of the first zone and within a second boundary, and a third zone that is outside the second boundary, wherein at least one of the first boundary or the second boundary is adjusted over a period of time, the first zone corresponding to a first user state in which the user is at least substantially gazing at the display element, the second zone corresponding to a second user state in which the user is gazing away from the display element and at least substantially facing the computing device, and the third zone corresponding to a third user state in which the user is gazing away from the display element and facing away from the computing device;

causing the display element to be in a first power state and activating at least one of a plurality of input elements of the computing device, based at least in part upon the gaze direction being determined to be directed towards the first zone;

causing the display element to be in a second power state and altering a functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed towards the second zone; and

causing the display element to be in a third power state and deactivating functionality of at least one of the plurality of input elements, based at least in part upon the gaze direction being determined to be directed toward the third zone.

21. The non-transitory computer-readable storage medium according to claim 20, wherein:

the first power state is at least one of an on power state or a state with at least one of a first contrast, a first brightness, or a first resolution;

the second power state is one of a plurality of states each comprising at least one of a second contrast, a second brightness, or a second resolution; and

the third power state is at least one of an off power state or a state with at least one of a third contrast, a third brightness, or a third resolution.

22. The non-transitory computer-readable storage medium according to claim 20, wherein the imaging element includes at least one of a digital camera element or an infrared radiation detector.

23. The non-transitory computer-readable storage medium according to claim 20, wherein the plurality of input elements includes at least one of a touch-sensitive element, a keyboard, a keypad, a roller button, a second imaging element, an audio-capturing element, a power button, a power charging element, or a physical switch.