Electronic devices with gaze detection capabilities

• US 10,025,380 B2
• Filed: 01/17/2014
• Issued: 07/17/2018
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hodge, Andrew, Rosenblatt, Michael
• Primary Examiner(s): HARRIS, DOROTHY H
• Application Number:
  US14/157,909
• Publication Number:
  US 20140132508A1
• Time in Patent Office: 1,642 Days
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: H04N 21/4436 Power management, e.g. shutting down unused components of the receiver power management in computer systems G06F1/3203; hibernate or awake process in computer systems G06F9/4418
• CPC Class Codes: H04W 52/027 by controlling a display operation or backlight unit
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

An electronic device may have gaze detection capabilities that allow the device to detect when a user is looking at the device. The electronic device may implement a power management scheme using the results of gaze detection operations. When the device detects that the user has looked away from the device, the device may dim a display screen and may perform other suitable actions. The device may pause a video playback operation when the device detects that the user has looked away from the device. The device may resume the video playback operation when the device detects that the user is looking towards the device. Gaze detector circuitry may be powered down when sensor data indicates that gazed detection readings will not be reliable or are not needed.

Claims

1. A method for minimizing power consumption in a portable electronic device that has a touch screen display and gaze detection circuitry, the method comprising:

performing a video playback operation in which video is displayed on the touch screen display;

with the gaze detection circuitry, capturing images of a user at a first rate and, while the touch screen display receives touch screen inputs from the user, determining that the user'"'"'s gaze is not directed towards the touch screen display based on the captured images; and

in response to determining, while the touch screen display receives the touch screen inputs from the user, that the user'"'"'s gaze is not directed towards the touch screen display, pausing the video playback operation and reducing the rate of image capture from the first rate to a second rate, wherein in response to a given period of user inactivity after determining that the user'"'"'s gaze is not directed towards the touch screen display, the gaze detection circuitry transitions from capturing images at the second rate to a turned off state.

2. The method defined in claim 1 further comprising:

after it has been determined that the user'"'"'s gaze is not directed towards the touch screen display, determining whether the user'"'"'s gaze has returned to the touch screen display; and

when it has been determined that the user'"'"'s gaze has returned to the touch screen display, resuming the video playback operation.

3. The method defined in claim 1 further comprising:

after it has been determined that the user'"'"'s gaze is not directed towards the touch screen display, determining whether the user'"'"'s gaze has returned to the touch screen display; and

in response to determining that the user'"'"'s gaze has returned to the touch screen display, presenting an on-screen selectable option to resume the video playback operation.

4. The method defined in claim 1 further comprising:

when it has been determined that the user'"'"'s gaze is not directed towards the touch screen display, placing the touch screen display in an operating mode that reduces power consumption by the touch screen display.

5. The method defined in claim 4 wherein the touch screen display is at a first brightness level during the video playback operation and wherein placing the touch screen display in the operating mode comprises reducing the brightness of the touch screen display to a second brightness level.

6. The method defined in claim 5 further comprising:

after it has been determined that the user'"'"'s gaze is not directed towards the touch screen display, determining whether the user'"'"'s gaze has returned to the touch screen display; and

when it has been determined that the user'"'"'s gaze has returned to the touch screen display, increasing the brightness of the touch screen display to the first brightness level.

7. The method defined in claim 4 wherein placing the touch screen display in the operating mode that reduces power consumption comprises turning off the touch screen display.

8. The method defined in claim 1, wherein determining that the user'"'"'s gaze is not directed towards the touch screen display comprises determining that the user'"'"'s gaze is not directed towards the touch screen display while the user is listening to an audio track with the electronic device.

9. The method defined in claim 1, wherein determining that the user'"'"'s gaze is not directed towards the touch screen display comprises determining that the user'"'"'s gaze is not directed towards the touch screen display while the user presses a button on the electronic device.

10. The method defined in claim 1, wherein in response to the given period of user inactivity after determining that the user'"'"'s gaze is not directed towards the touch screen display, the gaze detection circuitry transitions to the turned off state and does not perform gaze detection operations.

11. A method for operating a portable electronic device that has a touch screen display and a camera, the method comprising:

performing a video playback operation in which video is displayed on the touch screen display;

while the touch screen display receives touch screen inputs from the user, determining, with the camera, that a user is not looking at the touch screen display; and

in response to determining, while the touch screen display receives the touch screen inputs from the user, that the user is not looking at the touch screen display, pausing the video playback operation and turning off the camera.

12. The method defined in claim 11, further comprising:

in response to determining, while the touch screen display receives the touch screen inputs from the user, that the user is not looking at the touch screen display, reducing a brightness level of the touch screen display; and

in response to a given period of user inactivity while the touch screen display is at the reduced brightness level, turning off the touch screen display.

13. The method defined in claim 11 further comprising:

performing an audio playback operation that is associated with the video playback operation; and

in response to determining, while the touch screen display receives the touch screen inputs from the user, that the user is not looking at the touch screen display, pausing the audio playback operation and the video playback operation.

14. The method defined in claim 12, wherein the given period of user inactivity is a first given period of user inactivity, and wherein the brightness level is reduced from a normal brightness level to the reduced brightness level, the method further comprising:

in response to a second given period of user inactivity that is longer than the first given period of user inactivity, turning off the touch screen display while the touch screen display is at the normal brightness level.

15. An electronic device comprising:

a touch screen display that displays video in a video playback operation;

an optical sensor that captures images of a user of the electronic device while the touch screen display receives touch screen inputs from the user, wherein the optical sensor captures the images at a given rate; and

circuitry that processes the captured images of the user from the optical sensor to determine whether or not the user is looking at the touch screen display, wherein the circuitry pauses the video playback operation and the optical sensor captures the images of the user at a reduced rate when the circuitry determines, while the touch screen display receives the touch screen inputs, that the user is not looking at the touch screen display based on the captured images of the user, and wherein, in response to a given period of user inactivity after the circuitry determines that the user is not looking at the touch screen display, the optical sensor transitions from capturing the images of the user at the reduced rate to a turned-off state.

16. The electronic device defined in claim 15 further comprising:

an ambient light sensor that measures a brightness level of ambient light and wherein, when the light sensor makes a measurement that is less than a given brightness level, the circuitry suspends data processing operations to determine whether or not the user is looking at the touch screen display.

17. The electronic device defined in claim 15 wherein the touch screen display is disposed on a given surface of the electronic device and wherein the optical sensor comprises a camera disposed on the given surface of the electronic device.

18. The electronic device defined in claim 15 wherein the touch screen display is disposed on a front surface of the electronic device and wherein the optical sensor comprises a camera disposed on the front surface of the electronic device.

19. The electronic device defined in claim 15 wherein the electronic device is a cellular telephone.

20. The electronic device defined in claim 15 wherein the electronic device is a handheld electronic device.

21. The electronic device defined in claim 15 further comprising:

a light source that emits infrared light towards a user'"'"'s face, wherein the optical sensor receives infrared light emitted from the light source and reflected off of the user'"'"'s eyes.

22. The electronic device defined in claim 15 further comprising:

a light source that emits modulated infrared light towards a user'"'"'s face, wherein the optical sensor comprises an optical sensor that is synchronized with the light source to detect reflections of the modulated infrared light.

23. The electronic device defined in claim 15, wherein the circuitry does not determine whether or not the user is looking at the touch screen display while the optical sensor is in the turned-off state.