Delay of display event based on user gaze

• US 9,189,064 B2
• Filed: 09/05/2012
• Issued: 11/17/2015
• Est. Priority Date: 09/05/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chaudhri, Imran, Lemay, Stephen O.
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): LAM, VINH TANG
• Application Number:
  US13/604,462
• Publication Number:
  US 20140062853A1
• Time in Patent Office: 1,168 Days
• Field of Search: 345 7- 9, 345156-178, 715/781
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Methods and systems of delaying the execution of a display event based on a detected user gaze are provided. Display events may be generated and executed to change a user interface of a display. For example, an autocorrect algorithm can automatically replace a typed word with a corrected word in a text field, generating a display event that causes the corrected word to be displayed instead of the typed word. Such a display event may be executed as soon as possible after its generation. However, a gaze detection device can obtain information that indicates a user is not looking at the typed word on the display. In such a situation, it may be more intuitive to delay the execution of the display event until the gaze information indicates that the user is looking at the typed word.

Claims

1. A method comprising:

at a computer system with a display and a gaze detection device;

preparing to display, on the display, a display event associated with a target region of the display;

obtaining, from the gaze detection device, user gaze information including a gaze region on the display;

comparing the gaze region and the target region;

after comparing the gaze region and the target region;

in accordance with a determination that the gaze region and the target region do not overlap, delaying display of the display event on the display;

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is less than the threshold size, delaying display of the display event on the display; and

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is greater than the threshold size, executing the display event on the display without the delay.

2. The method of claim 1, further comprising:

obtaining additional user gaze information; and

executing the display event based on the additional user gaze information.

3. The method of claim 2, wherein executing the display event includes one or more of adding a user interface object to a display, removing a user interface object from a display, and animating a user interface object on a display.

4. The method of claim 1, wherein obtaining the target region includes obtaining a bounding box of the display event.

5. The method of claim 1, wherein the target region includes the entire display.

6. The method of claim 1, further comprising:

queuing the display event on an event queue;

wherein delaying the execution of the display event includes queuing the display event further back on the event queue.

7. The method of claim 1, further comprising:

obtaining an image from the gaze detection device; and

processing the image obtained from the gaze detection device;

wherein the user gaze information is obtained based on the processed image.

8. An electronic device, comprising:

a display;

a gaze detection device;

a processor to execute instructions; and

a memory coupled with the processor to store instructions, which when executed by the processor, cause the processor to perform operations to generate an application programming interface (API) that allows an API-calling component to perform the following operations;

preparing to display, on the display, a display event associated with a target region of the display;

obtaining, from the gaze detection device, user gaze information including a gaze region on the display;

comparing the gaze region and the target region;

after comparing the gaze region and the target region;

in accordance with a determination that the gaze region and the target region do not overlap, delaying display of the display event on the display;

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is less than the threshold size, delaying display of the display event on the display; and

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is greater than the threshold size, executing the display event on the display without the delay.

9. The electronic device of claim 8, the operations further comprising:

obtaining additional user gaze information; and

executing the display event based on the additional user gaze information.

10. The electronic device of claim 9, wherein executing the display event includes one or more of adding a user interface object to the display, removing a user interface object from the display, and animating a user interface object on the display.

11. The electronic device of claim 8, wherein obtaining the target region includes obtaining a bounding box of the display event.

12. The electronic device of claim 8, wherein the target region includes the entire display.

13. The electronic device of claim 8, the operations further comprising:

queuing the display event on an event queue;

wherein delaying the execution of the display event includes queuing the display event further back on the event queue.

14. The electronic device of claim 8, the operations further comprising:

obtaining an image from the gaze detection device; and

processing the image obtained from the gaze detection device;

wherein the user gaze information is obtained based on the processed image.

15. A non-transitory computer readable storage medium having stored therein instructions, which when executed by a device having a display and a gaze detection device, cause the device to perform a method comprising:

preparing to display, on the display, a display event associated with a target region of the display;

obtaining, from the gaze detection device, user gaze information including a gaze region on the display;

comparing the gaze region and the target region;

after comparing the gaze region and the target region;

in accordance with a determination that the gaze region and the target region do not overlap, delaying display of the display event on the display;

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is less than the threshold size, delaying display of the display event on the display; and

in accordance with a determination that the gaze region and the target region do overlap and that the size of the overlapping area is greater than the threshold size, executing the display event on the display without the delay.

16. The non-transitory computer readable storage medium of claim 15, the method further comprising:

obtaining additional user gaze information; and

executing the display event based on the additional user gaze information.

17. The non-transitory computer readable storage medium of claim 16, wherein executing the display event includes one or more of adding a user interface object to the display, removing a user interface object from the display, and animating a user interface object on the display.

18. The non-transitory computer readable storage medium of claim 15, the method further comprising:

queuing the display event on an event queue;

wherein delaying the execution of the display event includes queuing the display event further back on the event queue.

19. The non-transitory computer readable storage medium of claim 15, the method further comprising:

obtaining an image from the gaze detection device; and

processing the image obtained from the gaze detection device;

wherein the user gaze information is obtained based on the processed image.

20. The non-transitory computer readable storage medium of claim 15, wherein obtaining the target region includes obtaining a bounding box of the display event.

21. The non-transitory computer readable storage medium of claim 15, wherein the target region includes the entire display.