Apparatus and methods for dynamically correlating virtual keyboard dimensions to user finger size

• US 8,982,160 B2
• Filed: 04/16/2010
• Issued: 03/17/2015
• Est. Priority Date: 04/16/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Yee, Jadine N., Bernarte, Joel, Mandyam, Giridhar D., Huang, Joseph, Dobson, Janet L.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wu, Sing-Wai
• Application Number:
  US12/761,479
• Publication Number:
  US 20110254865A1
• Time in Patent Office: 1,796 Days
• US Class Current: 345/661
• CPC Class Codes: G06F 2203/04805 Virtual magnifying lens, i.e. window or frame movable on top of displayed information to enlarge it for better reading or selection
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Embodiments provide a user interface for computing devices equipped with a touchscreen user interface/display and a digital camera that enhances a portion of a displayed image within a user'"'"'s gaze. A user may calibrate their mobile device by touching a portion of the touchscreen with one or more fingers and following a moving image on the display with their eyes. The mobile device may track where a user is looking, and if the user is looking at the mobile device display, a portion of the display in the vicinity of the user'"'"'s gaze may be enhanced in size. In an embodiment, if the user is looking at a virtual keyboard, key icons near the user'"'"'s gaze may be increased in size commensurate with the user'"'"'s finger tip size. In this manner, a user can accurately select individual keys in a virtual keyboard that fits within a mobile device display.

Claims

1. A method for providing a user interface on a computing device equipped with a touchscreen display and a digital camera configured to be able to image the face of a user when the user is viewing the touchscreen display, comprising:

displaying a moving icon on the touchscreen display for a predetermined eye-tracking training period;

obtaining a first plurality of images of the eyes of the user with the digital camera during the predetermined eye-tracking training period and when the face of the user is at a first distance from the digital camera;

obtaining a second plurality of images of the eyes of the user with the digital camera when the face of the user is at a second distance from the digital camera;

comparing the obtained first plurality of images to known locations of the moving icon during the predetermined eye-tracking training period to determine an image analysis rule;

comparing the obtained second plurality of images to known locations of the moving icon to adjust the image analysis rule;

storing the adjusted image analysis rule in memory;

obtaining a digital image of the eyes of the user of the computing device with the digital camera after the predetermined eye-tracking training period;

determining a location of a gaze of the user based on the obtained digital image and the adjusted image analysis rule;

determining whether the gaze of the user is directed to a portion of the touchscreen display containing an image element without requiring additional user interaction with the user interface;

determining a center of the portion of the touchscreen display to which the gaze of the user is directed; and

enlarging the image element on the touchscreen display based upon a finger size calibration factor related to a size of a finger of the user and a distance from the center of the portion of the touchscreen display to which the gaze of the user is directed in response to determining that the gaze of the user is directed to the portion of the touchscreen display containing the image element.

2. The method of claim 1, further comprising enlarging a user interface icon when the user interface icon is near the determined location of the gaze of the user on the touchscreen display.

3. The method of claim 2, further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen display to match the enlarged user interface icon.

4. The method of claim 1, wherein:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a virtual keyboard image on the touchscreen display; and

enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen display to match the enlarged key icons.

5. The method of claim 1, further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating the finger size calibration factor based upon the measured dimension of the touch area; and

storing the calculated finger size calibration factor in the memory.

6. The method of claim 1, wherein:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a vicinity of displayed hyperlinks; and

enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging the displayed hyperlinks located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen display to match the enlarged displayed hyperlinks.

7. The method of claim 1, wherein the computing device is a mobile device.

8. The method of claim 1, wherein the computing device is a personal computer.

9. A computing device, comprising:

a processor;

a memory coupled to the processor;

a digital camera coupled to the processor and configured to be able to image the eyes of a user of the computing device; and

a touchscreen display coupled to the processor,

wherein the processor is configured with processor-executable instructions to perform operations comprising;

displaying a moving icon on the touchscreen display for a predetermined eye-tracking training period;

obtaining a first plurality of images of the eyes of the user with the digital camera during the predetermined eye-tracking training period and when the face of the user is at a first distance from the digital camera;

obtaining a second plurality of images of the eyes of the user with the digital camera when the face of the user is at a second distance from the digital camera;

comparing the obtained first plurality of images to known locations of the moving icon during the predetermined eye-tracking training period to determine an image analysis rule;

comparing the obtained second plurality of images to known locations of the moving icon to adjust the image analysis rule;

storing the adjusted image analysis rule in the memory;

obtaining a digital image of the eyes of the user of the computing device with the digital camera after the predetermined eye-tracking training period;

determining a location of a gaze of the user based on the obtained digital image and the adjusted image analysis rule;

determining whether the gaze of the user is directed to a portion of the touchscreen display containing an image element without requiring additional user interaction with a user interface;

determining a center of the portion of the touchscreen display to which the gaze of the user is directed; and

enlarging the image element on the touchscreen display based upon a finger size calibration factor related to a size of a finger of the user and a distance from the center of the portion of the touchscreen display to which the gaze of the user is directed in response to determining that the gaze of the user is directed to the portion of the touchscreen display containing the image element.

10. The computing device of claim 9, wherein the processor is configured with processor-executable instructions to perform operations further comprising enlarging a user interface icon when the user interface icon is near the determined location of the gaze of the user on the touchscreen display.

11. The computing device of claim 10, wherein the processor is configured with processor-executable instructions to perform operations further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen display to match the enlarged user interface icon.

12. The computing device of claim 9, wherein the processor is configured with processor-executable instructions to perform operations such that:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a virtual keyboard image on the touchscreen display; and

enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of a finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen display to match the enlarged key icons.

13. The computing device of claim 9, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating the finger size calibration factor based upon the measured dimension of the touch area; and

storing the calculated finger size calibration factor in the memory.

14. The computing device of claim 9, wherein the processor is configured with processor-executable instructions to perform operations such that:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a vicinity of displayed hyperlinks; and

enlarging the image element on the touchscreen display based upon a finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging the displayed hyperlinks located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen display to match the enlarged displayed hyperlinks.

15. The computing device of claim 9, wherein the computing device is a mobile device.

16. The computing device of claim 9, wherein the computing device is a personal computer.

17. A computing device, comprising:

means for displaying a moving icon on a touchscreen display for a predetermined eye- tracking training period;

means for obtaining a first plurality of images of eyes of a user with a digital camera during the predetermined eye-tracking training period and when the face of the user is at a first distance from the digital camera;

means for obtaining a second plurality of images of the eyes of the user when the face of the user is at a second distance from the digital camera;

means for comparing the obtained first plurality of images to known locations of the moving icon during the predetermined eye-tracking training period to determine an image analysis rule;

means for comparing the obtained second plurality of images to known locations of the moving icon to adjust the image analysis rule;

means for storing the adjusted image analysis rule in memory;

means for obtaining a digital image of the eyes of the user of the computing device after the predetermined eye-tracking training period;

means for determining a location of a gaze of the user based on the obtained digital image and the adjusted image analysis rule;

means for determining whether the gaze of the user is directed to a portion of the touchscreen display containing an image element without requiring additional user interaction with a user interface;

means for determining a center of the portion of the touchscreen display to which the gaze of the user is directed; and

means for enlarging the image element on the touchscreen display based upon a finger size calibration factor related to a size of a finger of the user and a distance from the center of the portion of the touchscreen display to which the gaze of the user is directed in response to determining that the gaze of the user is directed to the portion of the touchscreen display containing the image element.

18. The computing device of claim 17, further comprising means for enlarging a user interface icon when the user interface icon is near the determined location of the gaze of the user on the touchscreen display.

19. The computing device of claim 18, further comprising means for adjusting a mapping of displayed icons to coordinates of the touchscreen display to match the enlarged user interface icon.

20. The computing device of claim 17, wherein:

means for determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises means for determining whether the gaze of the user is directed to a virtual keyboard image on the touchscreen display; and

means for enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

means for enlarging key icons within the virtual keyboard image located near where the user is looking based upon the finger size calibration factor; and

means for adjusting a mapping of displayed key icons to coordinates of the touchscreen display to match the enlarged key icons.

21. The computing device of claim 17, further comprising:

means for measuring a dimension of a touch area on the touchscreen display;

means for calculating the finger size calibration factor based upon the measured dimension of the touch area; and

means for storing the calculated finger size calibration factor in the memory.

22. The computing device of claim 17, wherein:

means for determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises means for determining whether the gaze of the user is directed to a vicinity of displayed hyperlinks; and

means for enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

means for enlarging the displayed hyperlinks located near where the user is looking based upon the finger size calibration factor; and

means for adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen display to match the enlarged displayed hyperlinks.

23. The computing device of claim 17, wherein the computing device is a mobile device.

24. The computing device of claim 17, wherein the computing device is a personal computer.

25. A non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of a computing device to perform operations comprising:

displaying a moving icon on a touchscreen display for a predetermined eye-tracking training period;

obtaining a first plurality of images of eyes of a user of the computing device with a digital camera during the predetermined eye-tracking training period and when the face of the user is at a first distance from the digital camera;

obtaining a second plurality of images of the eyes of the user with the digital camera when the face of the user is at a second distance from the digital camera;

comparing the obtained first plurality of images to known locations of the moving icon during the predetermined eye-tracking training period to determine an image analysis rule;

comparing the obtained second plurality of images to known locations of the moving icon to adjust the image analysis rule;

storing the adjusted image analysis rule in memory;

obtaining a digital image of the eyes of the user of the computing device with the digital camera after the predetermined eye-tracking training period;

determining a location of a gaze of the user based on the obtained digital image and the adjusted image analysis rule;

determining whether the gaze of the user is directed to a portion of the touchscreen display containing an image element without requiring additional user interaction with a user interface;

determining a center of the portion of the touchscreen display to which the gaze of the user is directed; and

enlarging the image element on the touchscreen display based upon a finger size calibration factor related to a size of a finger of the user and a distance from the center of the portion of the touchscreen display to which the gaze of the user is directed in response to determining that the gaze of the user is directed to the portion of the touchscreen display containing the image element.

26. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to cause the processor to perform operations further comprising enlarging a user interface icon when the user interface icon is near the determined location of the gaze of the user on the touchscreen display.

27. The non-transitory processor-readable storage medium of claim 26, wherein the stored processor-executable instructions are configured to cause the processor to perform operations further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen display to match the enlarged user interface icon.

28. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a virtual keyboard image on the touchscreen display; and

enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen display to match the enlarged key icons.

29. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to cause the processor to perform operations further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating the finger size calibration factor based upon the measured dimension of the touch area; and

storing the calculated finger size calibration factor in the memory.

30. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that:

determining whether the gaze of the user is directed to the portion of the touchscreen display containing the image element comprises determining whether the gaze of the user is directed to a vicinity of displayed hyperlinks; and

enlarging the image element on the touchscreen display based upon the finger size calibration factor related to the size of the finger of the user and the distance from the center of the portion of the touchscreen display to which the gaze of the user is directed comprises;

enlarging the displayed hyperlinks located near where the user is looking based upon the finger size calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen display to match the enlarged displayed hyperlinks.

31. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to be executed by a processor of a mobile device.

32. The non-transitory processor-readable storage medium of claim 25, wherein the stored processor-executable instructions are configured to be executed by a processor of a personal computer.