APPARATUS AND METHODS FOR DYNAMICALLY CORRELATING VIRTUAL KEYBOARD DIMENSIONS TO USER FINGER SIZE

• US 20110254865A1
• Filed: 04/16/2010
• Published: 10/20/2011
• Est. Priority Date: 04/16/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: HUANG, Joseph, BERNARTE, Joel, DOBSON, Janet L., YEE, Jadine N., MANDYAM, Giridhar D.
• Granted Patent: US 8,982,160 B2
• Time in Patent Office: Days
• US Class Current: 345/661
• CPC Class Codes: G06F 2203/04805 Virtual magnifying lens, i.e. window or frame movable on top of displayed information to enlarge it for better reading or selection
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Embodiments provide user interface for computing devices equipped with a touchscreen user interface/display and a digital camera that enhances a portion of a displayed image within a user'"'"'s gaze. A user may calibrate their mobile device by touching a portion of the touchscreen with one or more fingers and following a moving image on the display with their eyes. The mobile device may track where a user is looking, and if the user is looking at the mobile device display, a portion of the display in the vicinity of the user'"'"'s gaze may be enhanced in size. In an embodiment, if the user is looking at a virtual keyboard, key icons near the user'"'"'s gaze may be increased in size commensurate with the user'"'"'s finger tip size. The in this manner, a user can accurately select individual keys in a virtual keyboard that fits within a mobile device display.

Claims

1. A method for providing a user interface on a computing device equipped with a touchscreen display and a digital camera configured to be able to image a user'"'"'s face when the user is viewing the touchscreen display, comprising:

obtaining a digital image of eyes of a user of the computing device with the digital camera;

determining a location of the user'"'"'s gaze based on the obtained digital image;

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking when it is determined that the user'"'"'s gaze is directed to a portion of the touchscreen display.

2. The method of claim 1, wherein enlarging a portion of an image presented on the touchscreen display comprises enlarging the portion of the image presented on the touch green display based upon a calibration factor related to a size of the user'"'"'s finger.

3. The method of claim 2, further comprising enlarging a user interface icon when the icon is near the determined location of the user'"'"'s gaze on the touchscreen display.

4. The method of claim 3, further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen to match the enlarged icon.

5. The method of claim 1, wherein:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a virtual keyboard image on the touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen to match the enlarged key icons.

6. The method of claim 1, further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating a finger size calibration factor based upon the measured dimensioned of the touch area; and

storing the calculated finger sized calibration factor in memory.

7. The method of claim 1, wherein determining a location of the user'"'"'s gaze based on the obtained digital image comprises applying a rule to the obtained digital image, the method further comprising:

displaying an icon on the touchscreen display;

obtaining a digital image of the eyes of the user with the digital camera;

determining a location of the user'"'"'s gaze by applying the rule to the obtained digital image;

adjusting the rule based on an actual location of the displayed icon on the touchscreen display; and

storing the adjusted rule in memory.

8. The method of claim 1, wherein:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a vicinity of a displayed hyperlink; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging displayed hyperlinks located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen to match the enlarged displayed hyperlinks.

9. The method of claim 1, wherein the computing device is a mobile device.

10. The method of claim 1, wherein the computing device is a personal computer.

11. A computing device, comprising:

a processor;

a memory coupled to the processor;

a digital camera coupled to the processor and configured to be able to image eyes of a user of the computing device; and

a touchscreen display coupled to the processor,

wherein the processor is configure with processor-executable instructions to perform operations comprising;

obtaining a digital image of the eyes of a user of the computing device with the digital camera;

determining a location of the user'"'"'s gaze based on the obtained digital image;

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking when it is determined that the user'"'"'s gaze is directed to a portion of the touchscreen display.

12. The computing device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that enlarging a portion of an image presented on the touchscreen display comprises enlarging the portion of the image presented on the touch green display based upon a calibration factor related to a size of the user'"'"'s finger.

13. The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations further comprising enlarging a user interface icon when the icon is near the determined location of the user'"'"'s gaze on the touchscreen display.

14. The computing device of claim 13, wherein the processor is configured with processor-executable instructions to perform operations further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen to match the enlarged icon.

15. The computing device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a virtual keyboard image on the touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen to match the enlarged key icons.

16. The computing device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating a finger size calibration factor based upon the measured dimensioned of the touch area; and

storing the calculated finger sized calibration factor in memory.

17. The computing device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that determining a location of the user'"'"'s gaze based on the obtained digital image comprises applying a rule to the obtained digital image, and

wherein the processor is configured with processor-executable instructions to perform operations further comprising:

displaying an icon on the touchscreen display;

obtaining a digital image of the eyes of the user with the digital camera;

determining a location of the user'"'"'s gaze by applying the rule to the obtained digital image;

adjusting the rule based on an actual location of the displayed icon on the touchscreen display; and

storing the adjusted rule in memory.

18. The computing device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a vicinity of a displayed hyperlink; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging displayed hyperlinks located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen to match the enlarged displayed hyperlinks.

19. The computing device of claim 11, wherein the computing device is a mobile device.

20. The computing device of claim 11, wherein the computing device is a personal computer.

21. A computing device, comprising:

a touchscreen display;

means for obtaining a digital image of the eyes of a user of the computing device;

means for determining a location of the user'"'"'s gaze based on the obtained digital image;

means for determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display; and

means for enlarging a portion of an image presented on the touchscreen display near where the user is looking when it is determined that the user'"'"'s gaze is directed to a portion of the touchscreen display.

22. The computing device of claim 21, wherein means for enlarging a portion of an image presented on the touchscreen display comprises means for enlarging the portion of the image presented on the touch green display based upon a calibration factor related to a size of the user'"'"'s finger.

23. The computing device of claim 22, further comprising means for enlarging a user interface icon when the icon is near the determined location of the user'"'"'s gaze on the touchscreen display.

24. The computing device of claim 23, further comprising means for adjusting a mapping of displayed icons to coordinates of the touchscreen to match the enlarged icon.

25. The computing device of claim 21, wherein:

means for determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises means for determining whether the user'"'"'s gaze is directed to a virtual keyboard image on the touchscreen display; and

means for enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

means for enlarging key icons within the virtual keyboard image located near where the user is looking based upon a user finger sized calibration factor; and

means for adjusting a mapping of displayed key icons to coordinates of the touchscreen to match the enlarged key icons.

26. The computing device of claim 21, further comprising:

means for measuring a dimension of a touch area on the touchscreen display;

means for calculating a finger size calibration factor based upon the measured dimensioned of the touch area; and

means for storing the calculated finger sized calibration factor in memory.

27. The computing device of claim 21, wherein means for determining a location of the user'"'"'s gaze based on the obtained digital image comprises means for applying a rule to the obtained digital image,

the computing device further comprising:

means for displaying an icon on the touchscreen display;

means for obtaining a digital image of the eyes of the user with the digital camera;

means for determining a location of the user'"'"'s gaze by applying the rule to the obtained digital image;

means for adjusting the rule based on an actual location of the displayed icon on the touchscreen display; and

means for storing the adjusted rule in memory.

28. The computing device of claim 21, wherein:

means for determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises means for determining whether the user'"'"'s gaze is directed to a vicinity of a displayed hyperlink; and

means for enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

means for enlarging displayed hyperlinks located near where the user is looking based upon a user finger sized calibration factor; and

means for adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen to match the enlarged displayed hyperlinks.

29. The computing device of claim 21, wherein the computing device is a mobile device.

30. The computing device of claim 21, wherein the computing device is a personal computer.

31. A processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor to perform operations comprising:

obtaining a digital image of the eyes of a user of the computing device with a digital camera;

determining a location of the user'"'"'s gaze based on the obtained digital image;

determining whether the user'"'"'s gaze is directed to a portion of a touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking when it is determined that the user'"'"'s gaze is directed to a portion of the touchscreen display.

32. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that enlarging a portion of an image presented on the touchscreen display comprises enlarging the portion of the image presented on the touch green display based upon a calibration factor related to a size of the user'"'"'s finger.

33. The processor-readable storage medium of claim 32, wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising enlarging a user interface icon when the icon is near the determined location of the user'"'"'s gaze on the touchscreen display.

34. The processor-readable storage medium of claim 33, wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising adjusting a mapping of displayed icons to coordinates of the touchscreen to match the enlarged icon.

35. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a virtual keyboard image on the touchscreen display; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging key icons within the virtual keyboard image located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of displayed key icons to coordinates of the touchscreen to match the enlarged key icons.

36. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising:

measuring a dimension of a touch area on the touchscreen display;

calculating a finger size calibration factor based upon the measured dimensioned of the touch area; and

storing the calculated finger sized calibration factor in memory.

37. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that determining a location of the user'"'"'s gaze based on the obtained digital image comprises applying a rule to the obtained digital image, and

wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising:

displaying an icon on the touchscreen display;

obtaining a digital image of the eyes of the user with the digital camera;

determining a location of the user'"'"'s gaze by applying the rule to the obtained digital image;

adjusting the rule based on an actual location of the displayed icon on the touchscreen display; and

storing the adjusted rule in memory.

38. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that:

determining whether the user'"'"'s gaze is directed to a portion of the touchscreen display comprises determining whether the user'"'"'s gaze is directed to a vicinity of a displayed hyperlink; and

enlarging a portion of an image presented on the touchscreen display near where the user is looking comprises;

enlarging displayed hyperlinks located near where the user is looking based upon a user finger sized calibration factor; and

adjusting a mapping of the displayed hyperlinks to coordinates of the touchscreen to match the enlarged displayed hyperlinks.

39. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to be executed by a processor of a mobile device.

40. The processor-readable storage medium of claim 31, wherein the stored processor-executable instructions are configured to be executed by a processor of a personal computer.