Method for user input from alternative touchpads of a handheld computerized device

• US 9,311,724 B2
• Filed: 02/19/2013
• Issued: 04/12/2016
• Est. Priority Date: 04/23/2010
• Status: Expired due to Fees

• Current Assignee: Handscape Inc., A Delaware Corporation
• Original Assignee: Handscape Inc., A Delaware Corporation
• Inventors: Luo, Tong
• Primary Examiner(s): Ma, Calvin C
• Application Number:
  US13/770,791
• Publication Number:
  US 20130155070A1
• Time in Patent Office: 1,148 Days
• Field of Search: 345/158, 345/173, 715/701, 715/702
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1643 the display being associated to a digitizer, e.g. laptops that can be used as penpads details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics

Abstract

The present invention relates to a handheld computerized device with a bit mapped display screen on the front panel, and a touchpad installed in an alternative location. More particularly, the invention relates to a method and graphical user interface that can, for example, enable the user to see the user'"'"'s finger position and motion from behind the device superimposed upon a virtual keyboard layout on the front panel. This can allow the user to use a touchpad keypad on the back of the device to input keystrokes and mouse actions, and these will be reflected on the display screen on the front of the handheld computerized device as “virtual fingers” or equivalent. The system operates by inputting touchpad data into a software biomechanical and anatomical model of the human hand, optionally supplemented by video or image information to refine this model.

Claims

1. A method of creating a virtual image of at least portions of the user'"'"'s hand or hands while operating a handheld computerized device, said handheld computerized device comprising at least one touchpad, at least one graphics display screen, at least one image sensor configured to photograph at least part of the region of space proximate said at least one touchpad, but not configured to photograph the surface of said touchpad, at least one processor, memory, and software, said method comprising:

obtaining data on the location and movement of the user'"'"'s fingers and/or hand using said touchpad, said user'"'"'s fingers and/or hand being positioned in an arbitrary manner with respect to said touchpad when contacting said touchpad, said data not being associated with an image of the user'"'"'s fingers from said image sensor, said at least one image sensor comprising at least one video camera;

said touchpad being located in a location of said handheld computerized device that is different from the location of said at least one display screen;

analyzing said data from said touchpad on the location and movement of said user'"'"'s fingers and/or hand according to a model of a human hand, and assigning said data on the location and movement of said user'"'"'s fingers and/or hand to specific fingers on said model of said human hand, thereby making predictions as to the location of the user'"'"'s hand and fingers;

using said model of said human hand, and said predictions of the location of the user'"'"'s hand and fingers, to compute a graphical representation of at least said user'"'"'s fingers;

further using images of portions of the user'"'"'s hand or hands that do not contain images pertaining to the position of the user'"'"'s fingers to further supplement or refine said model of said human hand; and

displaying said graphical representation of at least said user'"'"'s fingers on said at least one graphics display screen of said handheld computerized device.

2. The method of claim 1, wherein prior to subsequent use, the model of a human hand is calibrated in a calibration process by obtaining at least one full or partial image of the user hand or hands.

3. The method of claim 2, further analyzing said at least one full or partial image of the user'"'"'s hand or hands to compute one or more user parameters of said model of said human hand;

and using said one or more user parameters of said model of said human hand to replace one or more default parameters of said model of said human hand.

4. The method of claim 3, wherein said one or more user parameters comprise the length of at least one or more distal phalanges or intermediate phalanges or proximate phalanges or metacarpals of said user'"'"'s hand or hands.

5. The method of claim 1,

wherein the position of at least the distal phalanges or fingertips of said user'"'"'s fingers are not photographed by said at least one video camera when said user is touching said touchpad;

wherein said model of a human hand is further supplemented or refined by real-time video images of at least non-finger regions of said human hand obtained from said at least one camera.

6. The method of claim 1, wherein said touchpad is located on a side of said handheld computerized device that is different from the side of said handheld device that holds said at least one graphics display screen.

7. The method of claim 1, wherein said touchpad is capable of detecting the proximity of said user'"'"'s fingers above the surface of said touchpad;

wherein said touchpad is capable of distinguishing the distance between said user'"'"'s fingers and the surface of said touchpad when said distance is greater than zero.

8. The method of claim 1, in which said software comprises calibration software to enable the model of said human hand to be calibrated to the touchpad data to more accurately match biomechanical and anatomical characteristics of said user'"'"'s fingers and/or hand;

and

wherein said biomedical and anatomical characteristics comprise at least the length of the user'"'"'s fingers, thereby allowing said model of said human hand to determine the length of said user'"'"'s fingers for a plurality of different hand configurations.

9. The method of claim 1, used to assist user data entry into said handheld computerized device, further comprising:

displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device;

wherein said at least one data entry location is a graphical display of a keyboard or keypad comprised of a plurality of data entry locations, or wherein said at least one data entry location is a hyperlink.

10. The method of claim 9, in which said at least one data entry location is highlighted on said at least one graphics display screen whenever said computerized device determines that at least one finger on said user'"'"'s hand has left the touchpad and the position and motion history of said at least one finger is consistent with a capability said at least one finger on said user'"'"'s hand to strike a position on said touchpad that is consistent with the location of said at least one data entry location on said at least one graphics display screen;

wherein said highlighting commences before said at least one finger has struck the next position on said keypad.

11. The method of claim 9, in which said at least one data entry location comprises alphanumeric letter data entry locations, and wherein said software comprises predictive typing software configured to deduce the most probable letter or letters that the user is likely to type next, and the most probable finger or fingers that the user will use to type this most probable letter, and said computerized device utilizes keystroke predictions made by said predictive typing software, along with said data on the location and movement of said user'"'"'s fingers and/or hand using said touchpad, to compute the graphical representation of at least said user'"'"'s fingers.

12. The method of claim 9 wherein either the graphical representation of at least said user'"'"'s fingers on said at least one graphics display screen is semi-transparent so as to enable the graphical display and said user'"'"'s fingers to be simultaneously observed;

or

wherein said at least one data entry location is a graphical display of a semi-transparent keyboard or semi-transparent keypad, thus allowing details of a current application running on said graphics display to be visualized along with said semi-transparent keyboard or semi-transparent keypad.

13. A method of creating a virtual image of at least portions of the user'"'"'s hand or hands while operating a handheld computerized device, said handheld computerized device comprising at least one touchpad, at least one graphics display screen, at least one image sensor configured to photograph at least part of the region of space proximate said at least one touchpad, but not configured to photograph the surface of said touchpad, at least one processor, memory, and software, said method comprising:

further displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device;

obtaining data on the location and movement of the user'"'"'s fingers and/or hand using said touchpad, said user'"'"'s fingers and/or hand being positioned in an arbitrary manner with respect to said touchpad when contacting said touchpad, said data not being associated with an image of the user'"'"'s fingers from said image sensor, said at least one image sensor comprising at least one video camera;

said touchpad being located in a location of said handheld computerized device that is different from the location of said at least one display screen;

analyzing said data from said touchpad on the location and movement of said user'"'"'s fingers and/or hand according to a model of a human hand, assigning said data on the location and movement of said user'"'"'s fingers and/or hand to specific fingers on said model of said human hand, thereby making predictions as to the location of the user'"'"'s hand and fingers;

wherein the model of a human hand is supplemented or refined by obtaining at least one full or partial image of the user hand or hands;

using said model of said human hand, and said predictions of the location of the user'"'"'s hand and fingers, to compute a graphical representation of at least said user'"'"'s fingers;

further using images of portions of the user'"'"'s hand or hands that do not contain images pertaining to the position of the user'"'"'s fingers to further supplement or refine said model of said human hand;

displaying said graphical representation of at least said user'"'"'s fingers on said at least one graphics display screen of said handheld computerized device;

wherein distances between said graphical representation of at least said user'"'"'s fingers on said at least one graphics display screen, and said at least one data entry location, give information to said user to facilitate said user to position said user'"'"'s fingers and/or hand on said at least one touchpad to enter user data into said at least one data entry location.

14. The method of claim 13, further analyzing said at least one full or partial image of the user'"'"'s hand or hands to compute one or more user parameters of said model of said human hand;

and using said one or more user parameters of said model of said human hand to replace one or more default parameters of said model of said human hand;

wherein said one or more user parameters comprise the length of at least one or more distal phalanges or intermediate phalanges or proximate phalanges or metacarpals of said user'"'"'s hand or hands.

15. The method of claim 13,

wherein the position of at least the distal phalanges or fingertips of said user'"'"'s fingers are not photographed by said at least one video camera when said user is touching said touchpad;

wherein said model of a human hand is further supplemented or refined by real-time video images of at least non-finger regions of said human hand obtained from said at least one camera.

16. The method of claim 13, in which said software comprises calibration software to enable the model of said human hand to be calibrated to the touchpad data to more accurately match biomechanical and anatomical characteristics of said user'"'"'s fingers and/or hand;

and

wherein said biomedical and anatomical characteristics comprise at least the length of the user'"'"'s fingers, thereby allowing said model of said human hand to determine the length of said user'"'"'s fingers for a plurality of different hand configurations.

17. The method of claim 13, used to assist user data entry into said handheld computerized device, further comprising:

displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device;

wherein said at least one data entry location is a graphical display of a keyboard or keypad comprised of a plurality of data entry locations, or wherein said at least one data entry location is a hyperlink.

18. The method of claim 17, in which said at least one data entry location is highlighted on said at least one graphics display screen whenever said computerized device determines that at least one finger on said user'"'"'s hand has left the touchpad and the position and motion history of said at least one finger is consistent with a capability said at least one finger on said user'"'"'s hand to strike a position on said touchpad that is consistent with the location of said at least one data entry location on said at least one graphics display screen;

wherein said highlighting commences before said at least one finger has struck the next position on said keypad.

19. The method of claim 17, in which said at least one data entry location comprises alphanumeric letter data entry locations, and wherein said software comprises predictive typing software configured to deduce the most probable letter or letters that the user is likely to type next, and the most probable finger or fingers that the user will use to type this most probable letter, and said computerized device utilizes keystroke predictions made by said predictive typing software, along with said data on the location and movement of said user'"'"'s fingers and/or hand using said touchpad, to compute the graphical representation of at least said user'"'"'s fingers.

20. The method of claim 17 wherein either the graphical representation of at least said user'"'"'s fingers on said at least one graphics display screen is semi-transparent so as to enable the graphical display and said user'"'"'s fingers to be simultaneously observed;

or

wherein said at least one data entry location is a graphical display of a semi-transparent keyboard or semi-transparent keypad, thus allowing details of a current application running on said graphics display to be visualized along with said semi-transparent keyboard or semi-transparent keypad.