Fingertip location determinations for gesture input

• US 9,690,384 B1
• Filed: 09/26/2012
• Issued: 06/27/2017
• Est. Priority Date: 09/26/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Noble, Isaac S., Fotland, David A.
• Primary Examiner(s): Chang, Kent
• Assistant Examiner(s): Shah, Sujit
• Application Number:
  US13/627,261
• Time in Patent Office: 1,735 Days
• Field of Search: 345173
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3262 Power saving in digitizer or tablet
• CPC Class Codes: G06F 2200/1636 Sensing arrangement for detection of a tap gesture on the housing
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes

Abstract

A user can use a finger, or other such object, to provide input to a computing device. The finger can be positioned and/or oriented at a distance from the device, and the device can determine an input that the user is attempting to provide based on a position, motion, or gesture associated with the finger. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use another sensor (e.g., EMF) to determine one or more potential locations of the fingertip. The image information then can be used to determine which of the potential locations actually corresponds to the fingertip by analyzing that portion of the finger or hand represented in the image information.

Claims

1. A computer-implemented method, comprising:

capturing first image data using a camera of a computing device;

determining that there is one fingertip within a threshold distance of the computing device represented in the first image data;

activating a self-capacitance mode of the computing device, the computing device capable of operating in at least the self-capacitance mode and mutual capacitance mode;

detecting, using at least one capacitance sensor of the computing device operating in the self-capacitance mode, a position of the computing device closest to the fingertip when the computing device is not contacting the fingertip;

capturing second image data using the camera;

determining that there is more than one fingertip within the threshold distance represented in the second image;

activating the mutual capacitance mode of the computing device;

detecting, using at least one capacitance sensor of the computing device operating in the mutual capacitance mode, a first position of the computing device closest to a first fingertip when the computing device is not contacting the first fingertip; and

detecting, using at least one capacitance sensor of the computing device operating in the mutual capacitance mode, a second position of the computing device closest to a second fingertip when the computing device is not contacting the second fingertip.

2. The computer-implemented method of claim 1, further comprising:

determining orientation information associated with at least one portion of a hand represented in the first image data; and

comparing the orientation information to a digital finger model.

3. The computer-implemented method of claim 1, further comprising:

tracking a movement of the fingertip within a period of time that the computing device is operating in the self-capacitance mode; and

determining a gesture corresponding to the movement.

4. The computer-implemented method of claim 3, further comprising:

providing an input signal to an application executing on the computing device, the input signal corresponding to the gesture.

5. A computer-implemented method, comprising:

capturing first image data using one or more cameras of a computing device;

determining that there is one object within a threshold distance of the computing device represented in the first image data;

activating a self-capacitance mode of the computing device;

detecting, using one or more capacitance sensors of the computing device within a first period of time that the computing device is operating in the self-capacitance mode, a position of the computing device closest to the object when the computing device is not contacting the object;

capturing second image data using the one or more cameras;

determining that there is more than one object within the threshold distance represented in the second image;

activating the mutual capacitance mode of the computing device;

detecting, using one or more capacitance sensors within a second period of time that the computing device is operating in the mutual capacitance mode, a first position of the computing device closest to a first object when the computing device is not contacting the first object; and

detecting, within the second period of time, a second position of the computing device closest to a second object when the computing device is not contacting the second object.

6. The computer-implemented method of claim 5, further comprising:

determining that there is one finger or one stylus within the threshold distance represented in the first image data.

7. The computer-implemented method of claim 5, further comprising:

determining orientation information associated with the object represented in the first image data; and

comparing the orientation information to orientation information associated with an object model.

8. The computer-implemented method of claim 5, further comprising:

detecting a plurality of potential positions of the object within the threshold distance; and

rejecting at least one of the plurality of potential positions that corresponds to noise.

9. The computer-implemented method of claim 5, further comprising:

deactivating one or more capacitance sensors of the computing device positioned away from the position closest to the object.

10. The computer-implemented method of claim 5, further comprising:

providing the first position and the second position as a single input to the computing device.

11. The computer-implemented method of claim 5, further comprising:

capturing the first image data as at least one of still image information, stereoscopic image information, infrared image information, or video information.

12. The computer-implemented method of claim 5, further comprising:

storing at least one previously detected position of the computing device closest to the object; and

initiating detection of at least one portion of the object represented in the first image data based at least in part upon the at least one previously detected position.

13. The computer-implemented method of claim 5, further comprising:

tracking a movement of the object within a period of time that the computing device is operating in the self-capacitance mode; and

determining a gesture corresponding to the movement.

14. The computer-implemented method of claim 5, further comprising:

determining physical characteristics of a specific user; and

detecting at least one portion of the object represented in the first image data based at least in part upon the physical characteristics.

15. The computer-implemented method of claim 5, further comprising:

detecting a change in orientation of the computing device; and

detecting at least one portion of the object represented in the first image data based at least in part upon the change in orientation.

16. A computing device, comprising:

one or more processors;

one of more cameras;

one or more capacitance sensors; and

a memory device including instructions that, when executed by the one or more processors, cause the computing device to;

capture the first image data using one or more cameras of the computing device;

determine that there is one object within a threshold distance of the computing device represented in the first image data;

activate a self-capacitance mode of the computing device;

detect, using the one or more capacitance sensors within a period of time that the computing device is operating in the self-capacitance mode, a position of the computing device closest to the object when the computing device is not contacting the object;

capture second image data using the one or more cameras;

determine that there is more than one object within the threshold distance represented in the second image data;

activate a mutual capacitance mode of the computing device;

detect, using the one or more capacitance sensors within a second period of time that the computing device is operating in a mutual capacitance mode, a first position of the computing device closest to a first object when the computing device is not contacting the first object; and

detect, within the second period of time, a second position of the computing device closest to a second object when the computing device is not contacting the second object.

17. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

determine orientation information associated with the object represented in the first image data; and

compare the orientation information to orientation information associated with an object model.

18. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

track a movement of the object within a period of time that the computing device is operating in the self-capacitance mode; and

determine a gesture corresponding to the movement.

19. A non-transitory computer-readable storage medium including instructions that, when executed by one or more processors of a computing device, cause the computing device to:

capture first image data using one or more camera of the computing device;

determine that there is more than one object within a threshold distance of the computing device represented in the first image data;

activate a self-capacitance mode of the computing device;

detect, using one or more capacitance sensor of the computing device within a first period of time that the computing device is operating in a self-capacitance mode, a position of the computing device closest to the object when the computing device is not contacting the object;

capture second image data using the one or more cameras;

determine that there is more than one object within the threshold distance represented in the second image data;

activate a mutual capacitance mode of the computing device;

detect, using the one or more capacitance sensors within a second period of time that the computing device is operating in the mutual capacitance mode, a first position of the computing device closest to a first object when the computing device is not contacting the first object; and

detect, within the second period of time, a second position of the computing device closest to a second object when the computing device is not contacting the second object.

20. The non-transitory computer-readable storage medium of claim 19, wherein the instructions when executed further cause the computing device to:

determine orientation information associated with the object represented in the first image data; and

compare the orientation information to orientation information associated with an object model.

21. The non-transitory computer-readable storage medium of claim 19, wherein the instructions when executed further cause the computing device to:

track a movement of the object within a period of time that the computing device is operating in the self-capacitance mode; and

determine a gesture corresponding to the movement.

22. The non-transitory computer-readable storage medium of claim 21, wherein the instructions when executed further cause the computing device to:

provide an input signal to an application executing on the computing device, the input signal corresponding to the gesture.