Vein scanning device for automatic gesture and finger recognition

• US 10,698,497 B2
• Filed: 08/10/2018
• Issued: 06/30/2020
• Est. Priority Date: 09/29/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Brennan, Michael R.
• Primary Examiner(s): Lee, Jr., Kenneth B
• Application Number:
  US16/101,228
• Publication Number:
  US 20190101991A1
• Time in Patent Office: 690 Days
• Field of Search: None
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/14 Vascular patterns
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

This relates to a device capable of automatically determining a user'"'"'s gesture and/or finger positions based on one or more properties of the user'"'"'s veins and methods for operation thereof. The device can include one or more sensors (e.g., a camera) to capture one or more images of the user'"'"'s hand. The device can convert the image(s) to digital representations and can correlate the digital image(s) of the veins to one or more poses. From the pose(s), the device can determine the user'"'"'s hand movements, and one or more gestures and/or finger positions can be determined from the hand movements. The device can interpret the gestures and/or finger positions as one or more input commands, and the device can perform an operation based on the input command(s). Examples of the disclosure include using the user input commands in virtual reality applications.

Claims

1. A method for determining hand gestures by an electronic device, the method comprising:

capturing one or more first images of one or more veins in a hand at a first time;

capturing one or more second images of the one or more veins in the hand at a second time, different from the first time;

determining a first hand pose based on the one or more first images, wherein determining the first hand pose includes correlating the one or more veins in the one or more first images to one or more joints of the hand;

determining a second hand pose based on the one or more second images, wherein determining the second hand pose includes correlating the one or more veins in the one or more second images to one or more joints of the hand; and

determining a gesture based on at least the first hand pose and the second hand pose.

2. The method of claim 1,

wherein the determination of a gesture includes:

detecting one or more differences in properties of the one or more veins in the one or more first images and the one or more second images to determine one or more hand movements, wherein the gesture is further based on the one or more hand movements.

3. The method of claim 2, wherein the properties can be one or more of location, relative spacing, shape, displacement, and blood flow.

4. The method of claim 2, wherein the correlation of the one or more veins in the one or more first images and the correlation of the one or more veins in the one or more second are based on at least a predetermined model.

5. The method of claim 1, further comprising predicting a third hand pose based on the first hand pose and the second hand pose.

6. The method of claim 5, further comprising:

determining whether the prediction was unsuccessful; and

in accordance with the prediction being unsuccessful;

capturing one or more third images of the one or more veins in the hand at a third time; and

determining the third hand pose based at least on one or more third images.

7. The method of claim 5, further comprising:

determining whether the prediction was successful; and

dynamically updating a confidence value based on a determined success of the prediction.

8. The method of claim 7, wherein predicting the third hand pose includes:

determining or receiving an associated confidence value, wherein the prediction is based on the confidence value.

9. The method of claim 1, wherein the determination of the gesture is based on a sequence of poses including the first hand pose and the second hand pose.

10. The method of claim 1, further comprising:

determining the determined gesture was incorrect; and

in accordance with the gesture being incorrect, dynamically increasing a granularity of capturing the one or more first images, the one or more second images, or both.

11. The method of claim 1, further comprising:

determining one or more operations of the electronic device associated with the gesture; and

executing the one or more operations on the electronic device when the gesture is determined.

12. The method of claim 1, further comprising:

determining whether the gesture is included in a library file; and

in accordance with the gesture not being included in the library file, recording information associated with the gesture in the library file,

wherein the information includes one or more of the gesture, the one or more first images, the one or more second images, statistical models associated with the gesture, one or more applications associated with the gesture, and user characteristics.

13. The method of claim 1, further comprising:

updating a library file with the gesture.

14. The method of claim 13, further comprising:

classifying the gesture into a category of gestures; and

recording the gesture in the respective category.

15. The method of claim 12, the method further comprising:

updating an association of the recorded information to a second device operation, wherein the recorded information is associated with a first device operation prior to the update.

16. An electronic device comprising:

one or more sensors, the one or more sensors configured to capture one or more first images of one or more veins in a hand and capture one or more second images of the one or more veins in the hand; and

a processor configured to;

receive the captured one or more first images,

receive the captured one or more second images,

convert the captured one or more first images and the captured one or more second images to a first digital representation and a second digital representation, respectively,

detect one or more differences between the first digital representation and the second digital representation,

determine one or more hand movements based on the one or more differences, wherein the determination of the one or more hand movements includes correlating one or more veins in the one or more first images to one or more joints of the hand,

determine one or more hand gestures based on the determined one or more hand movements, and

perform one or more operations on the electronic device based on the determined one or more hand gestures.

17. The electronic device of claim 16, wherein the one or more sensors are located on one or more of a side of the electronic device, an underside of the electronic device, and a strap of the electronic device.

18. The electronic device of claim 17, wherein the one or more sensors are PPG sensors located on the underside of the electronic device.

19. The electronic device of claim 16, wherein the one or more sensors include a camera located on a crown of the electronic device.

20. The electronic device of claim 16, wherein the one or more sensors are one or more of near-IR sensors, IR sensors, or both.