Systems, apparatuses, and methods for gesture recognition and interaction

• US 10,725,533 B2
• Filed: 09/26/2014
• Issued: 07/28/2020
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Iyer, Ravishankar R., Tickoo, Omesh, Anderson, Glen J.
• Primary Examiner(s): To, Jennifer N
• Assistant Examiner(s): Yoon, Eric J
• Application Number:
  US14/498,704
• Publication Number:
  US 20160091964A1
• Time in Patent Office: 2,132 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Generally discussed herein are systems and apparatuses for gesture-based augmented reality. Also discussed herein are methods of using the systems and apparatuses. According to an example a method may include detecting, in image data, an object and a gesture, in response to detecting the object in the image data, providing data indicative of the detected object, in response to detecting the gesture in the image data, providing data indicative of the detected gesture, and modifying the image data using the data indicative of the detected object and the data indicative of the detected gesture.

Claims

1. A device comprising:

a camera to capture image data including video;

a display to provide a view of the video captured by the camera;

an object recognition module including circuitry to analyze the image data to recognize a first object and one or more fingers in the video, and in response to recognizing the first object, to provide data indicative of the recognized first object and in response to recognizing the one or more fingers, to extract data corresponding to the one or more fingers from the video;

a gesture recognition module including circuitry to analyze the video to recognize a first gesture performed by the recognized one or more fingers, and in response to recognizing the first gesture, to provide data indicative of the first recognized gesture;

wherein the object recognition module, in response to the data indicative of the first recognized gesture, is further to issue a command to circuitry of an image rendering module that causes the image rendering module to render a still image including the first object;

wherein the camera is to continue to capture second video while the display is providing the view of the still image;

wherein the object recognition module is to, while the display is providing the view of the till image, extract data corresponding to the one or more fingers from the second video;

the image rendering module including circuitry to modify the still image to include the extracted one or more fingers from the second video overlaid on the still image, to alter pixel values of or around the first object in the still image, and to augment the first object based on data indicative of a second recognized gesture; and

wherein the image rendering module is further to cause the display to provide a view of the modified still image including the augmented first object with the extracted one or more fingers from the second video overlaid thereon.

2. The device of claim 1, wherein the image rendering module is to modify the still image with a list of one or more user-selectable operations, which when selected, cause the image rendering module to modify the image data using a selected operation.

3. The device of claim 1, wherein, in response to the gesture recognition module recognizing the second gesture, the image rendering module is to modify the image data by performing a first operation in response to the object recognition module determining a recognized object is the first object, and wherein, in response to the gesture recognition module recognizing the same second gesture, the image rendering module is to modify the image data by performing a second operation, different from the first operation, in response to the object recognition module determining a recognized object is the second object, different from the first object.

4. The device of claim 1, further comprising:

a microphone to capture sounds;

a speech recognition module including circuitry to translate one or more sounds captured by the microphone and provide data indicative of the translated one or more sounds;

a context module including circuitry to determine one or more social circumstances of the user based on a location, a speed of the user, a translated sound, one or more objects in the image data, and one or more people in the image data, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working;

wherein the image rendering module is to modify the image by performing a first operation on the image data based on a first social circumstance of the one or more social circumstances determined by the context module, and the second gesture recognized by the gesture recognition module; and

wherein the image rendering module is to modify the image by performing a second, different operation on the image data based on a second social circumstance of the one or more social circumstances determined by the context module, different from the first social circumstance, and a third gesture recognized by the gesture recognition module.

5. The device of claim 4, wherein the device further comprises an authentication module including circuitry to receive the data indicative of the first recognized gesture to determine whether the first recognized gesture satisfies a policy including one or more gestures that must be performed before a user is allowed access to the functionality of the device, and in response to determining the policy has been satisfied, provide data indicating a valid authentication procedure has been performed.

6. The device of claim 5, further comprising;

an output module including a speaker to output audio;

wherein the authentication module is further to, in response to determining the recognized first gesture does not satisfy the policy, provide data indicating the authentication procedure has failed;

wherein the output module is to, in response to receiving data indicating the authentication procedure has failed, output a series of security questions to a user;

a speech recognition module includes circuitry to receive answers to the security questions from the user, and provide data indicative of the answers from the user; and

wherein the authentication module is to determine whether the policy has been satisfied based on the data indicative of the answers from the user.

7. The device of claim 1, comprising a microphone and a speech recognition module, wherein the speech recognition module includes circuitry to translate a sound captured by the microphone and provide data indicative of the translated sound that causes the output device to perform the operation using the translated sound.

8. A method comprising operations performed using a hardware processor, the operations comprising:

capturing, by a camera, image data including video,

displaying, by a display, a view of the video;

detecting, by circuitry of an object recognition module, in the video, a first object and one or more fingers;

extracting, by the circuitry of the object recognition module, the detected first object and the one or more fingers from the video;

displaying, by the display, the extracted one or more fingers overlaid on the video;

detecting, by circuitry of a gesture recognition module, a gesture performed by the one or more extracted one or more fingers;

issuing, by the circuitry of the object recognition module, a command to circuitry of an image rendering module that causes the image rendering module to render a still image including the first object, capturing, by the camera, second video while the display is providing the view of the still image;

while the display is providing the view of the still image, extracting, by the circuitry of the object recognition module, data corresponding to the one or more fingers from the second video;

modifying, by circuitry of an image rendering module, the still image to include the extracted one or more fingers from the second video overlaid on the still image;

altering pixel values of or around the first object in the still image, and augmenting the first object based on data indicative of a second recognized gesture; and

displaying the modified still image including the augmented first object, with the extracted one or more fingers overlaid on the modified still image.

9. The method of claim 8, wherein modifying the still image includes augmenting the still image with a list of one or more user-selectable operations, which when selected by a user, cause the selected operation to be performed on the image data.

10. The method of claim 8, wherein modifying the still image includes modifying the still image to include a plurality of targets and the operations further comprise determining whether the detected first object virtually touches one or more targets of the plurality of targets in a specified order, and in response to determining the detected first object has virtually touched the one or more targets in the specified order, providing data indicative of a valid authentication procedure being performed.

11. The method of claim 8, wherein modifying the still image includes modifying the still image by performing a first operation in response to detecting the first object and modifying the still image by performing a second, different operation in response to detecting a second, different object.

12. The method of claim 8, further comprising associating a sound captured by a microphone with the detected first object by augmenting the image data with text representing the sound proximate to the detected first object.

13. The method of claim 8, further comprising:

detecting one or more sounds using a microphone;

in response to detecting one or more sounds, translating the one or more sounds and providing data indicative of the translated one or more sounds;

determining one or more social circumstances of the user based on a location, a speed of the user, data indicative of the translated one or more sounds, the detected first object, and one or more people in the video, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working;

modifying the image data by performing a first operation on the image data based upon a first social circumstance of the determined one or more social circumstances and the second gesture, and

modifying the image data by performing a second, different operation on the image data based upon a second social circumstance of the determined one or more social circumstances and a third gesture, wherein the second social circumstance is different from the first social circumstance.

14. The method of claim 8, wherein detecting the first object includes using one or more of edge matching, color matching, and gradient matching.

15. At least one non-transitory machine-readable medium comprising instructions, when performed by a machine, cause the machine to:

capture image data including video;

display the video,

detect, in the video, a first object and one or more fingers;

extract the detected first object and the one or more fingers from the video;

display the extracted one or more fingers overlaid on the video;

detect a gesture performed by the extracted one or more fingers in the video;

render a still image including the first object,

capture second video while displaying the view of the still image;

while the display is providing the view of the still image, extract data corresponding to the one or more fingers from the second video;

modify the still image to include the extracted one or more fingers from the second video overlaid on the still image;

alter pixel values of or around the first object in the still image, and augment the first object based on data indicative of a second recognized gesture; and

display the modified still image including the augmented first object, with the extracted one or more fingers overlaid on the modified still image.

16. The machine-readable medium of claim 15, wherein the instructions for modifying the still image include instructions, when performed by the machine, cause the machine to augment the image data with a list of one or more user-selectable operations, which when selected by a user, cause a selected operation to be performed on the image data.

17. The machine-readable medium of claim 15, wherein the instructions for modifying the still image include instructions, when performed by the machine, cause the machine to modify the image data to include a plurality of targets and the operations further comprise determining whether the detected first object virtually touches one or more targets of the plurality of targets in a specified order, and in response to determining the detected first object has virtually touched the one or more targets in the specified order, provide data indicative of a valid authentication procedure being performed.

18. The machine-readable medium of claim 15, wherein the instructions for modifying the still image include instructions, when performed by the machine, cause the machine to modify the image data by performing a first operation in response to detecting the first object, and modifying the image data by performing a second operation different, from the first operation in response to detecting a second object different from the first object.

19. The machine-readable medium of claim 15, further comprising instructions, when performed by the machine, cause the machine to:

detect one or more sounds using a microphone;

in response to detecting one or more sounds, translate the one or more sounds and provide data indicative of the translated one or more sounds;

determine one or more social circumstances of the user based on a location, a speed of the user, data indicative of the translated one or more sounds, the detected first object, and one or more people in the image data, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working;

modify the image data by performing a first operation on the image data based upon a first social circumstance and the second recognized gesture; and

modify the image data by performing a second, different operation on the image data based upon a second social circumstance and a third recognized gesture, wherein the second social circumstance is different from the first social circumstance.

20. The machine-readable medium of claim 15, wherein the instructions to augment the detected first object include instructions when performed by the machine, cause the machine to label the detected first object with a name.