Method of virtual user interface interaction based on gesture recognition and related device

• US 10,678,342 B2
• Filed: 10/21/2018
• Issued: 06/09/2020
• Est. Priority Date: 10/21/2018
• Status: Active Grant

• Current Assignee: XRSPACE Co., Ltd.
• Original Assignee: XRSPACE Co., Ltd.
• Inventors: Chou, Peter, Chu, Feng-Seng, Lin, Yen-Hung, Ke, Shih-Hao, Chen, Jui-Chieh
• Primary Examiner(s): Dasgupta, Shourjo
• Application Number:
  US16/166,130
• Publication Number:
  US 20200125176A1
• Time in Patent Office: 597 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method of virtual user interface interaction based on gesture recognition comprises detecting two hands in a plurality of images, recognizing each hand'"'"'s gesture, projecting a virtual user interface on an open gesture hand when one hand is recognized with a point gesture and the other hand is recognized with an open gesture, tracking an index fingertip of the point gesture hand, determining whether the index fingertip of the point gesture hand is close to the open gesture hand within a predefined rule, interpreting a movement of the index fingertip of the point gesture hand as a click command when the index fingertip of the point gesture hand is close to the open gesture hand within the predefined rule, and in response to the click command, generating image data with a character object of the virtual user interface object.

Claims

1. A method of virtual user interface interaction based on gesture recognition for an interaction system, the method comprising:

detecting two hands in a plurality of images;

the detecting step comprises;

extracting depth pixels of an image with a user'"'"'s hand within a working distance;

classifying the depth pixels into a possible left hand group and a right hand group with Random Decision Forest (RDF);

taking a set of depth context surrounding an examined pixel as an input reference by the RDF, and outputting the possible group of the examined matching the left or right hand group into connected objects in a frame;

calculating a left or right hand contour radius;

cropping a left or right hand silhouette; and

extracting left or right hand depth information according to the left or right hand silhouette;

recognizing each hand'"'"'s gesture when the two hands are detected;

the recognizing step including;

extract hand subframes;

classifying pixels of the hand subframes into gesture groups with the RDF;

setting a majority group of the gesture groups as a gesture result; and

using a secondary group of the gesture groups as a gesture reference;

projecting a virtual user interface on an open gesture hand when one hand is recognized with a point gesture and the other hand is recognized with an open gesture;

tracking an index fingertip of the point gesture hand, for obtain a relative position of the index fingertip of the point gesture hand and the open gesture hand;

determining whether the index fingertip of the point gesture hand is close to the open gesture hand within a predefined rule; and

interpreting a movement of the index fingertip of the point gesture hand as a click command when the index fingertip of the point gesture hand is close to the open gesture hand within the predefined rule.

2. The method of claim 1, wherein the predefined rule includes a predetermined angle or a predetermined distance.

3. The method of claim 1, further comprising:

consider the open gesture hand as an unmovable object; and

canceling the virtual user interface if the open gesture hand moves.

4. The method of claim 1, further comprising:

determining whether the open gesture hand is within a predefined angle, or the open gesture hand is on a side of the point gesture hand.

5. The method of claim 4, further comprising:

consider the open gesture hand as an unmovable object when determining the open gesture hand is within the predefined angle, or the open gesture hand is on the side of the point gesture hand;

wherein the projecting step comprises projecting the virtual user interface at a location corresponding to the unmovable object when one hand is recognized with a point gesture and the other hand is recognized with an open gesture.

6. The method of claim 1, wherein the virtual user interface includes a virtual keyboard or a virtual number pad.

7. The method of claim 1, further comprising:

in response to the click command, generating image data with a character object of the virtual user interface based on the relative position.

8. An electronic device of an interaction system for virtual user interface interaction based on gesture recognition, the apparatus comprising:

a processing device for executing a program; and

a memory device coupled to the processing device for storing the program;

wherein the program instructs the processing device to perform the following steps;

detecting two hands in a plurality of images;

the detecting step comprises;

extracting depth pixels of an image with a user'"'"'s hand within a working distance;

classifying the depth pixels into a possible left hand group and a right hand group with Random Decision Forest (RDF);

taking a set of depth context surrounding an examined pixel as an input reference by the RDF, and outputting the possible group of the examined pixel;

matching the left or right hand group into connected objects in a frame;

calculating a left or right hand contour radius;

cropping a left or right hand silhouette; and

extracting left or right hand depth information according to the left or right hand silhouette;

recognizing each hand'"'"'s gesture when the two hands are detected;

the recognizing step comprises;

extract hand subframes;

classifying pixels of the hand subframes into gesture groups with the RDF;

setting a majority group of the gesture of groups as a gesture result; and

using a secondary group of the gesture of groups as a gesture reference;

projecting a virtual user interface on an open gesture hand when one hand is recognized with a point gesture and the other hand is recognized with an open gesture;

tracking an index fingertip of the point gesture hand, for obtain a relative position of the index fingertip of the point gesture hand and the open gesture hand;

determining whether the index fingertip of the point gesture hand is close to the open gesture hand within a predefined rule; and

interpreting a movement of the index fingertip of the point gesture hand as a click command when the index fingertip of the point gesture hand is close to the open gesture hand within the predefined rule.

9. The apparatus of claim 8, wherein the predefined rule includes a predetermined angle or a predetermined distance.

10. The apparatus of claim 8, wherein the program further instructs the processing means to perform the following steps:

consider the open gesture hand as an unmovable object; and

canceling the virtual user interface if the open gesture hand moves.

11. The apparatus of claim 8, wherein the program further instructs the processing means to perform the following steps:

determining whether the open gesture hand is within a predefined angle, or the open gesture hand is on a side of the point gesture hand.

12. The apparatus of claim 11, wherein the program further instructs the processing means to perform the following steps:

consider the open gesture hand as an unmovable object when determining the open gesture hand is within the predefined angle, or the open gesture hand is on the side of the point gesture hand;

wherein the projecting step comprises projecting the virtual user interface at a location corresponding to the unmovable object when one hand is recognized with a point gesture and the other hand is recognized with an open gesture.

13. The apparatus of claim 8, wherein the virtual user interface includes a virtual keyboard or a virtual number pad.

14. The apparatus of claim 8, wherein the program further instructs the processing means to perform the following step:

in response to the click command, generating image data with a character object of the virtual user interface based on the relative position.