Real-time 3D gesture recognition and tracking system for mobile devices

• US 10,108,270 B2
• Filed: 12/22/2014
• Issued: 10/23/2018
• Est. Priority Date: 01/05/2014
• Status: Active Grant

• Current Assignee: ManoMotion AB
• Original Assignee: ManoMotion AB
• Inventors: Yousefi, Shahrouz, Li, Haibo, Abedan Kondori, Farid
• Primary Examiner(s): Moon, Patrick
• Assistant Examiner(s): Shen, Peijie
• Application Number:
  US15/109,524
• Publication Number:
  US 20160334877A1
• Time in Patent Office: 1,401 Days
• Field of Search: None
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The disclosure relates to a device and a method in the device for recognizing a 3D gesture. The device is connected to a sensor and has access to a database of gesture images including indexable features of normalized gesture images. The indexable features include a position and an orientation for each pixel of edge images of the normalized gesture images. The method includes capturing (110) an image of the 3D gesture via the sensor, normalizing (120) the captured image, deriving (130) indexable features from the normalized captured image, and comparing (140) the derived indexable features with the indexable features of the database using a similarity function. The method also includes determining (150) a gesture image in the database matching the 3D gesture based on the comparison.

Claims

1. A method for recognizing a three dimensional, 3D, gesture, the method being performed in a device (20) having access to a database (52) of gesture images, the device communicating with a sensor (34) adapted to capture an image (33) of the 3D gesture, wherein the database (52) of gesture images comprises indexable features (54) of normalized gesture images, the indexable features comprising a position and an orientation for each pixel of edge images of the normalized gesture images, the method comprising:

capturing (110) the image (33) of the 3D gesture via the sensor,

normalizing (120) the captured image in accordance with the normalized gesture images of the database (52) to allow for a comparison,

deriving (130) indexable features (36) from the normalized captured image (33), the indexable features (36) comprising a position and an orientation for each pixel of an edge image of the normalized captured image,

comparing (140) the derived indexable features (36) with the indexable features (54) of the database using a similarity function, and

determining (150) a gesture image in the database (52) matching the 3D gesture based on the comparison;

wherein each entry in the database (52) of gesture images is tagged with associated 3D motion parameters comprising three orientation and three position parameters, the method further comprising;

retrieving 3D motion parameters associated with the determined gesture image matching the 3D gesture from the database (52).

2. The method according to claim 1, wherein comparing (140) the derived indexable features further comprises:

using (141) a direct similarity analysis to determine a plurality of gesture images in the database matching the captured image,

and wherein the gesture image in the database (52) matching the 3D gesture is determined (150) to be one of the plurality of gesture images.

3. The method according to claim 2, wherein comparing (140) the derived indexable features further comprises:

using (142) a reverse similarity analysis of the plurality of gesture images to determine a subset of the plurality of gesture images matching the captured image,

and wherein the gesture image in the database (52) matching the 3D gesture is determined (150) to be one of the subset of the plurality of gesture images.

4. The method according to claim 1, further comprising:

deriving additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

comparing the derived additional indexable features with additional indexable features of the database using the similarity function,

and wherein the gesture image in the database (52) matching the 3D gesture is determined based also on the comparison of the additional indexable features.

5. The method according to claim 1, wherein the gesture image matching the 3D gesture is determined based on a gesture map indicating gestures images that are close to each other in a sequence of gesture images.

6. The method according to claim 1, wherein capturing (110) the image comprises capturing a sequence of images of the 3D gesture.

7. The method according to claim 6, further comprising:

tracking a user gesture based on the sequence of images, and wherein the gesture image in the database (52) matching the 3D gesture is determined based also on the tracked user gesture.

8. The method according to claim 1, further comprising:

using (160) the determined gesture image matching the 3D gesture to modify a visual content presented on a display.

17. The method according to claim 2, further comprising:

deriving additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

comparing the derived additional indexable features with additional indexable features of the database using the similarity function,

and wherein the gesture image in the database (52) matching the 3D gesture is determined based also on the comparison of the additional indexable features.

18. The method according to claim 3, further comprising:

deriving additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

comparing the derived additional indexable features with additional indexable features of the database using the similarity function,

and wherein the gesture image in the database (52) matching the 3D gesture is determined based also on the comparison of the additional indexable features.

9. A device (20) for recognizing a three dimensional, 3D, gesture, the device being configured to have access to a database (52) of gesture images comprising indexable features (54) of normalized gesture images, the indexable features comprising a position and an orientation for each pixel of edge images of the normalized gesture images, the device being connectable to a sensor (34) adapted to capture an image (33) of the 3D gesture, and the device comprising a processing unit (24) adapted to:

capture the image (33) of the 3D gesture via the sensor (34),

normalize the captured image in accordance with the normalized gesture images of the database (52)) to allow for a comparison,

derive indexable features (36) from the normalized captured image (33), wherein the indexable features (36) comprise a position and an orientation for each pixel of an edge image of the normalized captured image,

compare the derived indexable features (36) with the indexable features (54) of the database using a similarity function, and

determine a gesture image in the database (52) matching the 3D gesture based on the comparison;

wherein each entry in the database (52) of gesture images is tagged with associated 3D motion parameters comprising three orientation and three position parameters, the processing unit (24) being further adapted to;

retrieve 3D motion parameters associated with the determined gesture image matching the 3D gesture from the database (52).

10. The device (20) according to claim 9, wherein the processing unit (24) is further adapted to compare the derived indexable features by:

using a direct similarity analysis to determine a plurality of gesture images in the database matching the captured image,

the processing unit (24) being further adapted to determine the gesture image in the database (52) matching the 3D gesture to be one of the plurality of gesture images.

11. The device (20) according to claim 10, wherein the processing unit (24) is further adapted to compare the derived indexable features by:

using a reverse similarity analysis of the plurality of gesture images to determine a subset of the plurality of gesture images matching the captured image,

the processing unit (24) being further adapted to determine the gesture image in the database (52) matching the 3D gesture to be one of the subset of the plurality of gesture images.

12. The device (20) according to claim 9, wherein the processing unit (24) is further adapted to:

derive additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

compare the derived additional indexable features with additional indexable features of the database using the similarity function, and

determine the gesture image in the database (52) matching the 3D gesture based also on the comparison of the additional indexable features.

13. The device (20) according to claim 9, wherein the processing unit (24) is further adapted to determine the gesture image matching the 3D gesture based on a gesture map indicating gestures images that are close to each other in a sequence of gesture images.

14. The device (20) according to claim 9, wherein the processing unit (24) is further adapted to capture a sequence of images of the 3D gesture via the sensor (34).

15. The device (20) according to claim 14, wherein the processing unit (24) is further adapted to:

track a user gesture based on the sequence of images, and

determine the gesture image in the database (52) matching the 3D gesture based also on the tracked user gesture.

16. The device (20) according to claim 9, wherein the processing unit (24) is further adapted to:

use the determined gesture image matching the 3D gesture to modify a visual content presented on a display.

19. The device (20) according to claim 10, wherein the processing unit (24) is further adapted to:

derive additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

compare the derived additional indexable features with additional indexable features of the database using the similarity function, and

determine the gesture image in the database (52) matching the 3D gesture based also on the comparison of the additional indexable features.

20. The device (20) according to claim 11, wherein the processing unit (24) is further adapted to:

derive additional indexable features comprising a position and an orientation for neighbour pixels of each pixel of the edge image from the normalized captured image,

compare the derived additional indexable features with additional indexable features of the database using the similarity function, and

determine the gesture image in the database (52) matching the 3D gesture based also on the comparison of the additional indexable features.