Systems and related methods for three dimensional gesture recognition in vehicles

• US 8,396,252 B2
• Filed: 05/20/2010
• Issued: 03/12/2013
• Est. Priority Date: 05/20/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek A.
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): SHAH, UTPAL D
• Application Number:
  US12/784,022
• Publication Number:
  US 20110286676A1
• Time in Patent Office: 1,027 Days
• Field of Search: 382/106, 382/154, 382/181, 382/190, 382/224, 382/225
• US Class Current: 382/106
• CPC Class Codes: B62D 1/00 Steering controls, i.e. means for initiating a change of direction of the vehicle
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method and system for performing gesture recognition of a vehicle occupant employing a time of flight (TOF) sensor and a computing system in a vehicle. An embodiment of the method of the invention includes the steps of receiving one or more raw frames from the TOF sensor, performing clustering to locate one or more body part clusters of the vehicle occupant, locating the palm cluster of the vehicle occupant, calculating the location of the tip of the hand of the vehicle occupant, determining whether the hand has performed a dynamic or a static gesture, retrieving a command corresponding to one of the determined static or dynamic gestures, and executing the command.

Claims

1. A method for performing gesture recognition in a vehicle, comprising the steps of:

receiving one or more raw frames from a time of flight (TOF) camera of a vehicle occupant;

performing clustering to determine one or more objects of interest of the vehicle occupant;

locating a body cluster of the vehicle occupant as one of the objects of interest;

locating one or more arm clusters of the vehicle occupant;

locating a head and shoulders of the vehicle occupant;

calculating one or more arm lengths of the vehicle occupant;

locating a palm cluster of the vehicle occupant; and

calculating a location of a tip of a hand of the vehicle occupant.

2. The method of claim 1, further comprising the step of cleaning a depth data of each of the one or more raw frames.

3. The method of claim 1, further comprising the steps of:

defining one or more predetermined search spaces in accordance with the location of the tip of the hand;

defining one or more partition spaces within the one or more search spaces; and

locating the hand within the partition and search spaces.

4. The method of claim 3, further comprising the steps of:

observing whether the hand has performed a dynamic or a static gesture;

if the hand has performed a dynamic gesture, matching the observed gesture with a set of known masks to determine an intended static gesture; and

if the hand has performed a dynamic gesture, extracting features from one or more clips of the observed gesture and match with one or more predefined motion streams to determine an intended dynamic gesture.

5. The method of claim 4, further comprising the steps of:

retrieving a command corresponding to one of the determined static and dynamic gesture; and

executing the command.

6. The method of claim 5, wherein the command is executed in accordance with a computing system and display within the vehicle.

7. The method of claim 4, wherein the known masks are determined for a particular individual vehicle occupant.

8. The method of claim 4, wherein the predefined motion streams are determined for a particular individual vehicle occupant.

9. The method of claim 1, further comprising the step of selecting the vehicle occupant from among a plurality of individuals in the vehicle.

20. The system of claim 1, wherein a processor is adapted to determine the vehicle occupant from among two or more individuals in the vehicle.

10. A method for performing gesture recognition of a vehicle occupant employing a time of flight (TOF) sensor and a computing system in a vehicle, comprising the steps of:

receiving one or more raw frames from the TOF sensor;

performing clustering to locate one or more body part clusters of the vehicle occupant;

locating a palm cluster of the vehicle occupant;

calculating a location of a tip of a hand of the vehicle occupant;

determining whether the hand has performed a dynamic or a static gesture;

retrieving a command corresponding to one of the determined static and dynamic gesture; and

executing the command.

11. A system for performing gesture recognition in a vehicle, comprising:

a time of flight (TOF) camera for providing one or more raw frames of a vehicle occupant; and

a processor for performing clustering to determine one or more objects of interest of the vehicle occupant, locating a body cluster of the vehicle occupant as one of the objects of interest, locating one or more arm clusters of the vehicle occupant, locating a head and shoulders of the vehicle occupant, calculating one or more arm lengths of the vehicle occupant, locating a palm cluster of the vehicle occupant, and calculating a location of a tip of a hand of the vehicle occupant.

12. The system of claim 11, wherein the processor is further adapted to clean a depth data of each of the one or more raw frames.

13. The system of claim 11, wherein the processor is further adapted to define one or more predetermined search spaces in accordance with the location of the tip of the hand, define one or more partition spaces within the one or more search spaces, and locate the hand within the partition and search spaces.

14. The system of claim 13, wherein the processor is further adapted to observe whether the hand has performed a dynamic or a static gesture, if the hand has performed a dynamic gesture, match the observed gesture with a set of known masks to determine an intended static gesture, and if the hand has performed a dynamic gesture, extract features from one or more clips of the observed gesture and match with one or more predefined motion streams to determine an intended dynamic gesture.

15. The system of claim 14, wherein the processor is further adapted to retrieve a command corresponding to one of the determined static and dynamic gesture, and execute the command to perform one or more vehicle actions.

16. The system of claim 15, further comprising a computing system for implementing the command.

17. The system of claim 15, wherein the computing system further comprises a display for displaying a response to the command to the vehicle occupant.

18. The system of claim 14, wherein the processor is adapted to determine the known masks for a particular individual vehicle occupant.

19. The system of claim 14, wherein the processor is adapted to determine the predefined motion streams for a particular individual vehicle occupant.