Recognizing in-air gestures of a control object to control a vehicular control system

• US 9,868,449 B1
• Filed: 05/29/2015
• Issued: 01/16/2018
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S., Yang, Hua, Gordon, Robert Samuel, Roy, Neeloy, Schunick, Justin, Durdik, Paul A.
• Primary Examiner(s): Khatib, Rami
• Assistant Examiner(s): Boomer, Jeffrey
• Application Number:
  US14/725,510
• Time in Patent Office: 963 Days
• CPC Class Codes: B60W 2540/00 Input parameters relating to occupants
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed relates to an embeddable motion sensory control device that detects gestures in a three dimensional (3D) sensory space within a vehicle cabin, detecting a gesture in the 3D sensory space and interpreting the gesture as a command to a (sub) system of the vehicle under control, and issuing the command when appropriate.

Claims

1. A method of recognizing in-air gestures of a control object to control a vehicular control system, the method including:

determining observation information characterizing in-air gestural motion of a control object from at least one image captured at time t0;

constructing a three-dimensional model to represent the control object by fitting one or more three-dimensional solid subcomponents modelled in software to a construct of the control object defined by the observation information based on the image captured at time t0;

responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improving conformance of the three-dimensional model to the modified observation information by;

determining an error indication including a difference in space between a point on another construct of the control object defined by the observation information based on the image captured at time t1 and a corresponding point on at least one of the three-dimensional solid subcomponents modelled in software fitted to the construct defined by the observation information based on the image captured at time t0; and

responsive to the error indication adjusting the three-dimensional model; and

interpreting motions of the control object wherein the motions are determined based at least in part from the adjusting the three-dimensional model as one or more gestures providing command input to a vehicular control system;

wherein the at least one image captured at time t0 and the another image captured at time t1 include;

capturing with a camera an ambient capture for a field of view of the camera in ambient lighting conditions;

illuminating a flash;

capturing with the camera a flash capture for the field of view of the camera while illuminated by the flash;

differencing the ambient capture and the flash capture to produce a clean image capture with reduced ambient lighting effects;

analyzing the clean image capture to construct a three-dimensional model of a control object in the field of view;

tracking, using the three-dimensional model, the control object by repeatedly producing and analyzing clean image captures of the control object and interpreting the gestures and outputting one or more control streams including signal which results in control of a vehicle system based on the interpretation of the gestures; and

wherein one or more sensors alternate between ambient captures and flash captures during at least part of gesture capture and analysis.

2. The method of claim 1, the method further including:

scanning the field of view of the camera by selectively illuminating for short durations respective ones of a pre-determined plurality of directionally oriented light sources that have overlapping fields of illumination;

measuring one or more differences in intensity of returning light emitted from the respective light sources and reflected from a control object in a vehicle environment;

analyzing the reflected light across a series of image frames for periodic brightness variations; and

determining positional information of the control object based at least in part upon one or more measured differences in intensity of the returning light.

3. The method of claim 2, wherein the short durations are in order of 5 to 40 microseconds.

4. The method of claim 2, wherein the pre-determined plurality of directionally oriented light sources includes between six to sixteen light sources.

5. The method of claim 2, wherein a rate of selective illumination of the light sources is determined by measuring one or more frequencies of the periodic brightness variations.

6. The method of claim 2, further including varying a rate of capture of the series of image frames.

7. The method of claim 2, wherein selectively illuminating the respective light sources further includes varying brightness of pairs of overlapping light sources by dimming a first, initially on light source while brightening a second, initially off light source.

8. The method of claim 2, further including capturing the image frames at a low-resolution by analyzing a reduced amount of image data lines per image frame.

9. The method of claim 1, wherein the flash has a duration of 5 to 300 microseconds.

10. The method of claim 1, wherein the flash has a duration of 5 to 40 microseconds.

11. The method of claim 1, wherein the flash is produced by six to sixteen light emitting diodes.

12. The method of claim 1, wherein the flash is produced by light emitting diodes designed to mix wide beams and narrow beams.

13. The method of claim 1, wherein the flash is produced by light emitting diodes operating in an ultraviolet frequency range and the capturing further includes using an ultraviolet (UV) sensitive or UV filtered camera.

14. The method of claim 1, wherein the flash is produced by light emitting diodes operating in an infrared (IR) frequency range and the capturing further includes using an IR sensitive or IR filtered camera.

15. The method of claim 1, wherein the flash capture is timed with the flash to reduce capture from ambient lighting.

16. The method of claim 1, wherein duration of the flash capture lasts no more than four times the duration of the flash.

17. The method of claim 1, wherein the one or more sensors operate, at least part time, at a reduced resolution that is half or less of available sensor resolution and at an increased capture rate enabled by the reduced resolution.

18. The method of claim 1, wherein one or more sensors capture binocular views in of at least the flash captures.

19. The method of claim 18, further including producing clean binocular captures by differencing and constructing the three-dimensional model of the control object from clean binocular views.

20. The method of claim 19, wherein the differencing to produce clean captures further includes comparing the flash capture to one or more ambient captures immediately before or after the flash capture.

21. The method of claim 19, wherein the differencing to produce clean captures further includes comparing the flashes capture to two or more ambient captures immediately before and after the flash capture.

22. The method of claim 1, further including:

capturing at least flash images of a passenger vehicle compartment and creating a 3D model of non-control object features; and

when constructing and following gestures of the 3D model of the control object, taking into account the non-control object features of the passenger vehicle compartment.