Touchless human machine interface

• US 10,025,388 B2
• Filed: 02/10/2011
• Issued: 07/17/2018
• Est. Priority Date: 02/10/2011
• Status: Active Grant

• Current Assignee: Continental Automotive Systems, Inc. (Continental AG)
• Original Assignee: Continental Automotive Systems, Inc. (Continental AG)
• Inventors: Tang, Leung Yu
• Primary Examiner(s): Moyer, Andrew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US13/024,392
• Publication Number:
  US 20120207345A1
• Time in Patent Office: 2,714 Days
• Field of Search: 382124, 715863, 345175
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1868 according to driving situations
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/1365 Matching; Classification

Abstract

A system and method for receiving input from a user is provided. The system includes at least one camera configured to receive an image of a hand of the user and a controller configured to analyze the image and issue a command based on the analysis of the image.

Claims

1. A system for receiving input from a user of a vehicle, the system comprising:

a first camera configured to selectively capture a first image being an image of a background;

the first camera additionally configured to selectively capture a second image after the first image and including both background and an image of a finger print of a user'"'"'s fingers and the user'"'"'s hand,

wherein the first camera is associated with a front console of the vehicle;

a controller configured to

display the second image of the user'"'"'s hand,

overlay graphics on the second image of the user'"'"'s hand, and

the controller coupled to the first camera and configured to

subtract the first image from the overlaid second image to produce a difference image comprising an image of the finger prints of the user,

the controller being additionally configured to

compare the image of the finger prints to a stored pattern and

determine from said comparison the identity of the fingers of the user,

the controller being additionally configured to

determine a three dimensional motion as a gesture of the fingers of the user'"'"'s hand without the identified fingers touching any portion of the front console based on a model of the fingers by

determining three-dimensional finger paths from another image from another camera associated with the front console and from the second image; and

matching the determined three-dimensional finger paths to a predetermined model of finger paths to

determine a gesture based on the determined three dimensional motion without the identified fingers touching any portion of the front console, and

control a device based on the determined gesture.

2. A system for receiving input from a user of a vehicle, the system comprising:

a first camera configured to selectively capture a first image of a background;

the first camera configured to selectively

capture a second image after the first image and including both

background and

an image of finger prints of a user'"'"'s fingers of the user'"'"'s hand;

wherein the first camera is associated with a front console of the vehicle; and

a controller coupled to the first camera and configured

to subtract the first image from the second image

to produce a difference image comprising an image of the finger prints of the user,

the controller being additionally configured to

compare the image of the finger prints to stored patterns and

determine from said comparison the identity of the fingers of the user,

the controller being additionally configured to

determine the location of the identified fingers of the user within a field of view of the first camera,

the controller being additionally configured to

determine a three dimensional motion as a gesture of the fingers of the user'"'"'s hand without the identified fingers touching any portion of the front console based on

using a model of the finger prints in three dimensional space to

determine positions of at least two of the identified fingers relative to one another in the first image from the first camera and a successive image from another camera based on triangulation; and

determine, based on

the determined positions and

the identified fingers

three-dimensional finger paths; and

match the determined three-dimensional finger paths to a predetermined model of finger paths to

determine a gesture based on the determined three dimensional motion without the fingers touching any portion of the front console, and

control a device based on the determined gesture.

3. The system according to claim 2, wherein the controller is configured to determine the positions of the at least two identified fingers on the hand by first generating an average of a plurality of background images that do not include the hand and thereafter calculating a difference between each image in the successive series of images that include the hand and captured images of the finger prints of the at least two fingers.

4. A system for receiving input command gestures from a user in a vehicle, the system comprising:

at least one camera located in the vehicle and associated with a front console of the vehicle and configured to

receive a first series of images of a background obtained from inside the vehicle and,

after the at least one camera receives the first series of images of background, receive a second image of finger prints of a user'"'"'s fingers and the user'"'"'s hand captured in the first series of images; and

a controller configured to

average the first series of images of the background received by the at least one camera captured prior to capturing the second image received by the at least one camera and to

subtract the average of the first series of images of the background from the second image to

produce an image of the finger prints of the user'"'"'s finger,

the controller additionally configured to

display the second image of the user'"'"'s hand,

overlay graphics on the second image of the user'"'"'s hand, and

determine the identity of the user'"'"'s fingers by

comparing the second image of the finger prints of the user'"'"'s finger to a stored pattern of finger prints and

issue a command based on

whether the second image of the finger prints of the user'"'"'s fingers matches the stored pattern of finger prints,

the controller being additionally configured to

determine a three dimensional motion as a gesture of the identified fingers of the user'"'"'s hand without the identified fingers touching any portion of the front console based on

determining three-dimensional finger paths from one image of the first series of images and one image from a second series of images from another camera associated with the front console of the vehicle;

matching the determined three-dimensional finger paths to a predetermined model of finger paths to

determine a gesture based on the determined three dimensional motion without the identified fingers touching any portion of the front console, and

control a device based on the determined gesture.

5. The system according to claim 4, wherein the controller is configured to determine a position of one of more fingers of the user'"'"'s fingers in an image.

6. The system according to claim 5, further comprising a display screen in the vehicle, and wherein the at least one camera is a plurality of cameras located in predetermined positions on the front console, each camera being configured and located to capture a series of images of finger prints, the controller being configured to identify a plurality of positions of the one or more fingers responsive to a determination that a plurality of images of the finger prints of one or more fingers match stored patterns of finger prints.

7. The system according to claim 6, wherein the controller is configured to analyze each series of images to identify positions of the one or more fingers in three dimensional space.

8. The system according to claim 6, wherein the controller is configured to determine the positions of at least two fingers relative to one another.

9. The system according to claim 4, wherein the controller is configured to determine a plurality of positions of at least one finger of the user'"'"'s fingers in each image of a series of images to determine a finger path.

10. The system according to claim 4, wherein the controller is configured to determine a plurality of positions of at least two fingers of the user'"'"'s fingers relative to one another in each image of a series of images to determine a gesture based on a plurality of finger paths of the at least two fingers in three dimensional space.

11. The system according to claim 4, wherein the controller is configured to determine a plurality of positions of at least two fingers relative to one another in each one image based on triangulation and a position and orientation of each camera.

12. The system according to claim 4, wherein the controller is configured to determine a plurality of positions of at least two fingers of the user'"'"'s fingers by generating an average of a plurality of background images and calculating a difference between each image of a series of images and the average of the plurality of background images.

13. The system according to claim 4, wherein the controller is configured to match a stored model of a finger print to a finger print on at least one of the fingers.

14. A method for receiving a touchless input to a system in a vehicle, from a user inside the vehicle, the method comprising:

capturing a series of images of a background, the series of images being captured by a first camera inside the vehicle and associated with a front console of the vehicle;

averaging the series of captured images of the background;

after the series of captured images of the background are averaged,

capture an image of the user'"'"'s hand viewed by the first camera,

overlay graphics on the captured image of the user'"'"'s hand viewed by the first camera, and

capturing based on the overlaid image a plurality of images of

finger prints of the user'"'"'s fingers,

the user'"'"'s hand, and

background

with the first camera;

subtracting the average of the series of captured images of the background from the plurality of captured images of the finger prints of the user'"'"'s fingers and the background to

produce a plurality of difference images of the finger prints of the user'"'"'s fingers;

analyzing the plurality of difference images of the finger prints of the user'"'"'s fingers to

identify fingers

determining a gesture made in three dimensional space by

determining movement of the hand without the identified fingers touching any portion of the front console based on

using a predetermined model of finger paths of the identified fingers by determining three-dimensional finger paths from another image from another camera and from one image of the plurality of captured images; and

matching the determined three-dimensional finger paths to the predetermined model of finger paths to

determine the gesture based on the determined movement of the hand without the identified fingers touching any portion of the front console; and

issuing a command to the system in the vehicle to control a device based on the determined gesture.

15. The method according to claim 14, wherein the series of images include at least one image of a plurality of images captured by a plurality of cameras on a front console of a vehicle.

16. The method according to claim 14, further comprising determining a plurality of positions of at least one finger of the user'"'"'s fingers in each image of a series of images to determine a finger path.

17. The method according to claim 16, further comprising determining a plurality of positions of at least two fingers relative to one another in each image of the series of images to determine finger paths to determine a gesture based on each finger path of the at least two fingers in three dimensional space.

18. The method according to claim 17, further comprising determining the positions of the at least two fingers relative to one another in each image based on triangulation and a position and orientation of each camera.

19. The method according to claim 18, further comprising determining the positions of the at least two fingers by generating an average of a plurality of background images and calculating a difference between each image in a series of images and the average of the plurality of background images.