ROBUST VIDEO-BASED HANDWRITING AND GESTURE RECOGNITION FOR IN-CAR APPLICATIONS

• US 20120105613A1
• Filed: 11/01/2010
• Published: 05/03/2012
• Est. Priority Date: 11/01/2010
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Weng, Fuliang, Cheng, You-Chi, Ren, Liu
• Granted Patent: US 8,817,087 B2
• Time in Patent Office: Days
• US Class Current: 348/77
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A method of receiving input from a user includes sensing a first trajectory of a center of mass of a hand of the user during a gesture made by the hand. A second trajectory of a finger tip of the hand of the user during the gesture made by the hand is also sensed. An alphanumeric character represented by the gesture made by the hand is determined dependent upon both the first trajectory and the second trajectory.

Claims

1. A method of receiving input from a user, said method comprising the steps of:

providing a surface within reach of a hand of the user, the surface being provided on a steering wheel or on an armrest of a motor vehicle;

sensing a first trajectory of a center of mass of the hand of the user during a gesture made by the hand;

sensing a second trajectory of a finger tip of the hand of the user during the gesture made by the hand, the sensing steps including capturing images of the gesture made by the hand with the surface being in a background of the images; and

determining an alphanumeric character represented by the gesture made by the hand, the determining being dependent upon both the first trajectory and the second trajectory.

2. The method of claim 1 wherein the surface has a color, the sensing steps being dependent upon the color.

3. The method of claim 1 wherein the surface has a vibration, the sensing steps being dependent upon the vibration.

4. The method of claim 1 wherein the determining step comprises using a lookup table or algorithm relating the trajectories to the determined alphanumeric character.

5. The method of claim 1 comprising the further steps of:

informing the user of the alphanumeric character and/or a word in which the alphanumeric character is included; and

receiving feedback from the user regarding whether the alphanumeric character and/or word is an alphanumeric character and/or word that the user intended to be represented by the gesture made by the hand.

6. The method of claim 5 comprising the further step, in response to receiving feedback from the user that the alphanumeric character or word is correct, of transmitting an input to an electronic system, the input being dependent upon the alphanumeric character or word.

7. The method of claim 5 wherein the sensing and determining steps are repeated until a word formed by the alphanumeric characters is identified, the informing step comprising audibly or visually informing the user of the identified word.

8. A method of entering input into a vehicle system, said method comprising the steps of:

capturing a plurality of images including a hand of a user during a gesture made by the hand;

identifying at least two separate and unconnected shapes in each of the images, the shapes having first image characteristics that are distinguishable from a second image characteristic associated with a background behind the hand in the images;

modifying the captured images by merging the separate and unconnected shapes in each of the images into a single continuous shape; and

determining a trajectory of at least one portion of the hand based on the modified images.

9. The method of claim 8 comprising the further steps of:

ascertaining an alphanumeric character that best matches the trajectory of the at least one portion of the hand; and

transmitting an input to an electronic system of the vehicle, the input being dependent upon the ascertained alphanumeric character.

10. The method of claim 9 wherein the capturing, identifying, modifying, determining and ascertaining steps are repeated until a word formed by the determined alphanumeric characters is identified, the input being dependent upon the identified word, the word being a command or information to the electronic system.

11. The method of claim 8 comprising the further step of providing a surface on a steering wheel and/or an armrest of the vehicle, the surface being the background of the images.

12. The method of claim 8 wherein the first and second image characteristics comprise colors.

13. The method of claim 8 wherein the merging includes considering an area in-between the separate and unconnected shapes to be, along with the shapes themselves, part of the user'"'"'s hand.

14. A method of entering input into a vehicle system, said method comprising the steps of:

capturing a plurality of images including a hand of a user during a gesture made by the hand;

ascertaining an image characteristic associated with a background behind the hand in the images;

identifying a location of a boundary of the hand in the images dependent upon the ascertained image characteristic of the background; and

determining a trajectory of at least one portion of the hand based on the identified boundary of the hand in the images.

15. The method of claim 14 wherein the image characteristic comprises a color of the background.

16. The method of claim 15 comprising the further step of deciding that a pixel in an image is part of the hand if:

the color of the background is a first color, and a difference between a color of the pixel and the first color exceeds a first threshold difference; and

the color of the background is a second color, and a difference between a color of the pixel and the second color exceeds a second threshold difference, the first threshold difference being different from the second threshold difference.

17. The method of claim 14 comprising the further step of providing a surface within reach of a hand of the user, the surface being provided on a steering wheel or on an armrest of a motor vehicle, the surface being in the background.

18. The method of claim 17 wherein the image characteristic comprises a speed and/or magnitude of vibration of the surface.

19. The method of claim 14 comprising the further steps of:

deciding upon an alphanumeric character that best matches the trajectory of the at least one portion of the hand; and

transmitting an input to an electronic system of the vehicle, the input being dependent upon the ascertained alphanumeric character.

20. The method of claim 19 comprising the further steps of:

informing the user of the alphanumeric character and/or a word in which the alphanumeric character is included; and

receiving feedback from the user regarding whether the alphanumeric character and/or word is an alphanumeric character and/or word that the user intended to be represented by the gesture made by the hand.