Interface for car-mounted devices

• US 7,437,488 B2
• Filed: 10/28/2004
• Issued: 10/14/2008
• Est. Priority Date: 12/17/2003
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Ito, Takafumi, Kumazawa, Yuuichi
• Primary Examiner(s): CHEN, ALAN S
• Application Number:
  US10/974,721
• Publication Number:
  US 20050134117A1
• Time in Patent Office: 1,447 Days
• Field of Search: 710/5, 710/15, 710/19
• US Class Current: 710/15
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Abstract

An interface for car-mounted devices includes hand data detecting means for determining devices to be controlled among the car-mounted devices based upon hand data corresponding to an operator hand state, voice recognizing means for determining how to change the state of the devices to be controlled based upon an operator voice utterance, and control signal-forming means for forming control signals for the devices to be controlled relying upon the devices to be controlled determined by said hand data detecting means and upon the content of change determined by said voice recognizing means.

Claims

1. An interface for car-mounted devices comprising:

hand data detecting means for detecting hand data indicative of a state of a hand of an operator;

voice recognizing means for recognizing a voice utterance of the operator; and

control signal-forming means for specifying a subject device to be controlled among the car-mounted devices and a commanded control operation of the subject device based upon a combination of a result of the detection made by the hand data detecting means and a result of the recognition made by the voice recognizing means, and thereafter forming a corresponding control signal for the subject device, wherein

when a preset demonstrative pronoun that has been set in advance is recognized by the voice recognizing means, the control signal-forming means specifies the subject device based on the hand data detected by the hand data detecting means, and forms the control signal, which changes an operating state of the subject device.

2. An interface for car-mounted devices according to claim 1, wherein the state of the hand of the operator expressed by the hand data includes at least one of a shape of the hand, a motion of the hand and a position of the hand.

3. An interface for car-mounted devices according to claim 1, wherein:

the control signal-forming means evaluates a combination of the subject device, which is detected by the hand data extracting means, and the commanded control operation of the subject device, which is recognized by the voice recognizing means; and

the control signal-forming means invalidates the formed control signal when the control signal-forming means determines that the commanded control operation cannot be implemented by the subject device upon the evaluation of the combination of the subject device and the commanded control operation of the subject device.

4. An interface for car-mounted devices according to claim 1, wherein:

the voice recognizing means determines the commanded control operation of the subject device when a command, which is indicated by the voice utterance of the operator, matches with a corresponding one of a plurality of predetermined control operations of the subject device; and

when the voice recognizing means determines the commanded control operation of the subject device, the voice recognizing means sends a signal of determination to the hand data detecting means as a trigger for determining the state of the hand of the operator.

8. An interface for car-mounted devices according to claim 1, further comprising operation permission means for detecting a position of the hand of the operator and permitting the operation of the hand data detecting means or the control signal-forming means when the position of the hand of the operator exists in a predetermined region in a frame of an image captured by a camera.

9. An interface for car-mounted devices according to claim 8, wherein the operation permission means detects the position of the hand of the operator based on the input image from the camera.

10. An interface for car-mounted devices according to claim 1, wherein the hand data detecting means includes:

extracting means for extracting a hand portion from an input image received from a camera that captures an image of the hand of the operator, wherein the hand portion is a region that is configured to generally match with a shape of the captured image of the hand; and

shape specifying means for comparing a shape of the hand portion picked up by the extracting means with shapes of templates that have been provided in advance and thereby specifying a shape of the most matched template as the shape of hand; and

wherein the shape of hand specified by the shape specifying means is used as the hand data.

11. An interface for car-mounted devices according to claim 1, wherein the hand data detecting means includes:

extracting means for extracting a hand portion from an input image received from a camera that captures an image of the hand of the operator, wherein the hand portion is a region that is configured to generally match with a shape of the captured image of the hand; and

position specifying means for specifying a center of gravity of the hand portion picked up by the extracting means as a position of hand,

wherein the position of hand specified by the position specifying means is used as the hand data.

12. An interface for car-mounted devices according to claim 1, wherein the hand data detecting means includes:

extracting means for extracting a hand portion from an input image received from a camera that captures an image of the hand of the operator, wherein the hand portion is a region that is configured to generally match with a shape of the captured image of the hand; and

position specifying means for specifying a center of gravity of the hand portion picked up by the extracting means as a position of hand; and

motion specifying means for specifying a time-series change in the position of hand specified by the position specifying means as the motion of hand,

wherein the motion of hand specified by the motion specifying means is used as the hand data.

13. An interface for car-mounted devices according to claim 10, wherein the extracting means uses at least either the brightness or the color data of pixels constituting the input image as the data for extracting, and uses, as the hand portion, a region in an allowable range in which the data for extracting have been set in advance in the input image.

14. An interface for car-mounted devices according to claim 13, wherein the hand data extracting means finds a difference between time-series frames in a predetermined region of the input image to pick up a region where there is a high possibility of existence of a motion of the hand in the input image, and sets an allowable range of data for extracting based on an average value of data for extracting in the region that is picked up.

15. An interface for car-mounted devices according to claim 1, wherein the hand data detecting means detects the hand data in a three-dimensional manner based on the input images from a plurality of cameras arranged at different positions.

16. An interface for car-mounted devices according to claim 1, wherein the control signal formed by the control signal-forming means is transmitted to the subject device through a communication network connecting the car-mounted devices to each other.

17. An interface for car-mounted devices according to claim 11, wherein the extracting means uses at least either the brightness or the color data of pixels constituting the input image as the data for extracting, and uses, as the hand portion, a region in an allowable range in which the data for extracting have been set in advance in the input image.

18. An interface for car-mounted devices according to claim 12, wherein the extracting means uses at least either the brightness or the color data of pixels constituting the input image as the data for extracting, and uses, as the hand portion, a region in an allowable range in which the data for extracting have been set in advance in the input image.

5. An interface for car-mounted devices comprising:

hand data detecting means for detecting hand data indicative of a state of a hand of an operator;

voice recognizing means for recognizing a voice utterance of the operator; and

control signal-forming means for specifying a subject device to be controlled among the car-mounted devices and a commanded control operation of the subject device based upon a combination of a result of the detection made by the hand data detecting means and a result of the recognition made by the voice recognizing means, and thereafter forming a corresponding control signal for the subject device, wherein

when the control signal forming means specifies the subject device based on the hand data detected by the hand data detecting means and determines that a reserved word, which is associated with the control operation of the subject device, is recognized by the voice recognizing means, the control signal-forming means forms the control signal, which commands the subject device to execute the operation corresponding to the reserved word.

6. An interface for car-mounted devices comprising:

hand data detecting means for detecting hand data indicative of a state of a hand of an operator;

voice recognizing means for recognizing a voice utterance of the operator; and

control signal-forming means for specifying a subject device to be controlled among the car-mounted devices and a commanded control operation of the subject device based upon a combination of a result of the detection made by the hand data detecting means and a result of the recognition made by the voice recognizing means, and thereafter forming a corresponding control signal for the subject device, wherein

when the voice recognizing means recognizes a preset demonstrative pronoun that has been set in advance, the control signal-forming means specifies the subject device based on the hand data detected by the hand data detecting means; and

when the voice recognizing means recognizes a reserved word, which is associated with the control operation of the subject device, during a preset allowable time period measured from the time of specifying the subject device, the control signal forming means forms the control signal, which commands the subject device to execute the operation corresponding to the reserved word.

7. An interface for car-mounted devices comprising:

hand data detecting means for detecting hand data indicative of a state of a hand of an operator;

voice recognizing means for recognizing a voice utterance of the operator; and

control signal-forming means for specifying a subject device to be controlled among the car-mounted devices and a commanded control operation of the subject device based upon a combination of a result of the detection made by the hand data detecting means and a result of the recognition made by the voice recognizing means, and thereafter forming a corresponding control signal for the subject device, wherein

when the voice recognizing means recognizes a preset demonstrative pronoun that has been set in advance, the control signal-forming means specifies the subject device based on the hand data detected by the hand data detecting means;

then

when the preset demonstrative pronoun is recognized by the voice recognizing means again during a preset allowable time period measured from the time of specifying the subject device, the control signal-forming means specifies the commanded control operation of the subject device based on the hand data detected by the hand data detecting means and forms the control signal, which commands the subject device to execute the commanded control operation.

19. An interface for car-mounted devices comprising:

hand data detecting means for detecting hand data indicative of a state of a hand of an operator;

voice recognizing means for recognizing a voice utterance of the operator; and

control signal-forming means for specifying a subject device to be controlled among the car-mounted devices and a commanded control operation of the subject device based upon a combination of a result of the detection made by the hand data detecting means and a result of the recognition made by the voice recognizing means, and thereafter forming a corresponding control signal for the subject device, wherein

the control signal-forming means specifies the subject device based on the result of the detection made by the hand data detecting means, and the control signal-forming means specifies the commanded control operation of the subject device based on the result of the recognition made by the voice recognizing means.