Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction

• US 10,706,300 B2
• Filed: 01/23/2018
• Issued: 07/07/2020
• Est. Priority Date: 01/23/2018
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Yamaoka, Masaaki, Horiuchi, Yuki
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US15/877,906
• Publication Number:
  US 20190228243A1
• Time in Patent Office: 896 Days
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Vehicle systems and methods for determining a target position are disclosed. A vehicle includes a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position. The vehicle also includes a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user. The vehicle also includes one or more processors and one or more non-transitory memory modules communicatively coupled to the processors. The processors store machine-readable instructions that, when executed, cause the one or more processors to determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system. The first point and the second point define a pointing axis of the hand of the user.

Claims

1. A vehicle, comprising:

a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position;

a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user;

one or more processors; and

one or more non-transitory memory modules communicatively coupled to the one or more processors and storing machine-readable instructions that, when executed, cause the one or more processors to perform at least the following;

determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system, wherein the first point and the second point define a pointing axis of the hand of the user;

calculate a virtual eye position based at least in part on the first point located on the hand of the user and the actual eye position;

determine a midpoint measured between a pair of virtual eyes;

determine a vector that originates at the midpoint and intersects the first point located on the hand of the user, wherein the vector represents a virtual gaze direction of the user;

calculate a first target position based on the virtual eye position;

calculate a second target position based on the pointing axis of the hand of the user;

determine the final target position based on the first target position and the second target position; and

control at least one vehicle system based at least in part on the final target position.

2. The vehicle of claim 1, wherein the machine-readable instructions further cause the one or more processors to:

determine a presence of an object located in an environment surrounding the vehicle that intersects with the vector representing the virtual gaze direction; and

identify the object that intersects with the vector as the first target position.

3. The vehicle of claim 1, wherein the machine-readable instructions further cause the one or more processors to:

calculate an angle between two respective vectors that intersect one another, wherein a first vector represents the virtual gaze direction and a second vector represents the pointing axis of the hand of the user; and

calculate an intersecting vector by dividing the angle into a first sub-angle and a second sub-angle, wherein the intersecting vector is directed towards the final target position.

4. A vehicle, comprising:

a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position;

a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user;

one or more processors; and

one or more non-transitory memory modules communicatively coupled to the one or more processors and storing machine-readable instructions that, when executed, cause the one or more processors to perform at least the following;

determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system, wherein the first point and the second point define a pointing axis of the hand of the user;

determine a vector by extending a line segment representing the pointing axis beyond the hand of the user, wherein the vector represents a pointing direction;

determine a presence of an object located in an environment surrounding the vehicle that intersects with the vector representing the pointing direction;

calculate a virtual eye position based at least in part on the first point located on the hand of the user and the actual eye position;

calculate a first target position based on the virtual eye position;

calculate a second target position based on the pointing axis of the hand of the user;

identify the object that intersects with the vector as the second target position;

determine the final target position based on the first target position and the second target position; and

control at least one vehicle system based at least in part on the final target position.

5. The vehicle of claim 4, wherein the machine-readable instructions further cause the one or more processors to:

determine that the user is pointing towards the second target position by the hand, wherein the hand includes a plurality of digits that each define a respective tip;

set the first point located on the hand of the user to the respective tip of a digit; and

determine the vector that represents the pointing direction by extending the line segment beyond the first point of the hand of the user.

6. The vehicle of claim 4, wherein the machine-readable instructions further cause the one or more processors to:

determine that the user is pointing towards the second target position by an article defining a proximate end and a distal end, wherein the user grasps the proximate end of the article with the hand and the distal end of the article is directed towards the second target position;

set the first point located on the hand of the user as the distal end of the article; and

determine the vector that represents the pointing direction by extending the line segment beyond the first point.

7. A method of determining a final target position that a user of a vehicle is gesturing towards, the method comprising:

determining, by a computer, a first point and a second point located on a hand of the user based at least in part on a gesture signal generated by a user detection system, wherein the first point and the second point define a pointing axis of the hand of the user;

calculating a virtual eye position based at least in part on the first point located on the hand of the user and an actual eye position of the user, wherein an eye location signal generated by a user gaze monitoring system generates the actual eye position;

determining a midpoint measured between a pair of virtual eyes;

determining a vector that originates at the midpoint and intersects the first point located on the hand of the user, wherein the vector represents a virtual gaze direction of the user;

calculating, by the computer, a first target position based on the virtual eye position;

calculating, by the computer, a second target position based on the pointing axis of the hand of the user;

determining the final target position based on the first target position and the second target position by the computer; and

controlling at least one vehicle system based at least in part on the final target position.

8. The method of claim 7, further comprising:

determining a presence of an object located in an environment surrounding the vehicle that intersects with the vector representing the virtual gaze direction; and

identifying the object that intersects with the vector as the first target position.

9. The method of claim 7, further comprising:

calculating an angle between two respective vectors that intersect one another, wherein a first vector represents the virtual gaze direction and a second vector represents the pointing axis of the hand of the user; and

calculating an intersecting vector by dividing the angle into a first sub-angle and a second sub-angle, wherein the intersecting vector is directed towards the final target position.