Line-of-sight tracking system and method

• US 9,160,929 B2
• Filed: 04/26/2013
• Issued: 10/13/2015
• Est. Priority Date: 12/11/2012
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group)
• Original Assignee: Hyundai Motor Company (Hyundai Motor Group)
• Inventors: Ro, Hee Jin, Lee, Seok Beom, Seok, Dong Hee, Park, Sung Min
• Primary Examiner(s): KOSTAK, VICTOR R
• Application Number:
  US13/871,678
• Publication Number:
  US 20140160249A1
• Time in Patent Office: 900 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A display system includes a line-of-sight tracking camera to detect an eyeball of a driver, first and second stereo cameras, a controlling unit, and a storing unit. The first and second stereo camera photographs a range corresponding to a field-of-view based on stereo camera-based line-of-sight information that is changed based on line-of-sight tracking camera-based line-of-sight information, and provide a photographed image. The controlling unit converts the line-of-sight tracking camera-based line-of-sight information into first and second stereo camera-based line-of-sight information based on pre-stored position and rotation information of the tracking camera and the first and second stereo cameras, and project the converted information onto the first and second stereo cameras, such that a three-dimensional line-of-sight coordinate is calculated. The storing unit stores information related to a system as well as the position and rotation information of the tracking camera and the first and second stereo cameras.

Claims

1. A display system, comprising:

a line-of-sight tracking camera configured to detect an eyeball of a driver;

first and second stereo cameras configured to photograph a range corresponding to a driver'"'"'s field-of-view based on stereo camera-based driver'"'"'s line-of-sight information that is changed based on line-of-sight tracking camera-based driver'"'"'s line-of-sight information, and provide a photographed image;

a controlling unit configured to convert the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into first and second stereo camera-based driver'"'"'s line-of-sight information based on pre-stored position information and rotation information of the line-of-sight tracking camera and the first and second stereo cameras, and project the converted information onto the first and second stereo cameras, such that a three-dimensional coordinate of a driver'"'"'s line-of-sight is calculated; and

a storing unit configured to store information related to the display system as well as the position information and the rotation information of the line-of-sight tracking camera and the first and second stereo cameras.

2. The display system according to claim 1, wherein:

the line-of-sight tracking camera-based driver'"'"'s line-of-sight information includes a line-of-sight tracking vector monocular eye point and a line-of-sight tracking vector, and

the controlling unit is configured to convert the line-of-sight tracking camera-based line-of-sight tracking vector monocular eye point into a stereo camera-based line-of-sight tracking vector monocular eye point.

3. The display system according to claim 2, wherein the controlling unit is configured to convert the line-of-sight tracking camera-based line-of-sight tracking vector monocular eye point (Location) (x_loc^eye, y_loc^eye, z_loc^eye, 1) into a stereo camera-based line-of-sight tracking vector monocular eye point (x_loc^global, y_loc^global, z_loc^global, 1) through the following Equation 1:

4. The display system according to claim 1, wherein:

the line-of-sight tracking camera-based driver'"'"'s line-of-sight information includes a line-of-sight tracking vector monocular eye point and a line-of-sight tracking vector, and

the controlling unit is configured to convert the line-of-sight tracking camera-based line-of-sight tracking vector into a stereo camera-based line-of-sight tracking vector.

5. The display system according to claim 4, wherein the controlling unit is configured to convert the line-of-sight tracking camera-based line-of-sight tracking vector (x_vec^eye,y_vec^eye,z_vec^eye,1) into a stereo camera-based line-of-sight tracking vector (Rotation) (x_vec^global,y_vec^global,z_vec^global,1) through the following Equation 2:

6. The display system according to claim 1, wherein the controlling unit is configured to calculate a driver'"'"'s line-of-sight gaze point from a stereo camera-based line-of-sight tracking vector projected onto the first and second stereo cameras and calculate the three-dimensional coordinate of the driver'"'"'s line-of-sight based on the calculated driver'"'"'s line-of-sight gaze point.

7. The display system according to claim 6, wherein the controlling unit is configured to generate a window corresponding to a preset main gaze range of a person'"'"'s line-of-sight on an image projected onto the first stereo camera, move a monocular eye point of the generated window according to a first stereo camera-based line-of-sight tracking vector, perform template matching such that the moved window corresponds to a line-of-sight tracking vector of the second stereo camera, and recognize a highest position as a driver'"'"'s line-of-sight gaze point as a result of the performing the template matching.

8. The display system according to claim 1, wherein the three-dimensional coordinate of the driver'"'"'s line-of-sight is P(Xp, Yp, Zp),

9. A display method providing a three-dimensional driver'"'"'s line-of-sight in a display system including a line-of-sight tracking camera and first and second stereo cameras, the display method comprising:

detecting an eyeball of a driver through the line-of-sight tracking camera to recognize line-of-sight tracking camera-based driver'"'"'s line-of-sight information;

converting the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into first and second stereo camera-based driver'"'"'s line-of-sight information based on pre-stored position information and rotation information of the line-of-sight tracking camera and the first and second stereo cameras;

recognizing a driver'"'"'s line-of-sight gaze point based on the first and second stereo camera-based driver'"'"'s line-of-sight information; and

converting the driver'"'"'s line-of-sight gaze point into a three-dimensional coordinate of a driver'"'"'s line-of-sight.

10. The display method according to claim 9, wherein:

the line-of-sight tracking camera-based driver'"'"'s line-of-sight information includes a line-of-sight tracking vector monocular eye point and a line-of-sight tracking vector, and

the converting of the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into the first and second stereo camera-based driver'"'"'s line-of-sight information includes converting the line-of-sight tracking camera-based line-of-sight tracking vector monocular eye point into a stereo camera-based line-of-sight tracking vector monocular eye point.

11. The display method according to claim 10, wherein the converting of the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into the first and second stereo camera-based driver'"'"'s line-of-sight information includes converting the line-of-sight tracking camera-based line-of-sight tracking vector monocular eye point (Location) (x_loc^eye, y_loc^eye, z_loc^eye,1) into a stereo camera-based line-of-sight tracking vector monocular eye point (x_loc^global, y_loc^global, z_loc^global, 1) through the following Equation 1:

12. The display method according to claim 9, wherein:

the line-of-sight tracking camera-based driver'"'"'s line-of-sight information includes a line-of-sight tracking vector monocular eye point and a line-of-sight tracking vector, and

the converting of the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into the first and second stereo camera-based driver'"'"'s line-of-sight information includes converting the line-of-sight tracking camera-based line-of-sight tracking vector into a stereo camera-based line-of-sight tracking vector.

13. The display method according to claim 12, wherein the converting of the line-of-sight tracking camera-based driver'"'"'s line-of-sight information into the first and second stereo camera-based driver'"'"'s line-of-sight information includes converting the line-of-sight tracking camera-based line-of-sight tracking vector (x_vec^eye,y_vec^eye,z_vec^eye,1) into a stereo camera-based line-of-sight tracking vector (Rotation) (x_vec^global,y_vec^global,z_vec^global,1) through the following Equation 2:

14. The display method according to claim 9, wherein the recognizing of the driver'"'"'s line-of-sight gaze point includes:

projecting the first and second stereo camera-based driver'"'"'s line-of-sight information onto the first and second stereo cameras; and

recognizing the driver'"'"'s line-of-sight gaze point based on the information projected onto the first and second stereo cameras.

15. The display method according to claim 14, wherein the recognizing of the driver'"'"'s line-of-sight gaze point based on the information projected onto the first and second stereo cameras includes:

generating a window corresponding to a preset main gaze range of a person'"'"'s line-of-sight on an image projected onto the first stereo camera;

moving a monocular eye point of the generated window according to a first stereo camera-based line-of-sight tracking vector;

performing template matching such that the moved window corresponds to a line-of-sight tracking vector of the second stereo camera; and

recognizing a highest position as a driver'"'"'s line-of-sight gaze point as a result of the performance.

16. The display method according to claim 14, wherein the three-dimensional coordinate of the driver'"'"'s line-of-sight is P(Xp, Yp, Zp),