Head mounted display providing eye gaze calibration and control method thereof

• US 9,529,442 B2
• Filed: 12/27/2013
• Issued: 12/27/2016
• Est. Priority Date: 01/09/2013
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Cho, Eunhyung, Yim, Jungbin, Choi, Jiho
• Primary Examiner(s): Awad, Amr
• Assistant Examiner(s): BRAY, STEPHEN A
• Application Number:
  US14/142,564
• Publication Number:
  US 20140191946A1
• Time in Patent Office: 1,096 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A control method of a head mounted display (HMD) is disclosed. The control method of the HMD includes detecting a first route of a first moving object and a second route of a second moving object in front of the HMD, detecting a third route along which user'"'"'s eye gaze moves, setting the first moving object as a reference object if the detected first route is substantially identical with the detected third route, setting the second moving object as the reference object if the detected second route is substantially identical with the detected third route, and performing an eye gaze calibration based on a route of the set reference object and the detected third route.

Claims

1. A head mounted display (HMD) comprising:

a camera unit configured to detect a first route of a first moving object and a second route of a second moving object in front of the HMD;

a display unit configured to display visual information;

an eye gaze detection unit configured to detect a third route along which an eye gaze of a user moves; and

a controller configured to perform an eye gaze calibration based on a route of a reference object and the detected third route,

wherein the controller is further configured to;

select the reference object from a plurality of real moving objects including the first moving object and the second moving object, wherein the reference object is selected by comparing the third route of the eye gaze with each route of the plurality of real moving objects, wherein the plurality of real moving objects are not images displayed by the display unit of the head mounted display;

change a position of the displayed visual image according to the result of the eye gaze calibration;

if the detected first route is substantially identical with the detected third route, set the first moving object as the reference object; and

if the detected second route is substantially identical with the detected third route, set the second moving object as the reference object.

2. The HMD according to claim 1, further comprising:

a display unit configured to display visual information,

wherein the controller further changes a position of the displayed visual information according to a result of the eye gaze calibration.

3. The HMD according to claim 2, wherein the visual information is displayed between an eye of the user and the reference object.

4. The HMD according to claim 1, wherein, if a plurality of moving objects having the same route is detected by the camera unit, the controller sets the reference object from among the detected moving objects according to object priority information.

5. The HMD according to claim 1, wherein

the route of the reference object is a route of a reference point located on the detected reference object, and

the reference point is a portion of the reference object.

6. The HMD according to claim 5, wherein

the reference point is decided as a portion of the reference object according to point priority information, and

if the reference object is a human being, a face of the human being is decided as the reference point.

7. The HMD according to claim 1, wherein, if the reference object moves from a first position to a second position, the controller

detects a user'"'"'s first eye gaze direction when the reference object is at the first position,

detects a user'"'"'s second eye gaze direction when the reference object is at the second position, and

performs the eye gaze calibration based on the first position, the second position, the first eye gaze direction, and the second eye gaze direction.

8. The HMD according to claim 1, wherein, if a position of the reference object moves from a first position to a second position, the controller continuously performs the eye gaze calibration based on the position of the reference object and the user'"'"'s eye gaze movement along the reference object.

9. The HMD according to claim 1, wherein, if a plurality of moving objects is detected by the camera unit, the controller puts at least one of the moving objects satisfying at least one selected from among a velocity condition, a size condition, a moving direction condition, and a color condition in a reference object candidate group.

10. The HMD according to claim 1, further comprising:

a display unit configured to display a virtual object, wherein

the controller further controls a position of the displayed virtual object according to a result of the eye gaze calibration, and

the virtual object is a graphical user interface provided to the user through the display unit.

11. The HMD according to claim 10, wherein the display unit displays the virtual object based on a position of the reference object using augmented reality technology.

12. The HMD according to claim 10, further comprising:

a distance measurement unit configured to measure a first distance from the HMD to the detected reference object, wherein

the controller further decides a display depth of the virtual object in response to the measured first distance.

13. The HMD according to claim 12, wherein

the display unit provides a left eye and a right eye of the user with a left image and a right image, respectively, and

the display unit expresses the depth of the virtual object using binocular disparity between the left image and the right image.

14. The HMD according to claim 12, wherein, if the reference object moves, the display unit changes the depth of the virtual object in response to a moved position of the reference object.

15. The HMD according to claim 1, wherein the eye gaze calibration is performed when a wearing position of the HMD is changed or when change of the user'"'"'s eye gaze exceeds a threshold level.

16. The HMD according to claim 1, wherein the eye gaze detection unit detects a user'"'"'s eye gaze direction based on movement of a pupil of the user.

17. The HMD according to claim 1, wherein the camera unit senses an image having the same range as a visual field of the user.

18. The HMD according to claim 1, wherein the controller performs the eye gaze calibration only when the route of the reference object and the third route of the user'"'"'s eye gaze are substantially identical with each other by a first rate or more.

19. The HMD according to claim 1, wherein the controller compares a direction, velocity, and movement distance of the first moving object moving along the first route with a direction, velocity, and movement distance of the user'"'"'s eye gaze to determine whether the detected first route is substantially identical with the detected third route.

20. A control method of a HMD comprising:

detecting, via a camera unit, a first route of a first moving object and a second route of a second moving object in front of the HMD;

display visual information on a display unit;

detecting, via an eye gaze detection unit, a third route along which user'"'"'s eye gaze moves;

selecting, via a controller, a reference object from a plurality of real moving objects including the first moving object and the second moving object, wherein the reference object is selected by comparing the third route of the eye gaze with each route of the plurality of real moving objects, wherein the plurality of real moving objects are not images displayed by the display unit of the head mounted display;

changing, via the controller, a position of the displayed visual information according to a result of the eye gaze calibration;

setting, via the controller, the first moving object as the reference object if the detected first route is substantially identical with the detected third route;

setting, via the controller, the second moving object as the reference object if the detected second route is substantially identical with the detected third route; and

performing, via the controller, an eye gaze calibration based on a route of the set reference object and the detected third route.