Head mounted display providing eye gaze calibration and control method thereof

• US 9,619,021 B2
• Filed: 09/18/2014
• Issued: 04/11/2017
• Est. Priority Date: 01/09/2013
• Status: Expired due to Fees

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Cho, Eunhyung, Chun, Sinae
• Primary Examiner(s): Awad, Amr
• Assistant Examiner(s): BRAY, STEPHEN A
• Application Number:
  US14/490,365
• Publication Number:
  US 20150002394A1
• Time in Patent Office: 936 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A control method of a head mounted display (HMD) is disclosed. The control method includes detecting a first route of a first moving object and a second route of a second moving object in front of the HMD, detecting a third route along which a user'"'"'s eye gaze moves, setting the first moving object as a reference object if the detected first route identical with the detected third route, setting the second moving object as the reference object if the detected second route identical with the detected third route, and setting a focus of the camera unit on the set reference object based on a route of the set reference object and the detected third route.

Claims

1. A head mounted display (HMD) comprising:

a camera unit configured to detect a first route of a first moving object and a second route of a second moving object in front of the HMD;

an eye gaze detection unit configured to detect a third route along which a user'"'"'s eye gaze moves;

a display unit configured to display visual information; and

a controller configured to;

control a focus of the camera unit based on a route of a reference object and the detected third route; and

display the visual information between an eye of the user and the reference object,

wherein the controller is further configured to;

select the reference object from a plurality of real moving objects including the first moving object and the second moving object, wherein the reference object is selected by comparing the third route of the eye gaze with each route of the plurality of real moving objects, wherein the plurality of real moving objects are not images displayed by the display unit,

if the detected first route of the first moving object is substantially identical with the detected third route, set the first moving object as the reference object,

if the detected second route of the second moving object is substantially identical with the detected third route, set the second moving object as the reference object, and

set the focus of the camera unit on the set reference object.

2. The HMD according to claim 1, wherein the controller is further configured to display the visual information based on a display property,

wherein first visual information corresponding to the reference object on which the focus of the camera unit is set has a display property different from second visual information corresponding to an object on which the focus of the camera unit is not set.

3. The HMD according to claim 2, wherein the display property comprises at least one selected from among sharpness, resolution, and brightness.

4. The HMD according to claim 1, wherein the controller is further configured to:

display first visual information corresponding to the reference object on which the focus of the camera unit is set with first sharpness and

display second visual information corresponding to an object on which the focus of the camera unit is not set with second sharpness,

wherein the first sharpness is higher than the second sharpness.

5. The HMD according to claim 1, wherein the controller is further configured to display a focusing indicator using the display unit,

wherein the focusing indicator indicates the reference object on which the focus of the camera unit is set.

6. The HMD according to claim 5, wherein the controller is further configured to display the focusing indicator as a virtual object,

wherein the virtual object is a graphical user interface provided to the user through the display unit.

7. The HMD according to claim 6, wherein the display unit displays the focusing indicator as the virtual object based on a position of the reference object using augmented reality technology.

8. The HMD according to claim 7, further comprising:

a distance measurement unit configured to measure a first distance from the HMD to the detected reference object,

wherein the controller is further configured to decide a display depth of the focusing indicator as the virtual object in response to the measured first distance.

9. The HMD according to claim 1, wherein the controller is further configured to:

if a plurality of moving objects having the same route is detected by the camera unit, set the reference object from among the detected moving objects according to object priority information.

10. The HMD according to claim 1, wherein the controller is further configured to:

set the route of the reference object as a route of a reference point located on the detected reference object, and

perform auto focusing based on the route of the reference point,

wherein the reference point is a portion of the reference object.

11. The HMD according to claim 10, wherein the reference point is set as a portion of the reference object according to point priority information.

12. The HMD according to claim 10, wherein the controller is further configured to:

if the focus of the camera unit based on the route of the reference point is controlled, set the focus of the camera unit on a portion of the reference object set as the reference point.

13. The HMD according to claim 1, wherein the controller is further configured to:

if a position of the reference object moves from a first position to a second position, detect a user'"'"'s first eye gaze direction when the reference object is located at the first position, and a user'"'"'s second eye gaze direction when the reference object is located at the second position, and

control the focus of the camera unit based on the first position, the second position, the first eye gaze direction, and the second eye gaze direction.

14. The HMD according to claim 1, wherein the controller is further configured to:

if a position of the reference object moves from a first position to a second position, continuously control the focus of the camera unit based on the position of the reference object and the user'"'"'s eye gaze moving along the reference object.

15. The HMD according to claim 1, wherein the controller is further configured to:

if a plurality of moving objects is detected by the camera unit, put at least one of the moving objects satisfying at least one selected from among a velocity condition, a size condition, a moving direction condition, and a color condition in a reference object candidate group.

16. The HMD according to claim 1, wherein

the eye gaze detection unit detects a user'"'"'s eye gaze direction based on movement of a pupil of the user and detects the third route based on the eye gaze direction, and

the camera unit senses an image having the same range as a visual field of the user.

17. The HMD according to claim 1, wherein

the camera unit periodically detects the first route and the second route based on an update cycle,

the eye gaze detection unit periodically detects the third route based on the update cycle, and

wherein the controller is further configured to update the reference object on which the focus of the camera unit is set every update cycle.

18. The HMD according to claim 1, wherein the controller is further configured to compare a direction, velocity, and movement distance of the first moving object moving along the first route with a direction, velocity, and movement distance of the user'"'"'s eye gaze to determine whether the detected first route identical with the detected third route.

19. A control method of an HMD comprising:

detecting, via a camera unit, a first route of a first moving object and a second route of a second moving object in front of the HMD;

detecting, via an eye gaze detection unit, a third route along which a user'"'"'s eye gaze moves;

selecting, via a controller, a reference object from a plurality of real moving objects including the first moving object and the second moving object, wherein the reference object is selected by comparing the third route of the eye gaze with each route of the plurality of real moving objects;

setting, via the controller, the first moving object as the reference object if the detected first route of the first moving object is substantially identical with the detected third route;

setting, via the controller, the second moving object as the reference object if the detected second route of the second moving object is substantially identical with the detected third route;

setting, via the controller, a focus of the camera unit on the set reference object based on a route of the set reference object and the detected third route, and

displaying on a display unit, via the controller, visual information between an eye of the user and the reference object,

wherein the plurality of real moving objects are not images displayed by the display unit.