HEAD MOUNT DISPLAY DEVICE PROVIDING EYE GAZE CALIBRATION AND CONTROL METHOD THEREOF

• US 20140191927A1
• Filed: 02/21/2013
• Published: 07/10/2014
• Est. Priority Date: 01/09/2013
• Status: Abandoned Application

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: CHO, Eunhyung
• Application Number:
  US13/772,444
• Publication Number:
  US 20140191927A1
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

An eye gaze calibration method includes sensing an image of surroundings located in front of the HMD device using a camera unit, detecting a reference object from the sensed image of the surroundings using an image processing unit, detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit, and performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller, wherein detecting the reference object includes detecting at least one object contained in the sensed image as a first object, detecting a hand of the user contained in the sensed image as a second object, detecting an event of the second object with respect to the first object, and deciding the first object, on which the event has been detected, as the reference object.

Claims

1. A head mounted display (HMD) device comprising:

a camera unit configured to sense an image of surroundings located in front of the HMD device;

an image processing unit configured to detect a reference object from the sensed image of the surroundings;

an eye gaze detection unit configured to detect an eye gaze direction in which a user gazes at the detected reference object; and

a controller configured to perform eye gaze calibration based on the detected reference object and the detected eye gaze direction,

wherein the image processing unit is configured to

detect at least one object contained in the sensed image of the surroundings as a first object,

detect a hand of the user contained in the sensed image of the surroundings as a second object,

detect an event of the second object with respect to the first object, and

decide the first object, on which the event has been detected, as the reference object.

2. The HMD device according to claim 1, wherein

the first object is an external device,

the second object is a tip of a finger of the user, and

the event of the second object with respect to the first object comprises an operation of controlling the external device.

3. The HMD device according to claim 2, wherein

the first object is a keyboard, and

the event of the second object with respect to the first object comprises an operation of pressing a key of the keyboard.

4. The HMD device according to claim 1, wherein

the controller performs the eye gaze calibration based on a reference point located on the detected reference object, and

the reference point is a portion of the first object contacted by the second object to control the first object.

5. The HMD device according to claim 4, wherein

in a case in which the reference object is an electronic instrument,

the reference point comprises a control button of the electronic instrument, and

the event comprises an operation of controlling the control button of the electronic instrument.

6. The HMD device according to claim 5, wherein the control button of the electronic instrument comprises at least one selected from among a power button, volume control buttons, and channel control buttons.

7. The HMD device according to claim 4, wherein

in a case in which the reference object is a device comprising a touch sensitive display,

the reference point comprises a soft button displayed on the touch sensitive display.

8. The HMD device according to claim 1, wherein

in a case in which the first object is a pen,

the event of the second object with respect to the first object comprises a writing operation using the pen, and

the reference object is a tip of the pen.

9. The HMD device according to claim 1, further comprising:

a display unit configured to display a virtual object, wherein

the display unit controls a position of the displayed virtual object according to a result of the eye gaze calibration, and

the virtual object is a graphical user interface provided to the user by the display unit.

10. The HMD device according to claim 9, wherein the display unit displays the virtual object based on a position of the reference object using an augmented reality method.

11. The HMD device according to claim 9, wherein the display unit is a transparent optical display unit.

12. The HMD device according to claim 9, further comprising:

a distance measurement unit configured to measure a first distance from the HMD device to the detected reference object, wherein

the HMD device decides depth in which the virtual object is displayed based on the measured first distance.

13. The HMD device according to claim 12, wherein the distance measurement unit measures the first distance using at least one selected from among an ultrasonic sensor, a laser sensor, and an infrared sensor.

14. The HMD device according to claim 12, wherein the display unit provides a left eye and a right eye of the user with a left image and a right image, respectively, and expresses the depth of the virtual object using binocular disparity between the left image and the right image.

15. The HMD device according to claim 1, wherein the controller performs the eye gaze calibration only in a case in which a difference between a direction of the reference object and the eye gaze direction in which the user gazes at the reference object is within an allowable error range.

16. The HMD device according to claim 1, wherein the eye gaze detection unit detects the eye gaze direction based on movement of a pupil of the user.

17. The HMD device according to claim 1, wherein the second object is detected by at least one of a shape and a gesture of the hand of the user contained in the sensed image of the surroundings.

18. The HMD device according to claim 1, wherein the camera unit senses an image having a range corresponding to a visual field of the user with respect to the surroundings.

19. An eye gaze calibration method of an HMD device comprising:

sensing an image of surroundings located in front of the HMD device using a camera unit;

detecting a reference object from the sensed image of the surroundings using an image processing unit;

detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit; and

performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller,

wherein the step of detecting the reference object comprises;

detecting at least one object contained in the sensed image of the surroundings as a first object;

detecting a hand of the user contained in the sensed image of the surroundings as a second object;

detecting an event of the second object with respect to the first object; and

deciding the first object, on which the event has been detected, as the reference object.

20. An eye gaze calibration method of an HMD device comprising:

sensing an image of surroundings located in front of the HMD device using a camera unit;

detecting a reference object from the sensed image of the surroundings using an image processing unit;

detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit; and

performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller,

wherein the step of detecting the reference object comprises;

detecting a hand of the user contained in the sensed image of the surroundings as a first object;

detecting at least one object contained in the sensed image of the surroundings as a second object;

detecting an event of the first object with respect to the second object; and

deciding the second object, on which the event has been detected, as the reference object.

21. An HMD device comprising:

a camera unit configured to sense an image of surroundings located in front of the HMD device;

an image processing unit configured to detect a reference object from the sensed image of the surroundings;

an eye gaze detection unit configured to detect an eye gaze direction in which a user gazes at the detected reference object; and

a controller configured to perform eye gaze calibration based on the detected reference object and the detected eye gaze direction,

wherein the image processing unit is configured to

detect a hand of the user contained in the sensed image of the surroundings,

detect a gesture performed by the hand of the user, and

decide the detected hand of the user together with the gesture as the reference object.