Movable display apparatus, robot having movable display apparatus and display method thereof

• US 9,298,254 B2
• Filed: 12/22/2009
• Issued: 03/29/2016
• Est. Priority Date: 01/20/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: Ha, Tae Sin, Han, Woo Sup, Choi, Byung Kwon
• Primary Examiner(s): Sax, Steven
• Assistant Examiner(s): Fibbi, Christopher J
• Application Number:
  US12/654,536
• Publication Number:
  US 20100185990A1
• Time in Patent Office: 2,289 Days
• Field of Search: 715/863, 382/153
• US Class Current: 1/1
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A movable display apparatus, a robot having the movable display apparatus, and a display method thereof, and, more particularly, a display method of a robot having an apparatus to display an image according to a visual point of a user are provided. It is possible to provide a convenient extended image service, by movably mounting the apparatus to display the image according to the visual point of the user and mounting the movable display apparatus in the robot so as to accurately display the image according to the visual point of the user using the mobility and motion of the robot. In addition, it is possible to provide an image, which is viewed like a three-dimensional image, via a two-dimensional display apparatus, by changing the displayed image according to a variation in the sightline of the user.

Claims

1. A movable display apparatus comprising:

a display unit configured to display an image;

a camera provided on the display unit and configured to acquire an image of a user;

a head mounted device (HMD) configured to measure a head rotation direction and a pupil direction of the user;

a control unit configured to recognize a sightline of the user using the image of the user acquired by the camera and the head rotation direction and the pupil direction measured by the HMD, control a posture of the display unit according to the recognized sightline of the user, and control the displaying of the image by the display unit according to the recognized sightline of the user;

a camera image processing unit configured to extract a face direction of the user from the image of the user acquired by the camera, wherein the control unit is further configured to estimate the head rotation direction for recognizing the sightline of the user using the extracted face direction and the measured head rotation direction of the user; and

a driving unit configured to move the display unit under the control of the control unit, wherein the display unit is configured to be movably provided on a manipulator to display the image, and the control unit is configured to compute a difference between a sightline position of the user based the recognized sightline of the user and a current position of the display unit, and generate a motion path of the manipulator.

2. The movable display apparatus according to claim 1, wherein the camera image processing unit is further configured to extract the pupil direction of the user from the user image acquired by the camera, wherein the control unit is further configured to estimate the pupil direction for recognizing the sightline of the user using the extracted pupil direction and the measured pupil direction of the user.

3. The movable display apparatus according to claim 1, wherein the control unit is further configured to control the displaying of the image by the display unit according to a variation in the sightline of the user.

4. The movable display apparatus according to claim 1, wherein the control unit is further configured to decide the posture of the display unit such that a center of mass (COM), according to the motion of the display unit, is positioned on a center of the display unit.

5. A robot comprising:

a display unit movably provided on a manipulator and configured to display an image;

a camera provided on the display unit and configured to acquire an image of a user;

a head mounted device (HMD) configured to measure a head rotation direction and a pupil direction of the user;

a control unit configured to recognize a sightline of the user using the image of the user acquired by the camera and the head rotation direction and the pupil direction measured by the HMD, control a posture of the display unit according to the recognized sightline of the user, and control the displaying of the image by the display unit according to the recognized sightline of the user;

a driving unit configured to move the manipulator and change the posture of the display unit under the control of the control unit; and

a camera image processing unit configured to extract a face direction of the user from the image of the user acquired by the camera,

wherein the control unit is further configured to estimate the head rotation direction for recognizing the sightline of the user using the extracted face direction and the measured head rotation direction of the user, compute a difference between a sightline position of the user based on the recognized sightline of the user and the current position of the display unit, and generate a motion path of the manipulator.

6. The robot according to claim 5, wherein the camera image processing unit is further configured to extract the pupil direction of the user from the image of the user acquired by the camera, and the control unit is further configured to estimate the pupil direction for recognizing the sightline of the user using the extracted pupil direction and the measured pupil direction of the user.

7. The robot according to claim 5, wherein the control unit is further configured to check a variation in the center of mass (COM) when the display unit is positioned at the sightline position of the user, and move a main body of the robot.

8. The robot according to claim 5, wherein the control unit is further configured to change the image displayed on the display unit according to a variation in the sightline of the user.

9. A display method of a robot, the method comprising:

providing an image service using a display unit movably provided on a manipulator;

acquiring an image of a user who receives the image service;

measuring a head rotation direction and a pupil direction of the user using a head mounted device (HMD); and

recognizing a sightline of the user using the acquired image of the user and the measured head rotation direction and pupil direction of the user;

controlling a posture of the display unit according to the recognized sightline of the user to track the sightline of the user; and

controlling the displaying of the image by the display unit according to the recognized sightline of the user,

extracting a face direction of the user from the acquired image of the user,

wherein the controlling a posture of the display unit includes estimating the head rotation direction for recognizing the sightline of the user using the extracted face direction and the measured head rotation direction of the user, computing a difference between a sightline position of the user based on the recognized sightline of the user and a current position of the display unit, and generating a motion path of the manipulator.

10. The method according to claim 9, further comprising extracting a face direction of the user from the acquired image of the user, wherein the recognizing of the sightline of the user includes estimating the head rotation direction of the user using the extracted face direction and the measured head rotation direction of the user.

11. The method according to claim 9, further comprising extracting the pupil direction of the user from the acquired image of the user, wherein the recognizing of the sightline of the user includes estimating the pupil direction of the user using the extracted pupil direction and the measured pupil direction of the user.

12. The method according to claim 10, wherein the recognizing of the sightline of the user includes recognizing a sightline position of the user using the head rotation direction and the pupil direction.

13. The method according to claim 12, wherein the tracking of the sightline of the user includes checking a variation in the center of mass (COM) of the robot when the display unit is positioned at the sightline position of the user, and moving a main body of the robot.

14. The method according to claim 9, wherein the tracking of the sightline of the user includes changing the image displayed on the display unit according to a variation in the sightline of the user.

15. A robot comprising:

a display unit movably provided on a manipulator and configured to display an image;

a camera provided on the display unit configured to acquire an image of a user;

a data communication unit configured to receive a head rotation direction and a pupil direction of the user;

a control unit configured to recognize a sightline of the user using the image of the user acquired by the camera and the head rotation direction and the pupil direction received by the data communication unit, control a posture of the display unit according to the recognized sightline of the user, and atoll control the displaying of the image by the display unit according to the recognized sightline of the user;

a camera image processing unit configured to extract a face direction of the user from the image of the user acquired by the camera, wherein the control unit is further configured to estimate the head rotation direction for recognizing the sightline of the user using the extracted face direction and the received head rotation direction of the user; and

a driving unit configured to move the manipulator and change the posture of the display unit under the control of the control unit, wherein the control unit is configured to compute a difference between a sightline position of the user based on the sightline of the user and the current position of the display unit, and generate a motion path of the manipulator.

16. The robot according to claim 15, further comprising:

a head mounted device (HMD) affixed to the head of the user and configured to transmit the head rotation direction and the pupil direction received by the data communication unit.

17. The robot according to claim 15, wherein the robot is configured to change at least one of a position and an angle of the display unit according to the sightline of the user recognized by the control unit.