Remote collaboration system with projector-camera based robot device and head mounted display, and remote interaction method using the same

• US 10,250,845 B1
• Filed: 07/19/2018
• Issued: 04/02/2019
• Est. Priority Date: 10/19/2017
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Ahn, Sang Chul, Kwon, Jin Uk
• Primary Examiner(s): Woo, Stella L.
• Application Number:
  US16/039,436
• Publication Number:
  US 20190124294A1
• Time in Patent Office: 257 Days
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085

Abstract

In a remote interaction method performed by a remote collaboration system comprising a robot device and a head mounted display, the robot device is located in a remote space and comprising a projector, a panoramic camera and a high resolution camera. The head mounted display is located in a local space apart from the remote space. A communication between the robot device and the head mounted display is established based on a communication request. The remote space is observed by the head mounted display based on first image information and second image information. The first image information is collected by the panoramic camera and received from the robot device. The second image information is collected by the high resolution camera and received from the robot device. Necessary information to be provided to the remote space is searched by the head mounted display based on a result of observing the remote space. The necessary information received from the head mounted display is provided by the robot device to the remote space using the projector.

Claims

1. A remote interaction method for providing a remote collaboration system, the system comprising a robot device and a head mounted display, the robot device comprising a projector, a panoramic camera and a high resolution camera, and the robot device and the head mounted display being located space apart from each other, the method comprising:

in response to receipt of a communication request, establishing a communication between the robot device and the head mounted display;

observing, by the head mounted display, a remote space based on first image information and second image information, the first image information, corresponding to panoramic image of the remote space, being collected by the panoramic camera, the second image information, corresponding to a high resolution image or a magnified image of a region in the remote space, being collected by the high resolution camera, and the first and second image information being transmitted from the robot device, wherein

observing the remote space is performed by following steps;

displaying, by the head mounted display, a panoramic image of the remote space based on the first image information,

in response to detection of a screen magnifying interaction while the head mounted display displaying the panoramic image, displaying, by the head mounted display, the panoramic image and a magnified image of a first region in the remote space together based on the first image information and the second image information,

in response to detection of the screen magnifying interaction while the head mounted display displaying the panoramic image and the magnified image together, displaying, by the head mounted display, the magnified image based on the second image information,

in response to detection of a screen demagnifying interaction, while the head mounted display displaying the panoramic image and the magnified image together, displaying, by the head mounted display, the panoramic image based on the first image information, and

in response to detection of the screen demagnifying interaction, while the head mounted display displaying the magnified image, displaying, by the head mounted display, the panoramic image and the magnified image together based on the first image information and the second image information;

searching, by the head mounted display, necessary information to be provided to the remote space based on a result of observing the remote space; and

providing, by the robot device, the necessary information received from the head mounted display to the remote space using the projector.

2. The remote interaction method of claim 1, wherein observing the remote space further comprises:

removing the magnified image displayed on the head mounted display when a removing interaction is performed while the head mounted display displays the panoramic image and the magnified image together, or a user interface object displayed on the head mounted display when the removing interaction is performed while the head mounted display displays the panoramic image or the magnified image with the user interface object, the user interface object being used for controlling an operation of the head mounted display.

3. The remote interaction method of claim 1, wherein searching the necessary information comprises:

in response to detection of a screen demagnifying interaction is performed while the head mounted display displays the panoramic image, observing, by the head mounted display, the local space based on third image information collected by a first camera included in the head mounted display;

performing a first searching operation on a plurality of first data stored in the head mounted display; and

obtaining the necessary information based on at least one of the result of observing the remote space, a result of observing the local space and a result of the first searching operation.

4. The remote interaction method of claim 3, wherein searching the necessary information further comprises:

performing a second searching operation on a plurality of second data stored in the robot device; and

performing a third searching operation on a plurality of third data associated with the necessary information by connecting an Internet network via the head mounted display, and

wherein the necessary information is obtained further based on at least one of a result of the second searching operation and a result of the third searching operation.

5. The remote interaction method of claim 1, wherein providing the necessary information to the remote space comprises:

in response to detection of the necessary information being provided as an image displaying, by the projector, the necessary information on the remote space; and

in response to detection of the necessary information being associated with the first region in the remote space, displaying, by the projector, a pointer on the first region in the remote space.

6. The remote interaction method of claim 5, wherein both the high resolution camera and the projector are controlled toward the first region in the remote space in response to detection of the head mounted display displaying the panoramic image and the magnified image together or only the magnified image and in response to detection of the necessary information being associated with the first region in the remote space and the projector displaying the pointer on the first region in the remote space.

7. The remote interaction method of claim 6, wherein the remote collaboration system further comprises a controller that is located in the local space and controls an operation of the head mounted display,

wherein direction controls of the high resolution camera and the projector are performed based on head tracking information obtained by the head mounted display, and

wherein a movement of the pointer displayed on the first region in the remote space is controlled by the controller.

8. A remote collaboration system comprising:

a robot device located in a remote space and comprising a projector, a panoramic camera and a high resolution camera, the projector being configured to provide necessary information to the remote space, the panoramic camera being configured to collect first image information of the remote space, the high resolution camera being configured to collect second image information of the remote space; and

a head mounted display located in a local space apart from the remote space, wherein

in response to receipt of a communication request, a communication between the robot device and the head mounted display is established, the head mounted display observes the remote space based on the first image information and the second image information, the first image information corresponding to panoramic image of the remote space being collected by the panoramic camera and the second image information corresponding to a high resolution image or a magnified image of a region in the remote space being collected by the high resolution camera, and the first and second image information being transmitted from the robot device, and

the head mounted display searches the necessary information based on a result of observing the remote space, and the robot device provides the necessary information received from the head mounted display to the remote space using the projector, and wherein

observing the remote space is performed by;

the head mounted display displays the panoramic image of the remote space based on the first image information,

in response to detection of a screen magnifying interaction while the head mounted display displaying the panoramic image, the head mounted display displays a panoramic image and a magnified image of a first region in the remote space together based on the first image information and the second image information,

in response to detection of the screen magnifying interaction while the head mounted display displaying the panoramic image and the magnified image together, the head mounted display displays the magnified image based on the second image information,

in response to detection of a screen demagnifying interaction, while the head mounted display displaying the panoramic image and the magnified image together, the head mounted display displays the panoramic image based on the first image information, and

in response to detection of the screen demagnifying interaction, while the head mounted display displaying the magnified image, the head mounted display displays the panoramic image and the magnified image together based on the first image information and the second image information.