Virtual reality system using mixed reality and implementation method thereof

• US 10,587,868 B2
• Filed: 05/25/2017
• Issued: 03/10/2020
• Est. Priority Date: 05/26/2016
• Status: Active Grant

• Current Assignee: Kyungpook National University
• Original Assignee: Kyungpook National University
• Inventors: Yun, Byoung-Ju, Lee, Sang-Hoon
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/302,094
• Publication Number:
  US 20190156586A1
• Time in Patent Office: 1,020 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/00 Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 9/75 : Chroma key

Abstract

A virtual reality system using mixed reality according to the present invention includes: a chroma-key environment booth in which a chroma-key screen is formed in a space corresponding to a viewing range of a user; a depth camera unit installed on an upper surface of the chroma-key environment booth and capturing the user and generating depth image information; an HMD device in which a stereoscopic camera worn on a head of the user and performing capturing at positions corresponding to both eyes of the user is positioned, and a head tracking marker is positioned on an upper end of the stereoscopic camera; and a mixed reality processing device detecting the head tracking marker positioned in the HMD device by receiving the depth image information and detecting position and rotation values of the head of the user in an actual environment.

Claims

1. A virtual reality system using mixed reality, comprising:

a chroma-key environment booth in which a chroma-key screen is formed in a space corresponding to a viewing range of a user;

a depth camera unit vertically installed on an upper surface of the chroma-key environment booth and capturing the user and generating depth image information;

a head mounted display (HMD) device having a stereoscopic camera, a head tracking marker, a direction sensor, and a HMD output unit,

the stereoscopic camera provided to correspond to both eyes of the user and located in front portion of the HMD device,

the head tracking marker located on an upper end of the stereoscopic camera,

the direction sensor located between the two eyes of the user and configured to sense a direction of the user'"'"'s head, and

the HMD output unit configured to output an image onto planes corresponding the both eyes of the user; and

a mixed reality processing device

configured to detect the head tracking marker positioned in the HMD device by receiving the depth image information,

configured to detect a position and a rotation value of the head of the user in an actual environment,

configured to correct the rotation value according to a direction sensing value detected by the direction sensor,

configured to apply the detected position value and the corrected rotation value to a virtual reality rendering camera for virtual reality generating output image information output to the HMD device to correspond to the position value and the corrected rotation value,

configured to overlay image information for a body of the user, which is captured by the stereoscopic camera on the output image information, and

configured to match a reference point in an actual environment, which is marked in the chroma-key environment booth and a reference point in the output image information for the virtual reality.

2. The virtual reality system using mixed reality of claim 1, wherein a space corresponding to the viewing range is acquired by adding a capturing angle of the stereoscopic camera to a predetermined angle.

3. The virtual reality system using mixed reality of claim 1, wherein the direction sensor is configured to detect a direction corresponding to a visual field of the user.

4. The virtual reality system using mixed reality of claim 1, wherein the mixed reality processing device receives and registers position and size information for an interactive object, detects whether the body of the user, which is detected by the depth camera overlaps with an area corresponding to the position and size of the interactive object, and adds, when the body and the interactive object overlap with each other, information indicating that the body and the interactive object overlap with each other to the output image information.

5. An implementation method of virtual reality using mixed reality, comprising:

capturing, by a depth camera vertically installed on an upper surface of a chroma-key environment booth in which a chroma-key screen is formed in a space corresponding to a viewing range of a user, the user to generate depth image information;

outputting, by a head mounted display (HMD) device worn on a head of the user, output image information from the outside by approaching both eyes of the user, wherein the HMD device has a stereoscopic camera, a head tracking marker, a direction sensor, and a HMD output unit, wherein the direction sensor is located between the two eyes of the user and configured to sense a direction of the user'"'"'s head;

capturing, by a stereoscopic camera installed at a position corresponding to the both eyes of the user and located in front of the HMD device, a front surface of the user, wherein the head tracking marker is formed on an upper end of the stereoscopic camera;

detecting, by a mixed reality processing device, the head tracking marker positioned in the HMD device by receiving the depth image information and detecting a position and a rotation value of the head of the user in an actual environment, correcting the rotation value according to a direction sensing value detected by the direction sensor, applying the detected position value and the corrected rotation value to a virtual reality rendering camera for virtual reality generating output image information output to the HMD device to correspond to the position value and the corrected rotation value, overlaying image information for a body of the user, which is captured by the stereoscopic camera on the output image information, and matching a reference point in an actual environment, which is marked in the chroma-key environment booth and a reference point in the output image information for the virtual reality.

6. The implementation method of virtual reality using mixed reality of claim 5, wherein a space corresponding to the viewing range is acquired by adding a capturing angle of the stereoscopic camera to a predetermined angle.

7. The implementation method of virtual reality using mixed reality of claim 5, wherein the direction sensor is configured to detect a direction corresponding to a visual field of the user.

8. The implementation method of virtual reality using mixed reality of claim 5, wherein the mixed reality processing device

receives and registers position and size information for an interactive object,

detects whether the body of the user, which is detected by the depth camera overlaps with an area corresponding to the position and size of the interactive object, and

adds, when the body and the interactive object overlap with each other, information indicating that the body and the interactive object overlap with each other to the output image information.