Method and apparatus for generating mixed reality environment using drone equipped with stereo camera

• US 10,684,678 B1
• Filed: 12/10/2019
• Issued: 06/16/2020
• Est. Priority Date: 11/06/2019
• Status: Active Grant

• Current Assignee: Industry Academy Cooperation Foundation of Sejong University
• Original Assignee: Industry Academy Cooperation Foundation of Sejong University
• Inventors: Choi, Soo Mi, Kim, Dong Hyun, Go, Yong Guk
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US16/708,622
• Time in Patent Office: 189 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

Disclosed are a method and an apparatus for generating a mixed reality environment using a drone equipped with a stereo camera including a processor and a memory connected to the processor, wherein the memory stores program commands executed by the process to receive an image acquired by a stereo camera equipped with a drone and a depth value, identify an object from the image, convert a 2D central point coordinate of the identified object to a 3D coordinate through 3D mapping using the depth value, linear-interpolate a 3D coordinate of the object using a 3D coordinate of the object in a previous frame, a 3D coordinate of the object in a current frame, and a weight value, determine a position of a virtual object in association with the linear-interpolated 3D coordinate of the object, and render a mixed reality image including the virtual object.

Claims

1. An apparatus for generating a mixed reality environment, comprising:

a processor; and

a memory connected to the processor,

wherein the memory stores program commands executed by the process to receive an image acquired by a stereo camera equipped with a drone and a depth value, identify an object from the image, convert a 2D central point coordinate of the identified object to a 3D coordinate through 3D mapping using the depth value, linear-interpolate a 3D coordinate of the object using a 3D coordinate of the object in a previous frame, a 3D coordinate of the object in a current frame, and a weight value, determine a position of a virtual object in association with the linear-interpolated 3D coordinate of the object, and render a mixed reality image including the virtual object.

2. The apparatus for generating the mixed reality environment of claim 1, wherein the 2D central point coordinate of the identified object is converted using a size of the image acquired by the stereo camera and a size of a plane to be rendered.

3. The apparatus for generating the mixed reality environment of claim 1, wherein a correction value for the linear-interpolation is determined according to the following equation:

F(V_f)=(1−

δ

)V_f-1+δ

V

wherein, δ

represents a weight value, V_f-1 represents a 3D coordinate of the object in the previous frame, V_f represents a 3D coordinate of the object in the current frame, and F(V_F) represents a correction value of the 3D coordinate of the object in the current frame.

4. The apparatus for generating the mixed reality environment of claim 3, wherein the weight value for the linear-interpolation is determined by a time taken from the previous frame to the current frame and at least one of a traveling speed and a rotational speed of the drone.

5. The apparatus for generating the mixed reality environment of claim 4, wherein a weight value for the linear-interpolation is determined according to the following equation:

6. A method for generating a mixed reality environment in an apparatus including a processor and a memory connected to the processor, the method comprising:

receiving an image acquired by a stereo camera equipped with a drone and a depth value;

identifying an object from the image;

converting a 2D central point coordinate of the identified object to a 3D coordinate through 3D mapping using the depth value;

linear-interpolating a 3D coordinate of the object using a 3D coordinate of the object in a previous frame, a 3D coordinate of the object in a current frame, and a weight value;

determining a position of a virtual object in association with the linear-interpolated 3D coordinate of the object; and

rendering a mixed reality image including the virtual object.

7. A program stored in a non-transitory recording medium for performing the method of claim 6.