THREE-DIMENSIONAL ENVIRONMENT SHARING SYSTEM AND THREE-DIMENSIONAL ENVIRONMENT SHARING METHOD

• US 20150213649A1
• Filed: 03/08/2013
• Published: 07/30/2015
• Est. Priority Date: 07/27/2012
• Status: Abandoned Application

• Current Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Original Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Inventors: Morishita, Koji, Nagai, Katsuyuki, Noda, Hisashi
• Application Number:
  US14/417,201
• Publication Number:
  US 20150213649A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A first image processing device includes a first image acquisition unit that acquires a first captured image from a first imaging unit, a detection unit that detects a known common real object from the first captured image, a setting unit that sets a three-dimensional coordinate space on the basis of the common real object, and a transmission unit that transmits three-dimensional position information of the three-dimensional coordinate space to a second image processing device, and a second image processing device includes an acquisition unit that acquires a second captured image from a second imaging unit, a detection unit that detects the known common real object from the second captured image, a setting unit that sets the same three-dimensional coordinate space as the three-dimensional coordinate space set by the first image processing device on the basis of the common real object, a reception unit that receives the three-dimensional position information from the first image processing device, and a processing unit that processes virtual three-dimensional object data to be synthesized with the second captured image by using the three-dimensional position information.

Claims

1. A three-dimensional environment sharing system comprising:

a first image processing device; and

a second image processing device,

wherein the first image processing device includes;

a first image acquisition unit that acquires a first captured image from a first imaging unit;

a first object detection unit that detects a known common real object from the first captured image acquired by the first image acquisition unit;

a first coordinate setting unit that sets a three-dimensional coordinate space on the basis of the common real object detected by the first object detection unit; and

a transmission unit that transmits three-dimensional position information of the three-dimensional coordinate space to the second image processing device,

wherein the second image processing device includes;

a second image acquisition unit that acquires a second captured image from a second imaging unit which is disposed at a position and in a direction different from a position and a direction of the first imaging unit and of which an imaging region overlaps at least part of an imaging region of the first imaging unit;

a second object detection unit that detects the known common real object from the second captured image acquired by the second image acquisition unit;

a second coordinate setting unit that sets the same three-dimensional coordinate space as the three-dimensional coordinate space set by the first image processing device on the basis of the common real object detected by the second object detection unit;

a reception unit that receives the three-dimensional position information from the first image processing device; and

an object processing unit that processes virtual three-dimensional object data to be synthesized with the second captured image by using the three-dimensional position information received by the reception unit.

2. The three-dimensional environment sharing system according to claim 1,

wherein the first image processing device further includes:

a first reference setting unit that calculates a position and a direction of the first imaging unit on the basis of the common real object detected by the first object detection unit;

a first image synthesizing unit that synthesizes a virtual three-dimensional object disposed at a position within the three-dimensional coordinate space corresponding to the three-dimensional position information with the first captured image on the basis of the position and the direction of the first imaging unit and the three-dimensional coordinate space; and

a first display processing unit that displays the synthesized image obtained by the first image synthesizing unit on a first display unit, and

wherein the second image processing device further includes;

a second reference setting unit that calculates a position and a direction of the second imaging unit on the basis of the common real object detected by the second object detection unit;

a second image synthesizing unit that synthesizes a virtual three-dimensional object corresponding to virtual three-dimensional object data processed by the object processing unit with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

a second display processing unit that displays the synthesized image obtained by the second image synthesizing unit on a second display unit.

3. The three-dimensional environment sharing system according to claim 1,

wherein the first imaging unit is a three-dimensional sensor,

wherein the first image acquisition unit further acquires depth information corresponding to the first captured image from the first imaging unit along with the first captured image,

wherein the first image processing device further includes:

a first reference setting unit that calculates a position and a direction of the first imaging unit on the basis of the common real object detected by the first object detection unit; and

a position calculation unit that acquires three-dimensional position information of a specific subject included in the first captured image by using the first captured image and the depth information, and converts the acquired three-dimensional position information of the specific subject into three-dimensional position information of the three-dimensional coordinate space on the basis of the position and the direction of the first imaging unit calculated by the first reference setting unit and the three-dimensional coordinate space, and

wherein the second image processing device further includes;

a second reference setting unit that calculates a position and a direction of the second imaging unit on the basis of the common real object detected by the second object detection unit;

a second image synthesizing unit that synthesizes a virtual three-dimensional object corresponding to virtual three-dimensional object data processed by the object processing unit with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

a second display processing unit that displays the synthesized image obtained by the second image synthesizing unit on a second display unit.

4. A three-dimensional environment sharing method performed by a first image processing device and a second image processing device, the method comprising:

causing the first image processing device to

acquire a first captured image from a first imaging unit;

detect a known common real object from the acquired first captured image;

set a three-dimensional coordinate space on the basis of the detected common real object; and

transmit three-dimensional position information of the three-dimensional coordinate space to the second image processing device, and

causing the second image processing device to

acquire a second captured image from a second imaging unit which is disposed at a position and in a direction different from a position and a direction of the first imaging unit and of which an imaging region overlaps at least part of an imaging region of the first imaging unit;

detect the known common real object from the second captured image acquired by the second image acquisition unit;

set the same three-dimensional coordinate space as the three-dimensional coordinate space set by the first image processing device on the basis of the detected common real object;

receive the three-dimensional position information from the first image processing device; and

process virtual three-dimensional object data to be synthesized with the second captured image by using the received three-dimensional position information.

5. The three-dimensional environment sharing method according to claim 4, further comprising:

causing the first image processing device to

calculate a position and a direction of the first imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object disposed at a position within the three-dimensional coordinate space corresponding to the three-dimensional position information with the first captured image on the basis of the position and the direction of the first imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a first display unit, and

causing the second image processing device to

calculate a position and a direction of the second imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object corresponding to the processed virtual three-dimensional object data with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a second display unit.

6. The three-dimensional environment sharing method according to claim 4,

wherein the first imaging unit is a three-dimensional sensor,

wherein the method further includes:

causing the first image processing device to

acquire depth information corresponding to the first captured image from the first imaging unit,

calculate a position and a direction of the first imaging unit on the basis of the detected common real object;

acquire three-dimensional position information of a specific subject included in the first captured image by using the first captured image and the depth information; and

convert the acquired three-dimensional position information of the specific subject into three-dimensional position information of the three-dimensional coordinate space on the basis of the position and the direction of the first imaging unit and the three-dimensional coordinate space, and

causing the second image processing device to

calculate a position and a direction of the second imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object corresponding to the processed virtual three-dimensional object data with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a second display unit.

7. (canceled)

8. A non-transitory computer-readable storage medium storing a program for causing a first image processing device and a second image processing device to perform a three-dimensional environment sharing method, the method including:

causing the first image processing device to

acquire, by the first image processing device, a first captured image from a first imaging unit;

detect a known common real object from the acquired first captured image;

set a three-dimensional coordinate space on the basis of the detected common real object; and

transmit three-dimensional position information of the three-dimensional coordinate space to the second image processing device, and

causing the second image processing device to

acquire a second captured image from a second imaging unit which is disposed at a position and in a direction different from a position and a direction of the first imaging unit and of which an imaging region overlaps at least part of an imaging region of the first imaging unit;

detect the known common real object from the second captured image acquired by the second image acquisition unit;

set the same three-dimensional coordinate space as the three-dimensional coordinate space set by the first image processing device on the basis of the detected common real object;

receive the three-dimensional position information from the first image processing device; and

process virtual three-dimensional object data to be synthesized with the second captured image by using the received three-dimensional position information.

9. The non-transitory computer-readable storage medium according to claim 8, the three-dimensional environment sharing method further comprising:

causing the first image processing device to

calculate a position and a direction of the first imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object disposed at a position within the three-dimensional coordinate space corresponding to the three-dimensional position information with the first captured image on the basis of the position and the direction of the first imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a first display unit, and

causing the second image processing device to

calculate a position and a direction of the second imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object corresponding to the processed virtual three-dimensional object data with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a second display unit.

10. The non-transitory computer-readable storage medium according to claim 8,

wherein the first imaging unit is a three-dimensional sensor,

wherein the three-dimensional environment sharing method further includes:

causing the first image processing device to

acquire depth information corresponding to the first captured image from the first imaging unit,

calculate a position and a direction of the first imaging unit on the basis of the detected common real object;

acquire three-dimensional position information of a specific subject included in the first captured image by using the first captured image and the depth information; and

convert the acquired three-dimensional position information of the specific subject into three-dimensional position information of the three-dimensional coordinate space on the basis of the position and the direction of the first imaging unit and the three-dimensional coordinate space, and

causing the second image processing device to

calculate a position and a direction of the second imaging unit on the basis of the detected common real object;

synthesize a virtual three-dimensional object corresponding to the processed virtual three-dimensional object data with the second captured image on the basis of the position and the direction of the second imaging unit and the three-dimensional coordinate space; and

display the synthesized image obtained through the synthesizing on a second display unit.