IMAGE GENERATION SYSTEM, IMAGE GENERATION METHOD, AND INFORMATION STORAGE MEDIUM

• US 20120056992A1
• Filed: 09/07/2011
• Published: 03/08/2012
• Est. Priority Date: 09/08/2010
• Status: Active Grant

• Current Assignee: Bandai Namco Entertainment Inc. (BANDAI NAMCO Holdings Inc.)
• Original Assignee: Namco Bandai Games Inc. (BANDAI NAMCO Holdings Inc.)
• Inventors: KURODA, Yosuke
• Granted Patent: US 9,049,428 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An image generation system includes a captured image acquisition section that acquires a captured image captured by an imaging section, a depth information acquisition section that acquires depth information about a photographic object observed within the captured image, an object processing section that performs a process that determines a positional relationship between the photographic object and a virtual object in a depth direction based on the acquired depth information, and synthesizes the virtual object with the captured image, and an image generation section that generates an image in which the virtual object is synthesized with the captured image.

Claims

1. An image generation system comprising:

a captured image acquisition section that acquires a captured image captured by an imaging section;

a depth information acquisition section that acquires depth information about a photographic object observed within the captured image;

an object processing section that performs a process that determines a positional relationship between the photographic object and a virtual object in a depth direction based on the acquired depth information, and synthesizes the virtual object with the captured image; and

an image generation section that generates an image in which the virtual object is synthesized with the captured image.

2. The image generation system as defined in claim 1,

the object processing section performing a process that synthesizes an image in which the photographic object is displayed on a front side of the virtual object when it has been determined that the photographic object is positioned on a front side of the virtual object in the depth direction based on the depth information.

3. The image generation system as defined in claim 2,

the object processing section determining the positional relationship between the virtual object and the photographic object in the depth direction based on the depth information about the photographic object and depth information about a placement position determination photographic object for determining a placement position of the virtual object.

4. The image generation system as defined in claim 3, further comprising:

a captured image storage section that stores image data about a plurality of live-view images captured by the imaging section,

the depth information acquisition section acquiring the depth information about the placement position determination photographic object based on the plurality of live-view images, and

the object processing section determining the positional relationship between the virtual object and the photographic object in the depth direction based on the depth information about the photographic object and the depth information about the placement position determination photographic object acquired from the plurality of live-view images.

5. The image generation system as defined in claim 4,

the depth information acquisition section acquiring the depth information about the placement position determination photographic object based on a live-view image among the plurality of live-view images, in which it has been determined that a placement position determination area of the placement position determination photographic object is not hidden behind the photographic object.

6. The image generation system as defined in claim 1,

the object processing section performing a hit determination process on the photographic object and the virtual object based on the depth information.

7. The image generation system as defined in claim 6,

the object processing section setting a hit volume for the virtual object based on depth information about a placement position determination photographic object for determining a placement position of the virtual object, and performing the hit determination process on the photographic object and the virtual object using the set hit volume.

8. The image generation system as defined in claim 6,

the object processing section performing at least one of a process that changes a display state of the virtual object and a process that outputs sound corresponding to the virtual object when it has been determined that the photographic object has hit the virtual object by the hit determination process.

9. The image generation system as defined in claim 1,

the object processing section performing a movement process of the virtual object based on the depth information.

10. The image generation system as defined in claim 9,

the object processing section moving the virtual object between a position of a placement position determination photographic object for determining a placement position of the virtual object and a position of the photographic object based on depth information about the placement position determination photographic object and the depth information about the photographic object.

11. The image generation system as defined in claim 1,

the captured image acquisition section acquiring a first captured image captured by a first camera included in the imaging section as a left-eye image, and acquiring a second captured image captured by a second camera included in the imaging section as a right-eye image, and

the depth information acquisition section acquiring the depth information based on parallax information obtained from the left-eye image and the right-eye image.

12. The image generation system as defined in claim 11,

the image generation section generating an image in which the virtual object is synthesized with a stereoscopic image generated using the left-eye image and the right-eye image.

13. The image generation system as defined in claim 1,

the captured image acquisition section acquiring a depth image from a depth camera included in the imaging section, and

the depth information acquisition section acquiring the depth information based on the depth image.

14. An image generation method comprising:

acquiring a captured image captured by an imaging section;

acquiring depth information about a photographic object observed within the captured image;

performing a process that determines a positional relationship between the photographic object and a virtual object in a depth direction based on the acquired depth information, and synthesizes the virtual object with the captured image; and

generating an image in which the virtual object is synthesized with the captured image.

15. The image generation method as defined in claim 14, further comprising:

performing a process that synthesizes an image in which the photographic object is displayed on a front side of the virtual object when it has been determined that the photographic object is positioned on a front side of the virtual object in the depth direction based on the depth information.

16. The image generation method as defined in claim 15, further comprising:

determining the positional relationship between the virtual object and the photographic object in the depth direction based on the depth information about the photographic object and depth information about a placement position determination photographic object for determining a placement position of the virtual object.

17. The image generation method as defined in claim 16, further comprising:

storing image data about a plurality of live-view images captured by the imaging section;

acquiring the depth information about the placement position determination photographic object based on the plurality of live-view images; and

determining the positional relationship between the virtual object and the photographic object in the depth direction based on the depth information about the photographic object and the depth information about the placement position determination photographic object acquired from the plurality of live-view images.

18. The image generation method as defined in claim 17, further comprising:

acquiring the depth information about the placement position determination photographic object based on a live-view image among the plurality of live-view images, in the live-view image, it has been determined that a placement position determination area of the placement position determination photographic object is not hidden behind the photographic object.

19. A computer-readable information storage medium storing a program that causes a computer to execute the image generation method as defined in claim 14.