IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND STORAGE MEDIUM

• US 20200137371A1
• Filed: 10/22/2019
• Published: 04/30/2020
• Est. Priority Date: 10/26/2018
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Adachi, Daichi
• Granted Patent: US 11,277,596 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

An image processing apparatus for generating a virtual viewpoint image based on a plurality of images obtained by capturing an object in an image-capturing region from a plurality of directions by using a plurality of image capturing apparatuses includes a viewpoint acquisition unit configured to acquire viewpoint information indicating a position of a virtual viewpoint, and a generation unit configured to generate the virtual viewpoint image corresponding to the position of the virtual viewpoint indicated by the acquired viewpoint information, the virtual viewpoint image including at least one of an image depending on a positional relationship between a boundary and the virtual viewpoint and/or an image representing the boundary, the boundary being between an inside and an outside of a predetermined region included in the image-capturing region.

Claims

1. An image processing apparatus for generating a virtual viewpoint image based on a plurality of images obtained by capturing an object in an image-capturing region from a plurality of directions by using a plurality of image capturing apparatuses, the image processing apparatus comprising:

a viewpoint acquisition unit configured to acquire viewpoint information indicating a position of a virtual viewpoint; and

a generation unit configured to generate the virtual viewpoint image corresponding to the position of the virtual viewpoint indicated by the acquired viewpoint information, the virtual viewpoint image including at least one of an image depending on a positional relationship between a boundary and the virtual viewpoint and an image representing the boundary, the boundary being between an inside and an outside of a predetermined region included in the image-capturing region,

wherein the object can be present across the inside and the outside of the predetermined region, and the virtual viewpoint is movable between the inside and the outside of the predetermined region.

2. The image processing apparatus according to claim 1, further comprising:

a determination unit configured to determine at least one of whether to display the image representing the boundary in the virtual viewpoint image and a pattern of the image, based on the positional relationship between the virtual viewpoint indicated by the viewpoint information and the boundary,

wherein the virtual viewpoint image is generated based on a result of determination by the determination unit.

3. The image processing apparatus according to claim 2, wherein, at least one of a color and a transparency of the image representing the boundary in the virtual viewpoint image is determined based on the positional relationship between the virtual viewpoint and the boundary.

4. The image processing apparatus according to claim 2, wherein, based on the positional relationship between the virtual viewpoint and the boundary, the image processing apparatus determines an image captured by which image capturing apparatus from among the plurality of image capturing apparatuses is to be used to determine a color of the image representing the boundary.

5. The image processing apparatus according to claim 1, further comprising:

an information acquisition unit configured to acquire region information including information about coordinates corresponding to vertexes of the predetermined region,

wherein the virtual viewpoint image is generated based on the acquired region information and the acquired viewpoint information.

6. The image processing apparatus according to claim 1, wherein the generation unit generates the virtual viewpoint image including an image, as the image corresponding to the positional relationship between the virtual viewpoint and the boundary, indicating the height of the virtual viewpoint and the height of the boundary.

7. The image processing apparatus according to claim 1, wherein the generation unit generates the virtual viewpoint image including an image, as the image corresponding to the positional relationship between the virtual viewpoint and the boundary, indicating whether the position of the virtual viewpoint is included in the predetermined region.

8. The image processing apparatus according to claim 1,

wherein the predetermined region is an underwater region, and

wherein the boundary of the predetermined region is a water surface.

9. The image processing apparatus according to claim 8, wherein the generation unit generates the virtual viewpoint image including a part of an image representing the water surface included in the image-capturing region of the plurality of image capturing apparatuses, the part being determined based on a position of the object.

10. The image processing apparatus according to claim 1, wherein the generation unit generates the virtual viewpoint image in which at least one of a color and a transparency of an image representing a water surface is different between a case where the position of the virtual viewpoint indicated by the viewpoint information is below the water surface and a case where the position of the virtual viewpoint indicated by the viewpoint information is above the water surface.

11. An image processing method for generating a virtual viewpoint image based on a plurality of images obtained by capturing an object in an image-capturing region from a plurality of directions by using a plurality of image capturing apparatuses, the image processing method comprising:

acquiring viewpoint information indicating a position of a virtual viewpoint; and

generating the virtual viewpoint image corresponding to the position of the virtual viewpoint indicated by the acquired viewpoint information, the virtual viewpoint image including at least one of an image depending on a positional relationship between a boundary and the virtual viewpoint and an image representing the boundary, the boundary being between an inside and an outside of a predetermined region included in the image-capturing region,

wherein the object can be present across the inside and the outside of the predetermined region, and the virtual viewpoint is movable between the inside and the outside of the predetermined region.

12. The image processing method according to claim 11, further comprising

determining at least one of whether to display the image representing the boundary in the virtual viewpoint image and a pattern of the image, based on the positional relationship between the virtual viewpoint indicated by the viewpoint information and the boundary,

wherein the virtual viewpoint image is generated based on a result of the determining

13. The image processing Method according to claim 11,

wherein the predetermined region is an underwater region, and

wherein the boundary is a water surface.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method for generating a virtual viewpoint image based on a plurality of images obtained by capturing an object in an image-capturing region from a plurality of directions by using a plurality of image capturing apparatuses, the image processing method comprising:

acquiring viewpoint information indicating a position of a virtual viewpoint; and

generating the virtual viewpoint image corresponding to the position of the virtual viewpoint indicated by the acquired viewpoint information, the virtual viewpoint image including at least one of an image depending on a positional relationship between a boundary and the virtual viewpoint and an image representing the boundary, the boundary being between an inside and an outside of a predetermined region included in the image-capturing region,

wherein the object can be present across an inside and an outside of the predetermined region, and the virtual viewpoint is movable between the inside and the outside of the predetermined region.