Information processing apparatus, information processing method and storing unit

• US 10,771,761 B2
• Filed: 12/10/2018
• Issued: 09/08/2020
• Est. Priority Date: 12/12/2017
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Mizuno, Shogo
• Primary Examiner(s): Bader, Robert
• Application Number:
  US16/214,685
• Publication Number:
  US 20190182470A1
• Time in Patent Office: 638 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An information processing apparatus includes one or more memories storing instructions, and one or more processors executing the instructions to determine, among a plurality of captured images obtained by a plurality of image capturing apparatuses, a display target image related to a virtual viewpoint image, based on a position of a virtual viewpoint and a view direction from the virtual viewpoint, the virtual viewpoint image being generated based on the display target image and the position of the virtual viewpoint and the view direction from the virtual viewpoint, and to cause a displaying unit to display the determined display target image in a displaying mode according to a degree of contribution of the determined display target image to generation of the virtual viewpoint image.

Claims

1. An information processing apparatus comprising:

one or more memories storing instructions; and

one or more processors executing the instructions;

to determine, among a plurality of captured images obtained by a plurality of cameras, a display target image related to a virtual viewpoint image, based on a position of a virtual viewpoint and a view direction from the virtual viewpoint, the virtual viewpoint image being generated based on the display target image and the position of the virtual viewpoint and the view direction from the virtual viewpoint; and

to cause a displaying unit to display the determined display target image in a displaying mode according to a degree of contribution of the determined display target image to generation of the virtual viewpoint image.

2. The information processing apparatus according to claim 1, wherein an image contributing to determination of a three-dimensional shape of an object included in the virtual viewpoint image is determined to be the display target image, the virtual viewpoint image being generated based on the determination of the three-dimensional shape of the object.

3. The information processing apparatus according to claim 1, wherein an image contributing to determination of a color of an object included in the virtual viewpoint image is determined to be the display target image, the virtual viewpoint image being generated based on the determination of the color of the object.

4. The information processing apparatus according to claim 1, wherein the display target image is displayed in a displaying mode according to a positional relationship between the virtual viewpoint and a camera obtaining the determined display target image.

5. The information processing apparatus according to claim 4, wherein the display target image is displayed in the displaying mode based on which direction a camera obtaining the determined display target image is located with respect to the virtual viewpoint.

6. The information processing apparatus according to claim 4, wherein the display target image is displayed in the displaying mode according to how far a camera obtaining the determined display target image is located from the virtual viewpoint.

7. The information processing apparatus according to claim 6, wherein, in a case when, among cameras respectively obtaining a plurality of determined display target images, a first camera is located farther from the virtual viewpoint than a second camera, a captured image obtained by the first camera being smaller than a captured image obtained by the second camera.

8. The information processing apparatus according to claim 1, wherein the virtual viewpoint image, the display target image, and the attribute information of a camera obtaining the display target image are displayed.

9. The information processing apparatus according to claim 1, wherein at least two captured images are determined, as the display target images, from among the plurality of captured images obtained by the plurality of cameras, based on a distance between cameras among the plurality of cameras.

10. The information processing apparatus according to claim 9, wherein captured images respectively obtained by two cameras having a greatest distance among the plurality of cameras.

11. The information processing apparatus according to claim 9, wherein a virtual viewpoint image being based on an intermediate position between cameras respectively obtaining the at least the two captured images is further displayed.

12. The information processing apparatus according to claim 1, wherein the one or more processors further execute the instructions:

to accept a user operation for designating one or more images among a plurality of determined display target images; and

to control operation, not to use, in the generation of the virtual viewpoint image, the image designated by the accepted user operation.

13. An information processing method that is performed by an information processing apparatus, the information processing method comprising:

determining, among a plurality of captured images obtained by a plurality of cameras, a display target image related to a virtual viewpoint image, based on a position of a virtual viewpoint and a view direction from the virtual viewpoint, the virtual viewpoint image being generated based on the display target image and the position of the virtual viewpoint and the view direction from the virtual viewpoint; and

causing a displaying unit to display the determined display target image in a displaying mode according to a degree of contribution of the determined display target image to generation of the virtual viewpoint image.

14. A non-transitory computer-readable storing unit that stores a program for causing a computer to function as each of following units of an information processing apparatus:

a determining unit configured to determine, among a plurality of captured images obtained by a plurality of cameras, a display target image related to a virtual viewpoint image, based on a position of a virtual viewpoint and a view direction from the virtual viewpoint, the virtual viewpoint image being generated based on the display target image and the position of the virtual viewpoint and the view direction from the virtual viewpoint; and

a display controlling unit configured to cause a displaying unit to display the determined display target image in a displaying mode according to a degree of contribution of the determined display target image to generation of the virtual viewpoint image.