IMAGE GENERATION APPARATUS AND IMAGE GENERATION METHOD

• US 20140240454A1
• Filed: 11/07/2011
• Published: 08/28/2014
• Est. Priority Date: 11/07/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Hirata, Shinichi, Furumura, Kyoko, Yamagishi, Takeshi
• Granted Patent: US 9,560,274 B2
• Time in Patent Office: Days
• US Class Current: 348/38
• CPC Class Codes: G03B 19/023 Multi-image cameras
• CPC Class Codes: G03B 37/00 Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Methods and apparatus provide for generating a component image in which, with images picked up using a front side camera and a rear side camera built in portable equipment, posture information of the portable equipment is associated; combining a plurality of component images between which the posture of the portable equipment is different to generate a synthesis image; changing over an image pickup mode between a double-sided synthesis image pickup mode in which a synthesis image of one half circumference picked up by the front side camera and a synthesis image of one half circumference picked up by the rear side camera are connected to each other to pick up a synthesis image of a whole circumference and a single-sided synthesis image pickup mode in which a synthesis image of a whole circumference is picked up by each of the front side camera and the rear side camera.

Claims

1. An image generation apparatus, comprising:

a component image generation unit configured to generate a component image in which, with images picked up using a plurality of cameras built in portable equipment and having optical axes different in at least one of a position and an orientation from each other, posture information of the portable equipment is associated;

a synthesis image generation unit configured to combine a plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover unit configured to change over an image pickup mode between a compound eye synthesis image pickup mode in which a compound eye synthesis image formed by connecting synthesis images picked up by the plurality of cameras to each other is picked up and a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation unit combining a plurality of component images picked up by each of the cameras to generate a single eye synthesis image and connecting the single eye synthesis images by the cameras to each other to generate a compound eye synthesis image.

2. The image generation apparatus according to claim 1, wherein, when the image pickup mode is the single eye synthesis image pickup mode, the synthesis image generation unit combines a plurality of component images picked up by each of the cameras to generate a single eye synthesis image and combines and outputs the single eye synthesis images by the cameras into and as a stereo synthesis image or a plurality of synthesis images which provide different senses of distance.

3. The image generation apparatus according to claim 1, further comprising:

an image correlation decision unit configured to decide a degree of correlation between the single eye synthesis images to be connected to each other in a boundary region in which the single eye synthesis images by the cameras are to be connected to each other;

when the degree of correlation is equal to or higher than a predetermined threshold value, the synthesis image generation unit connecting the single eye synthesis images by the cameras to each other to generate a compound eye synthesis image.

4. The image generation apparatus according to claim 1, further comprising:

a camera operator detection portion configured to detect catching of a camera operator in an image picked up by a front side camera;

a camera operator image pickup permission portion configured to issue an inquiry to a user about whether or not the catching of the camera operator is permitted; and

a region-of-interest setting portion configured to set, when the catching of the camera operator is permitted, based on an image of a facial expression of the camera operator picked up by a certain camera, a region-of-interest to an image picked up by a different camera;

the component image generation unit generating a component image in which the posture information of the portable equipment and the region-of-interest information set by the region-of-interest setting portion are associated with the image picked up using the different camera.

5. The image generation apparatus according to claim 1, wherein

the plurality of cameras built in the portable equipment and having the optical axes different in at least one of the position and the orientation from each other are a front side camera provided on the camera operator side and a rear side camera provided on the opposite side to the camera operator;

the image pickup mode changeover unit changes over the image pickup mode between a double-sided synthesis image pickup mode in which synthesis images of one half circumference picked up by the front side camera and synthesis images of one half circumference picked up by the rear side camera are connected to each other to pick up a synthesis image of a whole circumference and a single-sided synthesis image pickup mode in which a synthesis image of a whole circumference is picked up by each of the front side camera and the rear side camera; and

when the image pickup mode is the double-sided synthesis image pickup mode, the synthesis image generation unit combines a plurality of front side component images picked up by the front side camera to generate a front side synthesis image of one half circumference, combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of one half circumference, and then connects the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference to each other to generate a synthesis image of a whole circumference.

6. The image generation apparatus according to claim 5, wherein, when the image pickup mode is the single-sided synthesis image pickup mode, the synthesis image generation unit discards a plurality of front side component images picked up by the front side camera and combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of a whole circumference.

7. The image generation apparatus according to claim 5, wherein, when the image pickup mode is the single-sided synthesis image pickup mode, the synthesis image generation unit combines a plurality of front side component images picked up by the front side camera to generate a front side synthesis image of a whole circumference, combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of a whole circumference, and combines and outputs the front side synthesis image of the whole circumference and the rear side synthesis image of the whole circumference into and as a stereo synthesis image or synthesis images of a near view and a distant view.

8. The image generation apparatus according to claim 5, further comprising:

an image correlation decision unit configured to decide a degree of correlation between the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference in a boundary region in which the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference are connected to each other;

when the degree of correlation is equal to or higher than a predetermined threshold value, the synthesis image generation unit connecting the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference to each other to generate a synthesis image of the whole circumference.

9. The image generation apparatus according to claim 8, further comprising an image pickup guide portion configured to guide the user, when the degree of correlation is lower than the predetermined threshold value, so as to carry out image pickup for further one half circumference.

10. The image generation apparatus according to any claim 5, further comprising:

a camera operator detection portion configured to detect catching of a camera operator in an image picked up by the front side camera;

a camera operator image pickup permission portion configured to issue an inquiry to the user about whether or not the catching of the camera operator is permitted; and

a region-of-interest setting portion configured to set, when the catching of the camera operator is permitted, based on an image of a facial expression of the camera operator picked up by the front side camera, a region-of-interest to an image picked up by the rear side camera;

the component image generation unit generating a component image in which the posture information of the portable equipment and the region-of-interest information set by the region-of-interest setting portion are associated with the image picked up using the rear side camera.

11. An image generation method, comprising:

a component image generation step of generating a component image in which, with images picked up using a plurality of cameras built in portable equipment and having optical axes different in at least one of a position and an orientation from each other, posture information of the portable equipment is associated;

a synthesis image generation step of combining a plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover step of changing over an image pickup mode between a compound eye synthesis image pickup mode in which a compound eye synthesis image formed by connecting synthesis images picked up by the plurality of cameras to each other is picked up and a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation step combining a plurality of component images picked up by each of the cameras to generate a single eye synthesis image and connecting the single eye synthesis images by the cameras to each other to generate a compound eye synthesis image.

12. A non-transitory, computer-readable recording medium storing a program, the program causing a computer to implement:

a component image generation function for generating a component image in which, with images picked up using a plurality of cameras built in portable equipment and having optical axes different in at least one of a position and an orientation from each other, posture information of the portable equipment is associated;

a synthesis image generation function for combining a plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover function for changing over an image pickup mode between a compound eye synthesis image pickup mode in which a compound eye synthesis image formed by connecting synthesis images picked up by the plurality of cameras to each other is picked up and a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation function combining a plurality of component images picked up by each of the cameras to generate a single eye synthesis image and connecting the single eye synthesis images by the cameras to each other to generate a compound eye synthesis image.