Image generation apparatus and image generation method

• US 9,560,274 B2
• Filed: 11/07/2011
• Issued: 01/31/2017
• Est. Priority Date: 11/07/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Hirata, Shinichi, Furumura, Kyoko, Yamagishi, Takeshi
• Primary Examiner(s): Philippe, Gims
• Assistant Examiner(s): Pezzner, Benjamin
• Application Number:
  US14/354,671
• Publication Number:
  US 20140240454A1
• Time in Patent Office: 1,912 Days
• Field of Search: 348/38
• US Class Current: 1/1
• CPC Class Codes: G03B 19/023 Multi-image cameras
• CPC Class Codes: G03B 37/00 Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Methods and apparatus provide for generating a component image in which, with images picked up using a front side camera and a rear side camera built in portable equipment, posture information of the portable equipment is associated; combining a plurality of component images between which the posture of the portable equipment is different to generate a synthesis image; changing over an image pickup mode between a double-sided synthesis image pickup mode in which a synthesis image of one half circumference picked up by the front side camera and a synthesis image of one half circumference picked up by the rear side camera are connected to each other to pick up a synthesis image of a whole circumference and a single-sided synthesis image pickup mode in which a synthesis image of a whole circumference is picked up by each of the front side camera and the rear side camera.

Claims

1. An image generation apparatus, comprising:

a component image generation unit configured to generate a plurality of component images from a plurality of sequential images picked up using a plurality of cameras built in portable equipment, a first camera of the plurality of cameras being position toward a first side of a camera operator and a second camera of the plurality of cameras being positioned at least partially toward a second side of the camera operator, each of the plurality of cameras having optical axes different in at least one of a position and an orientation from each other, the component images including posture information of the portable equipment associated with the plurality of sequential images;

a synthesis image generation unit configured to combine the plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover unit configured to change over an image pickup mode between;

(i) a compound eye synthesis image pickup mode in which synthesis images of a first partial circumference picked up by the first camera of the plurality of cameras are connected to synthesis images of a second partial circumference picked up by the second camera of the plurality of cameras, and (ii) a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation unit;

(i) combines a plurality of first component images picked up by the first of the plurality of cameras to generate a first side synthesis image, (ii) combines a plurality of second component images picked up by the second of the plurality of cameras to generate a second side synthesis image, and (iii) connects the first side synthesis image and the second side synthesis image to each other to generate a compound synthesis image.

2. The image generation apparatus according to claim 1, wherein, when the image pickup mode is the single eye synthesis image pickup mode, the synthesis image generation unit combines a plurality of component images picked up by each of the cameras to generate a single eye synthesis image and combines and outputs the single eye synthesis images by the cameras into and as a stereo synthesis image or a plurality of synthesis images which provide different senses of distance.

3. The image generation apparatus according to claim 1, further comprising:

an image correlation decision unit configured to decide a degree of correlation between the single eye synthesis images to be connected to each other in a boundary region in which the single eye synthesis images by the cameras are to be connected to each other;

when the degree of correlation is equal to or higher than a predetermined threshold value, the synthesis image generation unit connects the single eye synthesis images by the cameras to each other to generate a compound eye synthesis image.

4. The image generation apparatus according to claim 1, further comprising:

a camera operator detection portion configured to detect catching of a camera operator in an image picked up by a front side camera;

a camera operator image pickup permission portion configured to issue an inquiry to a user about whether or not the catching of the camera operator is permitted; and

a region-of-interest setting portion configured to set, when the catching of the camera operator is permitted, based on an image of a facial expression of the camera operator picked up by a certain camera, a region-of-interest to an image picked up by a different camera;

the component image generation unit generating the component images such that the posture information of the portable equipment and the region-of-interest information set by the region-of-interest setting portion are associated with the image picked up using the different camera.

5. The image generation apparatus according to claim 1, wherein the first camera is a front side camera positioned toward a front side of the camera operator and the second cameral is a rear side camera positioned toward a rear side of the camera operator;

the image pickup mode changeover unit changes over the image pickup mode between;

(i) a double-sided synthesis image pickup mode in which synthesis images of one half circumference picked up by the front side camera and synthesis images of one half circumference picked up by the rear side camera are connected to each other to pick up a synthesis image of a whole circumference, and (ii) a single-sided synthesis image pickup mode in which a synthesis image of a whole circumference is picked up by each of the front side camera and the rear side camera; and

when the image pickup mode is the double-sided synthesis image pickup mode, the synthesis image generation unit combines a plurality of front side component images picked up by the front side camera to generate a front side synthesis image of one half circumference, combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of one half circumference, and then connects the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference to each other to generate a synthesis image of a whole circumference.

6. The image generation apparatus according to claim 5, wherein, when the image pickup mode is the single-sided synthesis image pickup mode, the synthesis image generation unit discards a plurality of front side component images picked up by the front side camera and combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of a whole circumference.

7. The image generation apparatus according to claim 5, wherein, when the image pickup mode is the single-sided synthesis image pickup mode, the synthesis image generation unit combines a plurality of front side component images picked up by the front side camera to generate a front side synthesis image of a whole circumference, combines a plurality of rear side component images picked up by the rear side camera to generate a rear side synthesis image of a whole circumference, and combines and outputs the front side synthesis image of the whole circumference and the rear side synthesis image of the whole circumference into and as a stereo synthesis image or synthesis images of a near view and a distant view.

8. The image generation apparatus according to claim 5, further comprising:

an image correlation decision unit configured to decide a degree of correlation between the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference in a boundary region in which the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference are connected to each other;

when the degree of correlation is equal to or higher than a predetermined threshold value, the synthesis image generation unit connects the front side synthesis image of the one half circumference and the rear side synthesis image of the one half circumference to each other to generate a synthesis image of the whole circumference.

9. The image generation apparatus according to claim 8, further comprising an image pickup guide portion configured to guide the user, when the degree of correlation is lower than the predetermined threshold value, so as to carry out image pickup for further one half circumference.

10. The image generation apparatus according to any claim 5, further comprising:

a camera operator detection portion configured to detect catching of a camera operator in an image picked up by the front side camera;

a camera operator image pickup permission portion configured to issue an inquiry to the user about whether or not the catching of the camera operator is permitted; and

a region-of-interest setting portion configured to set, when the catching of the camera operator is permitted, based on an image of a facial expression of the camera operator picked up by the front side camera, a region-of-interest to an image picked up by the rear side camera;

the component image generation unit generating a component image in which the posture information of the portable equipment and the region-of-interest information set by the region-of-interest setting portion are associated with the image picked up using the rear side camera.

11. An image generation method, comprising:

a component image generation step of generating a plurality of component images from a plurality of sequential images picked up using a plurality of cameras built in portable equipment, a first camera of the plurality of cameras being position toward a first side of a camera operator and a second camera of the plurality of cameras being positioned at least partially toward a second side of the camera operator, each of the plurality of cameras having optical axes different in at least one of a position and an orientation from each other, the component images including posture information of the portable equipment associated with the plurality of sequential images;

a synthesis image generation step of combining the plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover step of changing over an image pickup mode between;

(i) a compound eye synthesis image pickup mode in which synthesis images of a first partial circumference picked up by the first camera of the plurality of cameras are connected to synthesis images of a second partial circumference picked up by the second camera of the plurality of cameras, and (ii) a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation step;

(i) combines a plurality of first component images picked up by the first of the plurality of cameras to generate a first side synthesis image, (ii) combines a plurality of second component images picked up by the second of the plurality of cameras to generate a second side synthesis image, and (iii) connects the first side synthesis image and the second side synthesis image to each other to generate a compound synthesis image.

12. A non-transitory, computer-readable recording medium storing a program, the program causing a computer to implement:

a component image generation function for generating a plurality of component images from a plurality of sequential images picked up using a plurality of cameras built in portable equipment, a first camera of the plurality of cameras being position toward a first side of a camera operator and a second camera of the plurality of cameras being positioned at least partially toward a second side of the camera operator, each of the plurality of cameras having optical axes different in at least one of a position and an orientation from each other, the component images including posture information of the portable equipment associated with the plurality of sequential images;

a synthesis image generation function for combining the plurality of component images in which the posture of the portable equipment is different therebetween to generate a synthesis image; and

an image pickup mode changeover function for changing over an image pickup mode between;

(i) a compound eye synthesis image pickup mode in which synthesis images of a first partial circumference picked up by the first camera of the plurality of cameras are connected to synthesis images of a second partial circumference picked up by the second camera of the plurality of cameras, and (ii) a single eye synthesis image pickup mode in which a single eye synthesis image is picked up using each of the plurality of cameras;

when the image pickup mode is the compound eye synthesis image pickup mode, the synthesis image generation function;

(i) combines a plurality of first component images picked up by the first of the plurality of cameras to generate a first side synthesis image, (ii) combines a plurality of second component images picked up by the second of the plurality of cameras to generate a second side synthesis image, and (iii) connects the first side synthesis image and the second side synthesis image to each other to generate a compound synthesis image.