Image processing device and associated methodology for generating panoramic images

• US 10,237,474 B2
• Filed: 03/24/2016
• Issued: 03/19/2019
• Est. Priority Date: 08/02/2011
• Status: Active Grant

• Current Assignee: Sony Group Corp.
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sento, Kazuya
• Primary Examiner(s): Lee, Michael
• Application Number:
  US15/079,578
• Publication Number:
  US 20160205320A1
• Time in Patent Office: 1,090 Days
• Field of Search: 348 36
• CPC Class Codes: G03B 17/12 with means for supporting objectives, supplementary lenses, filters, masks, or turrets
• CPC Class Codes: G03B 17/20 visible in viewfinder
• CPC Class Codes: G03B 37/02 with scanning movement of lens or cameras
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 25/46 by combining or binning pixels

Abstract

One embodiment of an apparatus includes a reference position receiving unit configured to receive intermediate or end panorama reference position information input by a user, and a control unit configured to control an imaging device to begin generating a plurality of images to be used to generate a panoramic image based on the intermediate or end panorama reference position information input by the user after the reference position receiving unit receives the intermediate or end panorama reference position information.

Claims

1. An image capturing apparatus comprising:

a display; and

circuitry configured to

automatically determine an image capturing motion direction, based on a movement of the image capturing apparatus in one of a first direction and a second direction different from the first direction, the image capturing motion direction being a direction in which images are to be captured to form a synthesized image,

begin and continue capture of a plurality of images, at least part of which are used to generate the synthesized image, responsive to the image capturing apparatus being moved in the one of the first direction and the second direction determined to be the image capturing motion direction,

display, on the display throughout the capture of the plurality of images, first operation support information indicating the image capturing motion direction,

display, on the display, second operation support information in a case where the image capturing apparatus is moved different from the one of the first direction and the second direction determined to be the image capturing motion direction after the image capturing motion direction is determined, the second support information indicating a motion direction to cause the image capturing apparatus to move in one of the first direction and the second direction,

generate the synthesized image using at least a part of the plurality of images captured by the image capturing apparatus, the at least part of the plurality of images being combined together according to the image capturing motion direction determined in order to generate the synthesized image, and

display the synthesized image on the display.

2. The image capturing apparatus according to claim 1, wherein the circuitry is further configured to determine the first direction as the image capturing motion direction in a case that an amount of the movement of the image capturing apparatus in the first direction is greater than an amount of the movement in the second direction.

3. The image capturing apparatus according to claim 1, wherein the circuitry is further configured to display, on the display, the first operation support information during the capture of the plurality of images.

4. The image capturing apparatus according to claim 1, wherein the first direction is a horizontal direction and the second direction is a vertical direction.

5. The image capturing apparatus according to claim 1, wherein the image capturing motion direction is a swing direction.

16. The image capturing apparatus according to claim 1, wherein to determine the image capturing motion direction, the circuitry uses motion vectors corresponding to objects in the plurality of images.

17. The image capturing apparatus according to claim 16, wherein motion vectors that differ from an overall motion vector of the plurality of images are exempted from the determination of the image capturing motion direction.

18. The image capturing apparatus according to 1, wherein the circuitry starts the capture of the plurality of images as the image capturing apparatus reaches a predetermined position while being moved in the one of the first direction and the second direction determined to be the image capturing motion direction.

19. The image capturing apparatus according to claim 18, wherein the predetermined position is one of a left-most position or a right-most position of the synthesized image.

20. The image capturing apparatus according to claim 18, wherein the predetermined position is one of an upper-most position or a lower-most position of the synthesized image.

6. An image capturing method for an image capturing apparatus, comprising:

automatically determining, with circuitry, an image capturing motion direction, based on a movement of the image capturing apparatus in one of a first direction and a second direction different from the first direction, the image capturing motion direction being a direction in which images are to be captured to form a synthesized image;

beginning and continuing to capture, with the circuitry, a plurality of images, at least part of which are used to generate the synthesized image, responsive to the image capturing apparatus being moved in the one of the first direction and the second direction determined to be the image capturing motion direction;

displaying, with the circuitry and on a display throughout the capture of the plurality of images, first operation support information indicating the image capturing motion direction;

displaying, with the circuitry and on the display, second operation support information in a case where the image capturing apparatus is moved different from the one of the first direction and the second direction determined to be the image capturing motion direction after the image capturing motion direction is determined, the second support information indicating a motion direction to cause the image capturing apparatus to move in one of the first direction and the second direction;

generating, with the circuitry, the synthesized image using at least a part of the plurality of images captured by the image capturing apparatus, the at least part of the plurality of images being combined together according to the image capturing motion direction determined in order to generate the synthesized image; and

displaying, with the circuitry, the synthesized image on the display.

7. The image capturing method according to claim 6, further comprising determining the first direction as the image capturing motion direction in a case that an amount of the movement of the image capturing apparatus in the first direction is greater than an amount of the movement in the second direction.

8. The image capturing method according to claim 6, wherein the first operation support information is displayed, on the display, during the capture of the plurality of images.

9. The image capturing method according to claim 6, wherein the first direction is a horizontal direction and the second direction is a vertical direction.

10. The image capturing method according to claim 6, wherein the image capturing motion direction is a swing direction.

11. A non-transitory computer-readable medium encoded with computer-readable instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:

automatically determining an image capturing motion direction, based on a movement of an image capturing apparatus in one of a first direction and a second direction different from the first direction, the image capturing motion direction being a direction in which images are to be captured to form a synthesized image;

begin and continue to capture a plurality of images, at least part of which are used to generate the synthesized image, responsive to the image capturing apparatus being moved in the one of the first direction and the second direction determined to be the image capturing motion direction;

displaying, on a display throughout the capture of the plurality of images, first operation support information indicating the image capturing motion direction,

displaying, on the display, second operation support information in a case where the image capturing apparatus is moved different from the one of the first direction and the second direction determined to be the image capturing motion direction after the image capturing motion direction is determined, the second support information indicating a motion direction to cause the image capturing apparatus to move in one of the first direction and the second direction;

generating the synthesized image using at least a part of the plurality of images captured by the image capturing apparatus, the at least part of the plurality of images being combined together according to the image capturing motion direction determined in order to generate the synthesized image; and

displaying the synthesized image on the display.

12. The non-transitory computer-readable medium according to claim 11, further comprising determining the first direction as the image capturing motion direction in a case that an amount of the movement of the image capturing apparatus in the first direction is greater than an amount of the movement in the second direction.

13. The non-transitory computer-readable medium according to claim 11, wherein the first operation support information is displayed on the display, during the capture of the plurality of images.

14. The non-transitory computer-readable medium according to claim 11, wherein the first direction is a horizontal direction and the second direction is a vertical direction.

15. The non-transitory computer-readable medium according to claim 11, wherein the image capturing motion direction is a swing direction.