Image processing apparatus and image processing method

• US 10,567,671 B2
• Filed: 03/06/2015
• Issued: 02/18/2020
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ono, Yuki
• Primary Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US15/124,411
• Publication Number:
  US 20170019608A1
• Time in Patent Office: 1,810 Days
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/75 Unsharp masking
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

The present disclosure relates to an image processing apparatus, an image processing method, and a program therefor with which slow shutter photographing can be performed with ease. An image gradient extraction unit extracts image gradient components in an object movement direction from a long-exposure image out of input images. An initial label map generation unit generates an initial label map based on a gradient extraction result from the image gradient extraction unit. A foreground extraction unit extracts a foreground from the input images based on the label map from the initial label map generation unit or label map update unit to generate a synthesis mask. The present disclosure is applicable to, for example, an image pickup apparatus including an image processing function.

Claims

1. An image processing apparatus, comprising:

a display screen; and

circuitry configured to;

control the display screen to display a frame and a long-exposure image of a plurality of consecutively-photographed images;

determine a movement direction of an object in the long-exposure image based on the displayed frame;

extract image gradient information of the long-exposure image, wherein the image gradient information indicates image gradient components in the movement direction of the object in the long-exposure image;

generate a first synthesis mask based on the extracted image gradient information, wherein the plurality of consecutively-photographed images have different exposure times;

synthesize the plurality of consecutively-photographed images based on the generated first synthesis mask;

generate an output image based on the synthesis of the plurality of the consecutively-photographed images; and

control the display screen to display the output image.

2. The image processing apparatus according to claim 1, wherein the circuitry is further configured to:

generate an initial label map of the first synthesis mask based on a gradient intensity of the long-exposure image, wherein the gradient intensity is in the movement direction of the object;

extract a first foreground based on the initial label map; and

generate the first synthesis mask based on the extraction of the first foreground.

3. The image processing apparatus according to claim 2, wherein the circuitry is further configured to:

update the initial label map based on at least one of a representative color or an update area, wherein each of the representative color and the update area corresponds to the first foreground of the first synthesis mask;

extract a second foreground based on the updated initial label map; and

generate a second synthesis mask based on the extraction of the second foreground.

4. The image processing apparatus according to claim 2, wherein the circuitry is further configured to:

control the display screen to display a UI (User Interface); and

set a specific direction as the movement direction of the object in the long-exposure image based on the display of the UI.

5. The image processing apparatus according to claim 4, wherein the circuitry is further configured to set the specific direction as the movement direction of the object in the long-exposure image based on a user operation in the movement direction of the object in the long-exposure image.

6. The image processing apparatus according to claim 4, wherein the circuitry is further configured to set the specific direction as the movement direction of the object in the long-exposure image based on a user selection from object movement direction candidates in the plurality of consecutively-photographed images.

7. The image processing apparatus according to claim 1, wherein the circuitry is further configured to:

position each of the plurality of consecutively-photographed images; and

generate the first synthesis mask based on the image gradient information of the positioned long-exposure image.

8. An image processing method, comprising:

in an image processing apparatus that includes a display screen and circuitry;

controlling, by the circuitry, the display screen to display a frame and a long-exposure image of a plurality of consecutively-photographed images;

determining, by the circuitry, a movement direction of an object in the long-exposure image based on the displayed frame;

extracting, by the circuitry, image gradient information of the long-exposure image, wherein the image gradient information indicates image gradient components in the movement direction of the object in the long-exposure image;

generating, by the circuitry, a synthesis mask based on the extracted image gradient information, wherein the plurality of consecutively-photographed images have different exposure times;

synthesizing, by the circuitry, the plurality of consecutively-photographed images based on the generated synthesis mask;

generating, by the circuitry, an output image based on the synthesis of the plurality of the consecutively-photographed images; and

controlling, by the circuitry, the display screen to display the output image.

9. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to execute operations, the operations comprising:

controlling a display screen to display a frame and a long-exposure image of a plurality of consecutively-photographed images;

determining a movement direction of an object in the long-exposure image based on the displayed frame;

extracting image gradient information of the long-exposure image, wherein the image gradient information indicates image gradient components in the movement direction of the object in the long-exposure image;

generating a synthesis mask based on the extracted image gradient information, wherein the plurality of consecutively-photographed images have different exposure times;

synthesizing the plurality of consecutively-photographed images based on the generated synthesis mask;

generating an output image based on the synthesis of the plurality of the consecutively-photographed images; and

controlling the display screen to display the output image.