IMAGE PROCESSING DEVICE AND METHOD, AND PROGRAM

• US 20150348243A1
• Filed: 12/12/2013
• Published: 12/03/2015
• Est. Priority Date: 12/26/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: SUTOU, Yasuhiro, SHIMOMURA, Hideki, OKUBO, Atsushi, AOYAMA, Kazumi, TANAKA, Akichika
• Granted Patent: US 9,569,823 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G03B 17/48 adapted for combination with other photographic or optical apparatus
• CPC Class Codes: G03B 2217/005 Blur detection
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/215 : Motion-based segmentation
• CPC Class Codes: G06T 7/231 : using full search
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/633 : for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 : Vibration or motion blur correction
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera

Abstract

Provided is an image processing device including an acquisition unit configured to acquire information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit, a converted image generation unit configured to generate converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image, an evaluation value calculation unit configured to calculate an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images, and a selection unit configured to select a converted moving image with less blur based on an evaluation value calculated by the evaluation value calculation unit.

Claims

1. An image processing device comprising:

an acquisition unit configured to acquire information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;

a converted image generation unit configured to generate a plurality of converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image;

an evaluation value calculation unit configured to calculate an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images; and

a selection unit configured to select a converted moving image with less blur based on an evaluation value calculated by the evaluation value calculation unit.

2. The image processing device according to claim 1, further comprising:

a node difference calculation unit configured to calculate a difference between the converted image generated by the converted image generation unit and the original frame image as a node difference; and

an edge difference calculation unit configured to calculate a difference in a time series including the converted image generated by the converted image generation unit and the original frame image as an edge difference,

wherein the evaluation value calculation unit calculates an evaluation value used to evaluate a blur for each of converted moving images based on each of a sum of the edge difference and a sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combining the converted image and the original image.

3. The image processing device according to claim 2,

wherein the evaluation value calculation unit calculates an evaluation value used to evaluate a blur for each of the converted moving images by applying a weight to each of the sum of the edge difference and the sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combination of the converted image and the original image and by calculating a sum of the weighted values.

4. The image processing device according to claim 3, further comprising:

a weight input unit configured to input a weight to be applied to the sum of the edge difference and the sum of the node difference, a sum of the weight applied to each of the sum of the edge difference and the sum of the node difference being a value of 1,

wherein the evaluation value calculation unit calculates an evaluation value used to evaluate a blur for each of the converted moving images by applying a weight inputted by the weight input unit to each of the sum of the edge difference and the sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combination of the converted image and the original image and by calculating a sum of the weighted values.

5. The image processing device according to claim 1,

wherein the storage unit also stores information on an imaging direction being changed in accordance with the original frame image by having a plurality of predetermined angles of view and by rotating to a plurality of angles in association with the converted image when the converted image is generated by the converted image generation unit, and

wherein the image processing device further includes a display image generation unit configured to generate a display image that displays a route on a map based on the converted moving image or the imaging position information of the frame image of the moving image and displays by applying a different color to each imaging direction on the route based on the imaging direction information.

6. The image processing device according to claim 5, further comprising:

a specifying unit configured to specify a position on the route displayed on the map on the display image; and

a reproduction unit configured to reproduce the converted moving image that constitutes the route or the moving image from a frame image including information on an imaging position corresponding to a position specified by the specifying unit.

7. The image processing device according to claim 1,

wherein the converted image generation unit generates a converted image by converting each frame image that constitutes a moving image stored in the storage unit into a plurality of images having the different imaging directions by having a plurality of predetermined angles of view and by rotating to a plurality of angles by a technique including an angle of view shift, homography transformation, and viewpoint composition, using the frame image itself and preceding and succeeding frame images of the frame image.

8. The image processing device according to claim 1,

wherein the converted image generation unit generates a plurality of higher-resolution converted images having the different imaging directions for each frame image that constitutes a moving image stored in the storage unit by using a pixel generated by a phase shift equal to or smaller than a pixel based on the frame image itself and preceding and succeeding frame images of the frame image.

9. The image processing device according to claim 1, further comprising:

the imaging unit configured to capture the moving image,

wherein the acquisition unit stores the information on the imaging position and the imaging direction of the frame image in association with each of the frame images.

10. An image processing method comprising:

acquiring information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;

generating a plurality of converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image;

calculating an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images; and

selecting a converted moving image with less blur based on a calculated evaluation value.

11. A program for causing a computer to execute processing comprising:

an acquisition step of acquiring information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;

a converted image generation step of generating a plurality of converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image;

an evaluation value calculation step of calculating an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images; and

a selection step of selecting a converted moving image with less blur based on an evaluation value calculated by processing of the evaluation value calculation step.