IMAGE PROCESSING DEVICE AND ITS CONTROL METHOD, IMAGING APPARATUS, AND STORAGE MEDIUM

• US 20150138322A1
• Filed: 11/14/2014
• Published: 05/21/2015
• Est. Priority Date: 11/19/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kawamura, Yuta
• Granted Patent: US 10,311,595 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Provided is an image processing device that includes an acquisition unit configured to acquire an image data and a depth data correspond to the image data; a calculating unit configured to calculate a position and attitude change for each depth from the image data and the depth data; and a determining unit configured to determine a position and attitude change of a whole image by a position and attitude change data calculated by the calculating unit based on a statistic of the position and attitude change calculated by the calculating unit.

Claims

1. An image processing device comprising:

an acquisition unit configured to acquire an image data and a depth data correspond to the image data;

a calculating unit configured to calculate a position and attitude change for each depth from the image data and the depth data; and

a determining unit configured to determine a position and attitude change of a whole image by a position and attitude change data calculated by the calculating unit based on a statistic of the position and attitude change calculated by the calculating unit.

2. The image processing device according to claim 1, further comprising:

a grouping unit configured to group the depth data in a plurality of groups based on the position and attitude change data for each depth calculated by the calculating unit;

wherein the determining unit determines the position and attitude change of the whole image by a first position and attitude change data selected from the position and attitude change data, which respectively correspond to the groups obtained by the grouping unit, calculated by the calculating unit.

3. The image processing device according to claim 1, wherein the calculating unit compares a reference frame obtained by segmenting the depth data with a non-reference frame obtained by not segmenting the depth data so as to calculate the position and attitude change for each depth.

4. The image processing device according to claim 1, wherein the determining unit selects a group having a greatest proportion occupied by depth from a plurality of groups obtained by the grouping unit and then determines the position and attitude change data corresponding to the group as the position and attitude change data of the whole image.

5. The image processing device according to claim 4, further comprising:

a storage unit configured to store the position and attitude change data of the whole image determined by the determining unit,

wherein, when a group having the greatest proportion is present in plural, the determining unit reads data stored in the storage unit and then determines position and attitude change data with the smallest difference from the position and attitude change indicated by the read data as the position and attitude change data of the whole image.

6. The image processing device according to claim 4, further comprising:

a selecting unit configured to select any one of the plurality of groups obtained by the grouping unit,

wherein, when a group having the greatest proportion is present in plural, the determining unit determines the position and attitude change data corresponding to the group selected from the plurality of groups by the selecting unit as the position and attitude of the whole image.

7. The image processing device according to claim 4, wherein, when a group having the greatest proportion is present in plural, the determining unit does not determine the position and attitude change of the whole image for the reference frame obtained by segmenting the depth data.

8. An imaging apparatus comprising:

an image acquiring unit configured to acquire image data imaged by an imaging optical system and an imaging element;

a depth image acquiring unit configured to acquire depth data which is paired with the image data; and

an image processing device defined in claim 1.

9. The imaging apparatus according to claim 8, further comprising:

a correcting unit configured to perform image shake correction for the image data; and

a control unit configured to control the correcting unit by acquiring the position and attitude change data of the whole image, which has been determined by the determining unit, and calculating an image shake correction amount.

13. The image processing device according to claim 1, further comprising:

a processing unit configured to calculate a position and attitude change of a region for each motion by detecting the motion of an image from the image data consisting of a plurality of frames so as to calculate a second position and attitude change data from the position and attitude change of the region for each motion; and

a static region determining unit configured to determine the static region of the image by acquiring the first position and attitude change data and the second position and attitude change data,

wherein the determining unit determines the position and attitude change of the whole image based on the static region determined by the static region determining unit, the first position and attitude change data, and the second position and attitude change data.

14. The image processing device according to claim 13, wherein, when the static region determining unit determines a region in which a region occupied by an object image is the greatest region as the static region, the determining unit determines the position and attitude change of the whole image based on the second position and attitude change data.

15. The image processing device according to claim 13, wherein, when the static region determining unit determines a region in which the proportion occupied by depth is the greatest as the static region, the determining unit determines the position and attitude change of the whole image based on the first position and attitude change data.

16. The image processing device according to claim 13, further comprising:

a selecting unit configured to select the judgment criteria of the static region in accordance with an operation,

wherein the static region determining unit determines the static region in accordance with the judgment criteria selected by the selecting unit.

17. The image processing device according to claim 13, wherein the processing unit selects a position and attitude change corresponding to a region of which the proportion to the whole image is the greatest from among the regions for each motion.

18. The image processing device according to claim 13, wherein, when the difference between the first position and attitude change data and the second position and attitude change data is less than a threshold value, the determining unit determines the position and attitude change of the whole image by performing a weighting computation.

19. The image processing device according to claim 13, wherein, when the difference between the first position and attitude change data and the second position and attitude change data exceeds a threshold value, the determining unit reads data stored in the storage unit and then determines either the first position and attitude change data or the second position and attitude change data with a smaller difference from a position and attitude change indicated by the read data as the position and attitude change data of the whole image.

20. The image processing device according to claim 19, wherein, when the first position and attitude change data is different from the second position and attitude change data and no position and attitude change data is stored in the storage unit, the determining unit does not use both the first position and attitude change data and the second position and attitude change data as the position and attitude change data of the whole image.

21. The image processing device according to claim 13, wherein the calculating unit calculates depth data of the same position and attitude change including at least the rearmost depth from the depth data, and the determining unit estimates a first background region candidate from depth data of the same position and attitude change including the rearmost depth.

22. The image processing device according to claim 13, wherein motion of the image is the trajectory of motion vectors or corresponding points or points-of-interest.

23. The image processing device according to claim 21, wherein the processing unit comprises a second estimating unit configured to estimate a second background region candidate for the regions for each motion.

24. The image processing device according to claim 23, wherein the second estimating unit comprises:

a first extracting unit configured to extract a local feature amount from the image data;

a second extracting unit configured to extract a local feature amount from image data of a previously-learned image;

a database creating unit configured to create database from the local feature amount extracted by the second extracting unit; and

a collating unit configured to collate the local feature amount extracted by the first extracting unit with data in the database,

wherein the collating unit estimates the second background region candidate.

25. The image processing device according to claim 23, wherein the static region determining unit determines a background region based on the first background region candidate, the first position and attitude change data, the second background region candidate, and the second position and attitude change data and then determines the static region based on the background region.

26. The image processing device according to claim 25, wherein the determining unit determines the position and attitude change of the whole image based on the static region determined from the background region by the static region determining unit, the first position and attitude change data, and the second position and attitude change data.

27. An imaging apparatus comprising:

a first input unit configured to input image data of an image imaged by an imaging optical system and an imaging element;

a second input unit configured to input depth data which is paired with the image data; and

an image processing device defined in claim 13.

28. The imaging apparatus according to claim 27, further comprising:

a correcting unit configured to perform image shake correction for the image data; and

a control unit configured to control the correcting unit by acquiring the position and attitude change data of the whole image, which has been determined by the determining unit, and calculating an image shake correction amount.

10. A control method to be executed by an image processing device, the method comprising:

acquiring an image data and a depth data correspond to the image data;

calculating a position and attitude change for each depth from the image data and the depth data; and

determining a position and attitude change of a whole image by a position and attitude change data calculated in the calculating based on a statistic of the position and attitude change calculated in the calculating.

11. The control method according to claim 10, further comprising:

grouping the depth data in a plurality of groups based on the position and attitude change data for each depth calculated in the calculating;

wherein, in the determining, a position and attitude change of the whole image is determined by a first position and attitude change data selected from the position and attitude change data, which respectively correspond to the groups obtained in the grouping, calculated in the calculating.

12. A non-transitory storage medium on which is stored a program for causing a computer to execute the control method defined in claim 10.

29. A control method to be executed by an image processing device that calculates a position and attitude change of an object to be processed from image data and depth data, the method comprising:

acquiring, by the image processing device, the image data and depth data;

segmenting the depth data for each depth, and calculating a position and attitude change for each depth from the segmented depth data so as to calculate first position and attitude change data;

calculating a position and attitude change of a region for each motion by detecting the motion of an image from the image data consisting of a plurality of frames so as to calculate second position and attitude change data from a position and attitude change of the region for each motion;

determining the static region of the image by acquiring the first position and attitude change data and the second position and attitude change data; and

determining a position and attitude change of the whole image based on the determined static region, the first position and attitude change data, and the second position and attitude change data.

30. The control method according to claim 29, further comprising:

performing image shake correction using the data of the determined position and attitude change of the whole image.

31. A non-transitory storage medium on which is stored a program for causing a computer to execute the control method defined in claim 29.