IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, PROGRAM, AND INFORMATION RECORDING MEDIUM

• US 20150302587A1
• Filed: 05/30/2013
• Published: 10/22/2015
• Est. Priority Date: 09/26/2012
• Status: Active Grant

• Current Assignee: Rakuten Group, Inc.
• Original Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Inventors: Hirano, Hiromi, Bandara, Udana
• Granted Patent: US 10,452,953 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A plurality of captured images obtained by repeatedly shooting a rotating object are obtained. In a case where a feature point corresponding to a certain point on the object is specified in one captured image and another captured image, a position information obtaining unit obtains position information relating to a position of the feature point in the one captured image and the another captured image. A rotation angle information obtaining unit obtains rotation angle information relating to a rotation angle of the object between a point of time when the one captured image is shot and a point of time when the another captured image is shot based on position information. A moving image data generating unit generates moving image data by selecting frame images of the moving image data from the plurality of photographed images based on rotation angle information of each of the plurality of photographed images.

Claims

1. An image processing device comprising:

a captured image obtaining unit that obtains a plurality of captured images that are obtained by repeatedly shooting an object while the object rotates;

a feature point specifying unit that specifies one or more feature points in the captured images,

a position information obtaining unit that obtains, in a case where a feature point corresponding to a certain point on the object is specified in a first captured image and a second captured image that is shot after the first captured image, position information relating to positions of the feature point in the first captured image and the second captured image;

a rotation angle information obtaining unit that obtains rotation angle information relating to a rotation angle of the object between a point of time when the first captured image is shot and a point of time when the second captured image is shot, based on the position information; and

a moving image data generating unit that generates moving image data which shows how the object rotates, by selecting a plurality of frame images of the moving image data from among the plurality of captured images, based on the rotation angle information obtained by the rotation angle information obtaining unit for each of the plurality of captured images.

5. The image processing device according to claim 1,

wherein, in a case where a feature point corresponding to a certain point on the object is specified on a first pair of captured images, the position information obtaining unit obtains position information relating to a position of the feature point in a first captured image that is shot first among the first pair of captured images and in a second captured image that is shot later among the first pair of captured images,

wherein the rotation angle information obtaining unit obtains rotation angle information relating to a rotation angle of the object between a point of time when the first captured image of the first pair of captured images is shot and a point of time when the second captured image of the first pair of captured images is shot based on the position information of the first pair of captured images,

wherein, in a case where a feature point corresponding to a certain point on the object is specified in a second pair of captured images, the position information obtaining unit obtains position information relating to a position of the feature point of a first captured image that is shot first among the second pair of captured images and a second captured image that is shot later among the second pair of captured images,

wherein the rotation angle information obtaining unit obtains rotation angle information relating to a rotation angle of the object between a point of time when the first captured image of the second pair of captured images is shot and a point of time when the second captured image of the second pair of captured images is shot based on the position information of the second pair of captured images,

wherein, in a case where the second captured image of the first pair of captured images is identical to the first captured image of the second pair of captured images, the moving image data generating unit includes a unit that obtains cumulative rotation angle information relating to a rotation angle of the object between the point of time when the first captured image of the first pair of captured images is shot and the point of time when the second captured image of the second pair of captured images is shot, based on the rotation angle information of the first pair of captured images and the rotation angle information of the second pair of captured images, and

wherein the moving image data generating unit selects a plurality of frame images of the moving image data from the plurality of captured images based on the cumulative rotation angle information.

6. The image processing device according to claim 5,

wherein, in a case where the first captured image of the first pair of captured images is selected as a frame image of the moving image data, the moving image data generating unit determines whether or not to select the second captured image of the second pair of captured images as another frame image of the moving image data, based on a result of comparing the cumulative rotation angle information with the threshold value.

7. The image processing device according to claim 6,

wherein, in a case where the first captured image of the first pair of captured images is selected as a frame image of the moving image data and the cumulative rotation angle information is equal to or more than the threshold value, the moving image data generating unit selects either one of the second captured image of the second pair of captured images and a third captured image that was shot immediately before the second captured image of the second pair of captured images as another frame image of the moving image data.

8. The image processing device according to claim 1,

wherein the rotation angle information obtaining unit comprises:

a unit that obtains position difference information relating to a difference between the position of the feature point in the first captured image and the position of the feature point in the second captured image; and

a unit that obtains the rotation angle information based on the position difference information.

9. The image processing device according to claim 8 comprising:

a unit that sets an area in the captured image as an observation area based on a position of a rotation axis of the object in the captured image,

wherein the rotation angle information obtaining unit obtains the rotation angle information based on the position difference information in a case where the feature point is included in the observation area.

10. The image processing device according to claim 8,

wherein the rotation angle information obtaining unit obtains the rotation angle information using the position difference information of a feature point whose rotation radius of when the object rotates is large, in preference to the position difference information of a feature point whose rotation radius of when the object rotates is small.

11. The image processing device according to claim 8,

wherein the rotation angle information obtaining unit comprises:

a rotation radius information obtaining unit that obtains, in a case where a plurality of feature points respectively corresponding to a plurality of points on the object are specified in the first captured image and the second captured image, rotation radius information relating to a rotation radius of a point on the object corresponding to the feature point, for each of the plurality of feature points;

a unit that obtains, for each of the plurality of feature points, rotation angle information relating to a rotation angle of the point corresponding to the feature point on the object between a point of time when the first captured image is shot and a point of time when the second captured image is shot, based on the position difference information and the rotation radius information of the feature points; and

a unit that obtains the rotation angle information of the object based on the rotation angle information of each of the plurality of feature points.

12. The image processing device according to claim 11,

wherein the rotation radius information obtaining unit comprises:

a unit that obtains, in a case where the feature point corresponding to the certain point on the object is specified in at least three captured images, elliptical orbit information relating to an elliptical orbit of the feature point based on the position of the feature point in the at least three captured images; and

a unit that obtains the rotation radius information of the feature point based on the elliptical orbit information of the feature point.

13. The image processing device according to claim 11,

wherein, in a case where the feature point corresponding to the certain point on the object is first specified in an ith (i:

an integer of 1 or more) captured image and specified last in a jth (j;

an integer greater than i) captured image that is shot later than the ith captured image, the rotation radius information obtaining unit obtains the rotation radius information of the feature point based on a position of the feature point in the ith captured image and a position of the feature point in the jth captured image.

14. The image processing device according to claim 1,

wherein the rotation angle information obtaining unit comprises:

a unit that obtains, in a case where the feature point corresponding to the certain point on the object is specified in at least three captured images, elliptical orbit information relating to an elliptical orbit of the feature point; and

a unit that obtains the rotation angle information based on the positions of the feature point in the first captured image and in the second captured image and the elliptical orbit information.

15. The image processing device according to claim 1,

wherein the moving image data generating unit comprises:

a unit that obtains similarity information relating to a similarity between a reference captured image that is selected from the plurality of captured images and each of a plurality of captured images that are shot after a predetermined period of time passes from a point of time when the reference captured image is shot; and

a unit that selects one of the plurality of captured images that are shot after the predetermined period of time passes from the point of time when the reference captured image is shot as a captured image of a point of time when the object completes one rotation based on the similarity information, and

wherein the moving image data generating unit generates the moving image data showing how the object rotates once based on captured images that are shot before the point of time when the captured image selected as the captured image of the time when the object completes one rotation is shot.

16. The image processing device according to claim 15,

wherein the moving image data generating unit comprises:

a feature point number obtaining unit that obtains a number of feature points corresponding to feature points specified in the reference captured image, from among the feature points specified in the captured image that is shot after the predetermined period of time passes from the point of time when the reference captured image is shot;

a unit that selects candidates of the captured image of the point of time when the object completes one rotation, from among the plurality of captured images that are shot after the predetermined period of time passes from the point of time when the reference captured image is shot, based on a result obtained by the feature point number obtaining unit;

a unit that obtains similarity information relating to a similarity between the reference captured image and each of the captured images selected as the candidates; and

a unit that selects one of the captured images selected as the candidates as the captured image of a point of time when the object completes one rotation based on the similarity information.

17. The image processing device according to claim 1,

wherein the plurality of captured images include the 1st to mth (m:

an integer of 2 or more) captured images and m+1th to m+nth (n;

an integer of 1 or more) captured images, which are shot after the mth captured image,

wherein the moving image data generating unit comprises;

a selection number obtaining unit that obtains a number of the captured images selected as frame images of the moving image data out of the 1st to the mth captured images; and

a unit that selects the frame images of the moving image data out of the m+1th to the m+nth captured images, in a case where the feature point corresponding to the point on the object is not specified in the m+1th to the m+nth captured images, based on a ratio of a number of the 1st to the mth captured images to the number of the captured images obtained by the selection number obtaining unit.

18. The image processing device according to claim 1,

wherein the plurality of captured images include the 1st to mth (m:

an integer of 2 or more) captured images and m+1th to m+nth (n;

an integer of 1 or more) captured images, which are shot after the mth captured image,

wherein the moving image data generating unit comprises;

a selection number obtaining unit that obtains a number of the captured images selected out of the m+1th to the m+nth captured images as the frame images of the moving image data; and

a unit that selects the frame images of the moving image data out of the 1st to the mth captured images, in a case where the feature point corresponding to the point on the object is not specified in the 1st to the mth captured images, based on a ratio of a number of the m+1th to the m+nth captured images to the number of the captured images obtained by the selection number obtaining unit.

19. The image processing device according to claim 1, comprising:

a first determination unit that determines whether or not a feature amount of the feature point specified in the first captured image and a feature amount of the feature point specified in the second captured image are the same or substantially the same;

a second determination unit that determines whether or not a difference of a coordinate value of a direction of an axis corresponding to a rotation axis of the object between the position of the feature point specified in the first captured image and the position of the feature point specified in the second captured image is equal to or more than the threshold value; and

a unit that determines whether or not the feature point specified in the first captured image and the feature point specified in the second captured image are feature points corresponding to the certain point on the object based on a determination result of the first determination unit and the second determination unit.

20. The image processing device according to claim 1, comprising:

a determination unit that determines whether or not the feature amount of the feature point specified in the first captured image and the feature amount of the feature point specified in the second captured image are the same or substantially the same;

a comparing unit that compares (a) information relating to a difference of a coordinate value of a direction of a second axis perpendicular to a first axis corresponding to a rotation axis of the object between the position of a first feature point in the first captured image and the position of the first feature point in the second captured image with (b) information relating to a difference of a coordinate value of the direction of the second axis between the position of a second feature point in the first captured image and the position of the second feature point in the second captured image, in a case where the first feature point having the same or substantially the same feature amount in both of the first captured image and the second captured image, and the second feature point having the same or substantially the same feature amount in both of the first captured image and the second captured image are specified; and

a unit that determined whether or not the feature point specified in the first captured image and the feature point specified in the second captured image are feature points corresponding to the certain point on the object, based on a determination result of the determination unit and a comparing result of the comparing unit.

21. The image processing device according to claim 1,

wherein the captured image obtaining unit obtains a plurality of captured images that are obtained by repeatedly shooting the object while the object rotates a plurality of times, and

wherein the moving image data generating unit generates the moving image data that shows how the object rotates once based on the plurality of captured images that are obtained by repeatedly shooting the object while the object rotates the plurality of times.

2-4. -4. (canceled)

22. An image processing method comprising:

obtaining a plurality of captured images that are obtained by repeatedly shooting an object while the object rotates;

specifying one or more feature points in the captured images;

obtaining, in a case where a feature point corresponding to a certain point on the object is specified in a first captured image and a second captured image that is shot after the first captured image, position information relating to positions of the feature point in the first captured image and the second captured image;

obtaining rotation angle information relating to a rotation angle of the object between a point of time when the first captured image is shot and a point of time when the second captured image is shot, based on the position information; and

generating moving image data by selecting a plurality of frame images of the moving image data, which shows how the object rotates, from among the plurality of captured images, based on the rotation angle information obtained for each of the plurality of captured images.

23. (canceled)

24. A non-transitory computer-readable information storage medium that stores a program for causing a computer to:

obtain a plurality of captured images that are obtained by repeatedly shooting an object while the object rotates;

specify one or more feature points in the captured images,

obtain, in a case where a feature point corresponding to a certain point on the object is specified in a first captured image and a second captured image that is shot after the first captured image, position information relating to positions of the feature point in the first captured image and the second captured image;

obtain rotation angle information relating to a rotation angle of the object between a point of time when the first captured image is shot and a point of time when the second captured image is shot, based on the position information; and

generate moving image data by selecting a plurality of frame images of the moving image data, which shows how the object rotates, from among the plurality of captured images, based on the rotation angle information obtained for each of the plurality of captured images.