Information processing method and information processing apparatus

• US 9,429,418 B2
• Filed: 02/22/2011
• Issued: 08/30/2016
• Est. Priority Date: 02/25/2010
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Mitarai, Yusuke, Matsugu, Masakazu, Yoshii, Hiroto
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Kholdebarin, Iman K
• Application Number:
  US13/580,560
• Publication Number:
  US 20120321173A1
• Time in Patent Office: 2,016 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/03 by measuring coordinates of points
• CPC Class Codes: G01B 11/26 for measuring angles or tapers; for testing the alignment of axes
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A multi view-point image composed of a great number of images according to a shape of an object is generated or an information processing method used for generating a three-dimensional model or performing image processing of arbitrary view-point object recognition is provided, and based on a plurality of captured images obtained by imaging of the object from a plurality of view points by an imaging means, a relative position and orientation with respect to the object relative to the imaging means for each of the plurality of view points is calculated, and based on the calculated plurality of relative positions and orientations, a missing position and orientation of the imaging means in a direction in which imaging by the imaging means is missing is calculated, and an image used for displaying the calculated missing position and orientation on a display means is generated.

Claims

1. An information processing apparatus comprising:

a memory; and

a processor coupled to the memory which executes the following;

obtaining a plurality of relative positions and orientations with respect to an object relative to an imaging unit for each of a plurality of view points based on a plurality of images captured by imaging the object from the plurality of view points by the imaging unit;

deriving, as a lacking relative position and orientation, a relative position and orientation of the imaging unit from which imaging by the imaging unit is lacking with respect to the object based on the obtained plurality of relative positions and orientations; and

outputting information which indicates the derived lacking relative position and orientation,

wherein the deriving includes;

setting a plurality of virtual relative positions and orientations from which the object is imaged approximately uniformly with the object at a center;

determining whether each of the virtual relative positions and orientations is similar to at least one of the obtained relative positions and orientations; and

deriving the lacking relative position and orientation based on the virtual relative positions and orientations to each of which it is not determined that any of the obtained relative positions and orientations is similar.

2. The information processing apparatus according to claim 1, wherein the determining is performed based on an inner product of a vector connecting the virtual positions and orientations with the object and a vector connecting the relative positions and orientations with the object.

3. The information processing apparatus according to claim 1, wherein the captured image includes a plurality of markers arranged on the object and in a periphery of the object, and three-dimensional positions of the plurality of markers in a real space are known.

4. The information processing apparatus according to claim 3, each of a plurality of markers has a color different from a color of the object and the plurality of markers which are different from a marker.

5. The information processing apparatus according to claim 3, wherein obtaining the relative position and orientation with respect to the object relative to the imaging unit is performed by detecting positions of the plurality of markers in the captured image.

6. The information processing apparatus according to claim 3, positions of the detected plurality of markers are four or more.

7. The information processing apparatus according to claim 1, wherein obtaining the relative position and orientation with respect to the object relative to the imaging unit for each of the plurality of positions and orientations is performed by comparing characteristics of the three-dimensional model of the object stored in advance and characteristics of the object in the captured image.

8. The information processing apparatus according to claim 1, wherein the deriving further includes deriving, if the obtained relative position and orientation from a position in a vicinity of a relative position and orientation which can be used in imaging a complex structure of a three-dimensional model of the object stored in advance is small in number, a relative position and orientation in the vicinity of the relative position and orientation which can be used in imaging a complex structure as a lacking relative position and orientation.

9. The information processing apparatus according to claim 1, the apparatus further comprises an imaging moving unit used for moving the imaging unit.

10. The information processing apparatus according to claim 1, further comprising:

a display unit to display the generated image.

11. The information processing apparatus according to claim 1, wherein in the outputting, an image indicating the lacking relative position and orientation is generated and the generated image is output to a display unit.

12. An information processing apparatus comprising:

a memory;

a processor coupled to the memory which executes the following;

obtaining a plurality of images, of an object, captured by an imaging unit and a relative position and orientation of the imaging unit, with respect to the object, for each of the plurality of captured images;

generating a classifier for estimating an orientation of the object in an image based on the obtained plurality of images and the obtained relative positions and orientations of the imaging unit;

generating an image capable of being observed in a case where observation of the object is performed in a first relative position and orientation which is different from any of the obtained relative positions and orientations;

deriving, in a case where classification accuracy of when the classifier is applied to the generated image is lower than a threshold value, the first relative position and orientation as a lacking relative position and orientation; and

outputting information indicating the derived lacking relative position and orientation.

13. The information processing apparatus according to claim 12, further comprises normalizing the plurality of captured images in a unified format, and the classifier allows learning of the plurality of captured images which have been normalized.

14. The information processing apparatus according to claim 12, further comprising deriving the lacking relative position and orientation by comparing the relative position and orientation and an orientation of the object in the generated image, which is estimated based on the classifier.

15. The information processing apparatus according to claim 12, wherein the first relative position and orientation is an intermediate position and orientation of the obtained relative positions and orientations used for imaging the plurality of captured images learned by the classifier.

16. The information processing apparatus according to claim 12, wherein the plurality of captured images learned by the classifier are two captured images.

17. The information processing apparatus according to claim 12, further comprising generating an image indicating the lacking relative position and orientation and output the generated image to a display unit.

18. An information processing apparatus comprising:

a memory;

a processor coupled to the memory which executes the following;

obtaining a plurality of images, of an object, captured by an imaging unit and a relative position and orientation of the imaging unit, with respect to the object, for each of the plurality of captured images;

generating an approximately three-dimensional model based on the obtained plurality of images and the obtained relative positions and orientations of the imaging unit;

generating, based on the obtained image, an image capable of being observed in a case where observation is performed in a second relative position and orientation which is different from the obtained relative position and orientation;

performing rendering to generate an image, which is to be observed from the second relative position and orientation, based on the approximately three-dimensional model;

deriving, in a case where a degree of similarity between the generated image and the image generated by rendering is lower than a threshold value, the second relative position and orientation as a lacking relative position and orientation; and

outputting information indicating the derived lacking relative position and orientation.

19. The information processing apparatus according to claim 18, further comprising generating an image indicating the lacking relative position and orientation and output the generated image to a display unit.

20. An information processing method comprising:

obtaining a plurality of relative positions and orientations with respect to an object relative to an imaging unit for each of a plurality of view points based on a plurality of images captured by imaging the object from the plurality of view points by an imaging unit;

deriving, as a lacking relative position and orientation, a relative position and orientation of the imaging unit from which imaging by the imaging unit is lacking with respect to the object based on the obtained plurality of relative positions and orientations; and

outputting information which indicates the derived lacking relative position and orientation,

wherein the deriving includes;

setting a plurality of virtual relative positions and orientations from which the object is imaged approximately uniformly with the object at a center;

determining whether each of the virtual relative positions and orientations is similar to at least one of the obtained relative positions and orientations; and

deriving the lacking relative position and orientation based on the virtual relative positions and orientations to each of which it is not determined that any of the obtained relative positions and orientations is similar.

21. A non-transitory storage medium storing a computer-executable program configured to allow a computer to execute the information processing method according to claim 20.

22. An information processing method comprising:

obtaining a plurality of images, of an object, captured by an imaging unit and a relative position and orientation of the imaging unit, with respect to the object, for each of the plurality of captured images;

generating a classifier for estimating an orientation of the object in an image based on the obtained plurality of images and the obtained relative positions and orientations of the imaging unit;

generating an image capable of being observed in a case where observation of the object is performed in a first relative position and orientation which is different from any of the obtained relative positions and orientations;

deriving, in a case where classification accuracy of when the classifier is applied to the generated image is lower than a threshold value, the first relative position and orientation as a lacking relative position and orientation; and

outputting information indicating the derived lacking relative position and orientation.

23. A non-transitory storage medium storing a computer-executable program configured to allow a computer to execute the information processing method according to claim 22.

24. An information processing method comprising:

obtaining a plurality of images, of an object, captured by an imaging unit and a relative position and orientation of the imaging unit, with respect to the object, for each of the plurality of captured images;

generating an approximately three-dimensional model based on the obtained plurality of images and the obtained relative positions and orientations of the imaging unit;

generating, based on the obtained image, an image capable of being observed in a case where observation is performed in a second relative position and orientation which is different from the obtained relative position and orientation;

performing rendering to generate an image, which is to be observed from the second relative position and orientation, based on the approximately three-dimensional model;

deriving, in a case where a degree of similarity between the generated image and the image generated by rendering is lower than a threshold value, the second relative position and orientation as a lacking relative position and orientation; and

outputting information indicating the derived lacking relative position and orientation.

25. A non-transitory storage medium storing a computer-executable program configured to allow a computer to execute the information processing method according to claim 24.