Pose estimation method and apparatus

• US 7,313,266 B2
• Filed: 03/30/2007
• Issued: 12/25/2007
• Est. Priority Date: 08/10/2001
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Ishiyama, Rui
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): Strege; John B
• Application Number:
  US11/730,314
• Publication Number:
  US 20070172127A1
• Time in Patent Office: 270 Days
• Field of Search: 382/154, 382/289, 345/419, 345/582
• US Class Current: 382/154
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A three-dimensional image data is formulated and saved in a memory for indicating a three-dimensional shape of an object and reflectivity or color at every point of the object. For each of multiple pose candidates, an image space is created for representing brightness values of a set of two-dimensional images of the object which is placed in the same position and orientation as the each pose candidate. The brightness values are those which would be obtained if the object is illuminated under varying lighting conditions. For each pose candidate, an image candidate is detected within the image space using the 3D model data and a distance from the image candidate to an input image is determined. Corresponding to the image candidate whose distance is smallest, one of the pose candidates is selected. The image space is preferably created from each of a set of pose variants of each pose candidate.

Claims

1. A pose estimation method comprising the steps of:

a) measuring reflectivity or color at every point of a reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

b) extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object;

c) creating a plurality of pose candidates from the extracted feature points of said reference object and the extracted feature points of said input image;

d) creating, for each of said pose candidates, an image space of basis images representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose candidate and illuminated under varying lighting conditions by projecting said basis textures onto said image space of said target object;

e) detecting, for each of said pose candidates, an image candidate within said image space of basis images by using said basis textures and determining a distance from the image candidate to said input image of said target object; and

f) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

2. The method of claim 1, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

3. The method of claim 1, wherein step (e) uses the least squares algorithm to detect said image candidate.

4. The method of claim 1, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (c) creates said image space for each of said pose variants.

5. The method of claim 4, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

6. The method of claim 5, wherein said specified range is determined based on a possible error of said each pose candidate.

7. A pose estimation method comprising the steps of:

a) measuring reflectivity or color at every point of a reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

b) extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object;

c) estimating a possible error of the extracted feature points of said input image;

d) creating a plurality of pose candidates from the extracted feature points of said reference object and the extracted feature points of said input image;

e) successively reading one of said pose candidates;

f) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate over a range determined by said possible error estimated by step (c);

g) creating, for each of said pose variants, an image space of basis images representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose variant, and illuminated under varying lighting conditions by projecting said basis textures onto image space of said pose candidates;

h) detecting, for each said pose variant, an image candidate within said image space of basis images by using said basis textures and determining a distance from the image candidate to said input image of said target object;

i) repeating steps (e) to (h) to produce a plurality of said image candidates for each of said pose candidates; and

j) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

8. The method of claim 7, wherein step (h) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

9. The method of claim 7, wherein step (h) uses the least squares algorithm to detect said image candidate.

10. A pose estimation apparatus comprising:

a three-dimensional model creating mechanism for measuring reflectivity or color at every point of a reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

a feature extracting mechanism for extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object;

a pose candidate creating mechanism for creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image;

an image space creating mechanism for creating, for each of said pose candidates, basis images of an image space representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose candidate and illuminated under varying lighting conditions;

an image candidate detecting mechanism for detecting, for each of said pose candidates, an image candidate within said image space by using said basis textures and determining a distance from the image candidate to said input image of said target object; and

an image candidate selecting mechanism for selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

11. The apparatus of claim 10, wherein the image candidate detecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

12. The apparatus of claim 11, wherein the image candidate detecting mechanism uses the least squares algorithm to detect said image candidate.

13. The apparatus of claim 10, further comprising a pose variant creating mechanism for creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein said image space creating mechanism creates said image space for each of said pose variants.

14. The apparatus of claim 13, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

15. The apparatus of claim 14, wherein said specified range is determined based on a possible error of said each pose candidate.

16. A pose estimation apparatus comprising:

a memory;

a 3D model data and measuring reflectivity or color at every point of said reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

a feature points extracting and estimating mechanism for extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object and estimating a possible error of the extracted feature points of said input image;

a pose candidate creating mechanism for creating a plurality of pose candidates from the extracted feature points of said reference object and the extracted feature points of said input image;

a pose variants creating mechanism for successively retrieving a pose candidate from said memory and creating a plurality of pose variants from the retrieved pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from the retrieved pose candidate over a range determined by the estimated possible error;

an image space creating mechanism for creating, for each of said pose variants, an image space of basis images representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose variant, and illuminated under varying lighting conditions by projecting said basis textures onto image space of said pose candidates; and

an image candidate detecting and selecting mechanism for detecting, for each said pose variant, an image candidate within said image space of basis images by using said basis textures, determining a distance from the image candidate to said input image of said target object and selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

17. The apparatus of claim 16, wherein said image candidate detecting and selecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

18. The apparatus of claim 17, wherein said image candidate detecting and selecting mechanism uses the least squares algorithm to detect said image candidate.

19. A computer-readable storage medium containing a computer-executable program that causes a computer to execute the steps of:

a) measuring reflectivity or color at every point of a reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

b) extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object;

c) creating a plurality of pose candidates from the extracted feature points of said reference object and the extracted feature points of said input image;

d) creating, for each of said pose candidates, an image space of basis images representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose candidate and illuminated under varying lighting conditions by projecting said basis textures onto said image space of said target object;

e) detecting, for each of said pose candidates, an image candidate within said image space of basis images by using said basis textures and determining a distance from the image candidate to said input image of said target object; and

f) selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

20. The computer-readable storage medium of claim 19, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

21. The computer-readable storage medium of claim 20, wherein step (e) uses the least squares algorithm to detect said image candidate.

22. The computer-readable storage medium of claim 19, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (c) creates said image space for each of said pose variants.

23. The computer-readable storage medium of claim 22, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

24. The computer-readable storage medium of claim 23, wherein said specified range is determined based on a possible error of said each pose candidate.

25. A computer-readable storage medium containing a computer-executable program that causes a computer to execute the steps of:

a) measuring reflectivity or color at every point of a reference object under varying illuminations to produce a plurality of illumination variation textures, each representing a brightness value of said every point of the reference object, and calculating basis textures approximating said illumination variation textures;

b) extracting feature points from said reference object and extracting feature points from an input image of a target object corresponding to the feature points of the reference object;

c) estimating a possible error of the extracted feature points of said input image;

d) creating a plurality of pose candidates from the extracted feature points of said reference object and the extracted feature points of said input image;

e) successively reading one of said pose candidates;

f) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate over a range determined by said possible error estimated by step (c);

g) creating, for each of said pose variants, an image space of basis images representing brightness values of a plurality of two-dimensional images of said target object which would be obtained if said target object were placed in the same position and orientation as said each pose variant, and illuminated under varying lighting conditions by projecting said basis textures onto image space of said pose candidates;

h) detecting, for each said pose variant, an image candidate within said image space of basis images by using said basis textures and determining a distance from the image candidate to said input image and said target;

i) repeating steps (e) to (h) to produce a plurality of said image candidates for each of said pose candidates; and

j) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

26. The computer-readable storage medium of claim 25, wherein step (h) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

27. The computer-readable storage medium of claim 26, wherein step (h) uses the least squares algorithm to detect said image candidate.