Pose estimation method and apparatus

• US 20030035098A1
• Filed: 08/02/2002
• Published: 02/20/2003
• Est. Priority Date: 08/10/2001
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Ishiyama, Rui
• Granted Patent: US 7,218,773 B2
• Time in Patent Office: Days
• US Class Current: 356/72
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A three-dimensional image data is formulated and saved in a memory for indicating a three-dimensional shape of an object and reflectivity or color at every point of the object. For each of multiple pose candidates, an image space is created for representing brightness values of a set of two-dimensional images of the object which is placed in the same position and orientation as the each pose candidate. The brightness values are those which would be obtained if the object is illuminated under varying lighting conditions. For each pose candidate, an image candidate is detected within the image space using the 3D model data and a distance from the image candidate to an input image is determined. Corresponding to the image candidate whose distance is smallest, one of the pose candidates is selected. The image space is preferably created from each of a set of pose variants of each pose candidate.

Claims

1. A pose estimation method by using a plurality of pose candidates, comprising the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) creating, for each of said pose candidates, an image space representing brightness values of a plurality of two-dimensional images of said object which is placed in the same position and orientation as said each pose candidate, wherein said brightness values would be obtained if said object is illuminated under varying lighting conditions;

c) detecting, for each of said pose candidates, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to an input image; and

d) selecting one of the pose candidates which corresponds to the image candidate whose distance to an input image is smallest.

2. The method of claim 1, wherein step (a) comprises the steps of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the model data indicates said basis textures as well as said three-dimensional shape.

3. The method of claim 1, wherein step (c) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

4. The method of claim 3, wherein step (c) uses the least squares algorithm to detect said image candidate.

5. The method of claim 1, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (b) creates said image space for each of said pose variants.

6. The method of claim 5, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

7. The method of claim 6, wherein said specified range is determined based on a possible error of said each pose candidate.

8. A pose estimation method by using a plurality of pose candidates stored in a memory, comprising the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) successively reading one of said pose candidates from said memory;

c) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate;

d) creating, for each of said pose variants, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose variant, wherein said brightness values would be obtained if said object is illuminated under varying lighting conditions;

e) detecting, for each said pose variant, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to an input image;

f) repeating steps (b) to (e) to produce a plurality of said image candidates for each of said pose candidates;

g) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest;

h) comparing the pose candidate selected by step (g) with a previously selected pose candidate; and

i) replacing the previously selected pose candidate with the pose candidate currently selected by step (g) if the currently selected pose candidate is better than the previously selected pose candidate, and repeating steps (b) to (g) until the previously selected pose candidate is better than the currently selected pose candidate.

9. The method of claim 8, wherein step (a) comprises the steps of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

10. The method of claim 8, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

11. The method of claim 8, wherein step (e) uses the least squares algorithm to detect said image candidate.

12. A pose estimation method comprising the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) extracting feature points from said object and extracting feature points from an input image;

c) creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in a memory;

d) creating, for each of said pose candidates, an image space representing brightness values of a plurality of two-dimensional images of an object placed in the same position and orientation as said each pose candidate, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions;

e) detecting, for each of said pose candidates, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to said input image; and

f) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

13. The method of claim 12, wherein step (a) comprises the steps of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

14. The method of claim 12, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

15. The method of claim 12, wherein step (e) uses the least squares algorithm to detect said image candidate.

16. The method of claim 12, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (c) creates said image space for each of said pose variants.

17. The method of claim 16, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

18. The method of claim 17, wherein said specified range is determined based on a possible error of said each pose candidate.

19. A pose estimation method comprising the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) extracting feature points from said object and extracting feature points from an input image;

c) estimating a possible error of the extracted feature points of said input image;

d) creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in a memory;

e) successively reading one of said pose candidates from said memory;

f) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate over a range determined by said possible error estimated by step (c);

g) creating, for each of said pose variants, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose variant, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions;

h) detecting, for each said pose variant an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to said input image;

i) repeating steps (e) to (h) to produce a plurality of said image candidates for each of said pose candidates; and

j) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

20. The method of claim 19, wherein step (a) comprises the steps of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

21. The method of claim 19, wherein step (h) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

22. The method of claim 20, wherein step (h) uses the least squares algorithm to detect said image candidate.

23. A pose estimation apparatus comprising:

a memory for storing a plurality of pose candidates;

a three-dimensional model creating mechanism for formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

an image space creating mechanism for successively retrieving a pose candidate from said memory and creating an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as the retrieved pose candidate, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions;

an image candidate detecting mechanism for detecting an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to an input image; and

a selecting mechanism for selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

24. The apparatus of claim 23, wherein the three-dimensional model creating mechanism performs the functions of measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object, calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

25. The apparatus of claim 23, wherein the image candidate detecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

26. The apparatus of claim 25, wherein the image candidate detecting mechanism uses the least squares algorithm to detect said image candidate.

27. The apparatus of claim 23, further comprising a pose variant creating mechanism for creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein said image space creating mechanism creates said image space for each of said pose variants.

28. The apparatus of claim 27, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

29. The apparatus of claim 28, wherein said specified range is determined based on a possible error of said each pose candidate.

30. A pose estimation apparatus comprising:

a first memory for storing a plurality of pose candidates;

a 3D model data formulating mechanism for formulating 3D model data for indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

a pose variants creating mechanism for successively reading a pose candidate from said first memory and creating a plurality of pose variants of the retrieved pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from the retrieved pose candidate, and storing the pose variants in a second memory;

an image space creating mechanism for successively reading a pose variant from said second memory and creating an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as the retrieved pose variant, wherein said brightness values would be obtained if said object is illuminated under varying lighting conditions;

an image candidate detecting and selecting mechanism for successively detecting, in correspondence to the retrieved pose variant, an image candidate within said image space by using said 3D model data, determining a distance from the image candidate to an input image, and selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest; and

a pose candidate comparing and replacing mechanism for comparing the selected pose candidate with a previously selected pose candidate, and replacing the previously selected pose candidate with the currently selected pose candidate if the currently selected pose candidate is better than the previously selected pose candidate until the previously selected pose candidate is better than the currently selected pose candidate.

31. The apparatus of claim 30, wherein the three-dimensional model creating mechanism performs the functions of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

32. The apparatus of claim 30, wherein the image candidate detecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

33. The apparatus of claim 32, wherein the image candidate detecting mechanism uses the least squares algorithm to detect said image candidate.

34. A pose estimation apparatus comprising:

a three-dimensional model creating mechanism for formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

a feature extracting mechanism for extracting feature points from said object and extracting feature points from an input image;

a pose candidate creating mechanism for creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in a memory;

a image space creating mechanism for creating, for each of said pose candidates, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose candidate, wherein said brightness values would be obtained if said object is illuminated under varying lighting conditions;

an image candidate detecting mechanism for detecting, for each of said pose candidates, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to said input image; and

an image candidate selecting mechanism for selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

35. The apparatus of claim 34, wherein the three-dimensional model creating mechanism performs measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object, calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

36. The apparatus of claim 34, wherein the image candidate detecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

37. The apparatus of claim 36, wherein the image candidate detecting mechanism uses the least squares algorithm to detect said image candidate.

38. The apparatus of claim 34, further comprising a pose variant creating mechanism for creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein said image space creating mechanism creates said image space for each of said pose variants.

39. The apparatus of claim 38, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

40. The apparatus of claim 39, wherein said specified range is determined based on a possible error of said each pose candidate.

41. A pose estimation apparatus comprising:

a memory;

a 3D model data formulating mechanism for formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

a feature points extracting and estimating mechanism for extracting feature points from said object and extracting feature points from an input image and estimating a possible error of the extracted feature points of said input image;

a pose candidate creating mechanism for creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in said memory;

a pose variants creating mechanism for successively retrieving a pose candidate from said memory and creating a plurality of pose variants from the retrieved pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from the retrieved pose candidate over a range determined by the estimated possible error;

an image space creating mechanism for creating, for each of said pose variants, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose variant, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions; and

an image candidate detecting and selecting mechanism for detecting, for each said pose variant, an image candidate within said image space by using said 3D model data, determining a distance from the image candidate to said input image and selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

42. The apparatus of claim 41, wherein the 3D model data formulating mechanism performs the functions of:

measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object;

calculating basis textures approximating said plurality of illumination variation textures; and

formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

43. The apparatus of claim 41, wherein said image candidate detecting and selecting mechanism detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

44. The apparatus of claim 43, wherein said image candidate detecting and selecting mechanism uses the least squares algorithm to detect said image candidate.

45. A computer-readable storage medium containing a computer-executable program which comprises the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) creating, for each of said pose candidates, a image space representing brightness values of a two-dimensional image object of same position and orientation as said each pose candidate, which brightness values would be obtained if said image object is illuminated under varying lighting conditions;

c) detecting, for each of said pose candidates, an image candidate within said image space by using said 3D model data and determining a distance from said image candidate to an input image; and

d) selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

46. The computer-readable storage medium of claim 45, wherein step (a) comprises the steps of measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object, calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

47. The computer-readable storage medium of claim 45, wherein step (c) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

48. The computer-readable storage medium of claim 47, wherein step (c) uses the least squares algorithm to detect said image candidate.

49. The computer-readable storage medium of claim 45, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (b) creates said image space for each of said pose variants.

50. The computer-readable storage medium of claim 49, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

51. The computer-readable storage medium of claim 50, wherein said specified range is determined based on a possible error of said each pose candidate.

52. A computer-readable storage medium containing a computer-executable program which comprises the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) successively reading one of said pose candidates from said memory;

c) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate;

d) creating, for each of said pose variants, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose variant, wherein said brightness values would be obtained if said object is illuminated under varying lighting conditions;

e) detecting, for each said pose variant, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to an input image;

f) repeating steps (b) to (e) to produce a plurality of said image candidates for each of said pose candidates;

g) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest;

h) comparing the pose candidate selected by step (g) with a previously selected pose candidate; and

i) replacing the previously selected pose candidate with the pose candidate currently selected by step (g) if the currently selected pose candidate is better than the previously selected pose candidate, and repeating steps (b) to (g) until the previously selected pose candidate is better than the currently selected pose candidate.

53. The computer-readable storage medium of claim 52, wherein step (a) comprises the steps of measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object, calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

54. The computer-readable storage medium of claim 52, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

55. The computer-readable storage medium of claim 54, wherein step (e) uses the least squares algorithm to detect said image candidate.

56. A computer-readable storage medium containing a computer-executable program which comprises the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) extracting feature points from said object and extracting feature points from an input image;

c) creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in a memory;

d) creating, for each of said pose candidates, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose candidate, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions;

e) detecting, for each of said pose candidates, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to said input image; and

f) selecting one of the pose candidates which corresponds to the image candidate whose distance to said input image is smallest.

57. The computer-readable storage medium of claim 56, wherein step (a) comprises the steps of measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object, calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

58. The computer-readable storage medium of claim 56, wherein step (e) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

59. The computer-readable storage medium of claim 58, wherein step (e) uses the least squares algorithm to detect said image candidate.

60. The computer-readable storage medium of claim 57, further comprising the step of creating a plurality of pose variants from each of said pose candidates such that each of the pose variants is displaced in position and orientation by a predetermined amount from said each pose candidate, and wherein step (c) creates said image space for each of said pose variants.

61. The computer-readable storage medium of claim 60, wherein each of said pose variants is axially and rotationally displaced by said predetermined amount in a three-dimensional coordinate system over a specified range centered about said each pose candidate.

62. The computer-readable storage medium of claim 61, wherein said specified range is determined based on a possible error of said each pose candidate.

63. A computer-readable storage medium containing a computer-executable program which comprises the steps of:

a) formulating 3D model data indicating a three-dimensional shape of an object and reflectivity or color at every point of said object;

b) extracting feature points from said object and extracting feature points from an input image;

c) estimating a possible error of the extracted feature points of said input image;

d) creating a plurality of pose candidates from the extracted feature points of said object and the extracted feature points of said input image and storing the pose candidates in a memory;

e) successively reading one of said pose candidates from said memory;

f) creating a plurality of pose variants from said one pose candidate such that each of the pose variants is displaced in position and orientation by a predetermined amount from said one pose candidate over a range determined by said possible error estimated by step (c);

g) creating, for each of said pose variants, an image space representing brightness values of a plurality of two-dimensional images of said object placed in the same position and orientation as said each pose variant, wherein said brightness values would be obtained if said image object is illuminated under varying lighting conditions;

h) detecting, for each said pose variant, an image candidate within said image space by using said 3D model data and determining a distance from the image candidate to said input image;

i) repeating steps (e) to (h) to produce a plurality of said image candidates for each of said pose candidates; and

j) selecting one of the pose candidates corresponding to the image candidate whose distance to said input image is smallest.

64. The computer-readable storage medium of claim 63, wherein step (a) comprises the steps of measuring reflectivity or color at every point of said object under varying illuminations to produce a plurality of illumination variation textures each representing a brightness value of said every point of said object calculating basis textures approximating said plurality of illumination variation textures, and formulating said 3D model data so that the image data indicates said basis textures as well as said three-dimensional shape.

65. The computer-readable storage medium of claim 63, wherein step (h) detects said image candidate when distance from the image candidate to said input image is nearest within said image space.

66. The computer-readable storage medium of claim 65, wherein step (h) uses the least squares algorithm to detect said image candidate.