Object posture estimation/correction system using weight information

• US 20060165293A1
• Filed: 08/27/2004
• Published: 07/27/2006
• Est. Priority Date: 08/29/2003
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Hamanaka, Masahiko
• Granted Patent: US 7,706,601 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

An object pose estimating and matching system is disclosed for estimating and matching the pose of an object highly accurately by establishing suitable weighting coefficients, against images of an object that has been captured under different conditions of pose, illumination. Pose candidate determining unit determines pose candidates for an object. Comparative image generating unit generates comparative images close to an input image depending on the pose candidates, based on the reference three-dimensional object models. Weighting coefficient converting unit determines a coordinate correspondence between the standard three-dimensional weighting coefficients and the reference three-dimensional object models, using the standard three-dimensional basic points and the reference three-dimensional basic points, and converts the standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on the pose candidates. Weighted matching and pose selecting unit calculates weighted distance values or similarity degrees between said input image and the comparative images, using the two-dimensional weighting coefficients, and selects one of the comparative images whose distance value up to the object is the smallest or whose similarity degree with respect to the object is the greatest, thereby to estimate and match the pose of the object.

Claims

1. An object pose estimating and matching system comprising:

reference three-dimensional object model storage means for storing, in advance, reference three-dimensional object models of objects;

reference three-dimensional weighting coefficient storage means for storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models;

pose candidate determining means for determining pose candidates for an object;

comparative image generating means for generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

weighting coefficient converting means for converting said reference three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

5. An object pose estimating and matching system according to claim 1, further comprising:

three-dimensional object model registering means for registering reference three-dimensional object models in said reference three-dimensional object model storage means;

matching and pose selecting means for calculating distance values or similarity degrees between a learning input image and comparative images generated depending on the pose candidates from said reference three-dimensional object models, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object; and

three-dimensional weighting coefficient generating means for generating reference three-dimensional weighting coefficients by learning the degree of importance in matching of each pixel on a three-dimensional model, based on a pixel correspondence between said reference three-dimensional object models, a two-dimensional image determined from an optimum pose, and the three-dimensional model, using the input image and the comparative image of the optimum pose, and registering the generated reference three-dimensional weighting coefficients in said reference three-dimensional weighting coefficient storage means.

17. An object pose estimating and matching system according to claim 1, wherein said object comprises a face.

2. An object pose estimating and matching system comprising:

reference three-dimensional object model storage means for storing, in advance, reference three-dimensional object models of objects;

standard three-dimensional weighting coefficient storage means for storing, in advance, standard three-dimensional weighting coefficients;

reference three-dimensional basic point storage means for storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

standard three-dimensional basic point storage means for storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

pose candidate determining means for determining pose candidates for an object;

comparative image generating means for generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

weighting coefficient converting means for determining a coordinate correspondence between said standard three-dimensional weighting coefficients and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

7. An object pose estimating and matching system according to claim 2, further comprising:

three-dimensional object model registering means for registering reference three-dimensional object models in said reference three-dimensional object model storage means; and

three-dimensional basic point registering means for determining reference three-dimensional basic points with respect to said reference three-dimensional object models, and registering the determined reference three-dimensional basic points in said reference three-dimensional basic point storage means.

3. An object pose estimating and matching system comprising:

reference three-dimensional object model storage means for storing, in advance, reference three-dimensional object models of objects;

variation-specific reference three-dimensional weighting coefficient storage means for storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models and image variations;

pose candidate determining means for determining pose candidates for an object;

variation estimating means for determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

comparative image generating means for generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

weighting coefficient converting means for converting said reference three-dimensional weighting coefficients corresponding to the estimated variation into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

6. An object pose estimating and matching system according to claim 3, further comprising:

three-dimensional object model registering means for registering reference three-dimensional object models in said reference three-dimensional object model storage means;

matching and pose selecting means for calculating distance values or similarity degrees between a learning input image and comparative images generated depending on the pose candidates from said reference three-dimensional object models, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object; and

three-dimensional weighting coefficient generating means for generating variation-specific reference three-dimensional weighting coefficients by learning the degree of importance in matching of each pixel on a three-dimensional model with respect to respective image variations of the input image, based on a pixel correspondence between said reference three-dimensional object models, a two-dimensional image determined from an optimum pose, and the three-dimensional model, using the input image and the comparative image of the optimum pose, and registering the generated variation-specific reference three-dimensional weighting coefficients in said variation-specific reference three-dimensional weighting coefficient storage means.

15. An object pose estimating and matching system according to claim 3, wherein said image variations comprise variations in illuminating conditions.

16. An object pose estimating and matching system according to claim 3, wherein said image variations comprise shape deformations.

4. An object pose estimating and matching system comprising:

reference three-dimensional object model storage means for storing, in advance, reference three-dimensional object models of objects;

variation-specific standard three-dimensional weighting coefficient storage means for storing, in advance, standard three-dimensional weighting coefficients corresponding to image variations;

reference three-dimensional basic point storage means for storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

standard three-dimensional basic point storage means for storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

pose candidate determining means for determining pose candidates for an object;

variation estimating means for determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

comparative image generating means for generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

weighting coefficient converting means for determining a coordinate correspondence between said standard three-dimensional weighting coefficients corresponding to the estimated variation and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

8. An object pose estimating and matching system comprising:

pose-specific reference image storage means for storing, in advance, pose-specific reference images of an object;

pose-specific reference weighting coefficient storage means for storing, in advance, pose-specific reference weighting coefficients corresponding to said reference images;

normalizing means for normalizing an input image to generate a normalized image; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said normalized image and said reference images, using said pose-specific weighting coefficients, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

12. An object pose estimating and matching system according to claim 8, further comprising:

pose-specific reference image registering means for registering pose-specific reference images in said pose-specific reference image storage means;

matching and pose selecting means for calculating distance values or similarity degrees between a learning input image and said reference images, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object; and

pose-specific weighting coefficient generating means for generating pose-specific reference weighting coefficients by learning the degree of importance in matching of each pixel with respect to respective poses, using the input image and the reference image of an optimum pose, and registering the generated pose-specific reference weighting coefficients in said pose-specific reference weighting coefficient storage means.

9. An object pose estimating and matching system comprising:

pose-specific reference image storage means for storing, in advance, pose-specific reference images of an object;

pose-specific standard weighting coefficient storage means for storing, in advance, pose-specific standard weighting coefficients;

normalizing means for normalizing an input image to generate a normalized image; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said normalized image and said reference images, using said pose-specific weighting coefficients, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

14. An object pose estimating and matching system according to claim 9, further comprising:

pose-specific reference image registering means for registering pose-specific reference images in said pose-specific reference image storage means.

10. An object pose estimating and matching system comprising:

pose-specific reference image storage means for storing, in advance, pose-specific reference images of an object;

pose- and variation-specific reference weighting coefficient storage means for storing, in advance, pose- and variation-specific reference weighting coefficients corresponding to said reference images and image variations;

standard three-dimensional object model storage means for storing, in advance, standard three-dimensional object models;

normalizing means for normalizing an input image to generate a normalized image;

variation estimating means for determining a correspondence between an area of a three-dimensional object model and the normalized image, using pose information of said reference images and said standard three-dimensional object models, and estimating a variation based on image information of a given area of said normalized image; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said normalized image and said reference images, using the pose information of said reference images and said pose- and variation-specific weighting coefficients corresponding to the estimated variation, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

13. An object pose estimating and matching system according to claim 10, further comprising:

pose-specific reference image registering means for registering pose-specific reference images in said pose-specific reference image storage means;

matching and pose selecting means for calculating distance values or similarity degrees between a learning input image and said reference images, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object; and

pose- and variation-specific weighting coefficient generating means for generating pose- and variation-specific reference weighting coefficients by learning the degree of importance in matching of each pixel with respect to respective poses and respective image variations of said input image, using the input image and the reference image of an optimum pose, and registering the generated pose- and variation-specific reference weighting coefficients in said pose- and variation-specific reference weighting coefficient storage means.

11. An object pose estimating and matching system comprising:

pose-specific reference image storage means for storing, in advance, pose-specific reference images of an object;

pose- and variation-specific standard weighting coefficient storage means for storing, in advance, pose- and variation-specific standard weighting coefficients corresponding to image variations;

standard three-dimensional object model storage means for storing, in advance, standard three-dimensional object models;

normalizing means for normalizing an input image to generate a normalized image;

variation estimating means for determining a correspondence between an area of a three-dimensional object model and the normalized image, using pose information of said reference images and said standard three-dimensional object models, and estimating a variation based on image information of a given area of said normalized image; and

weighted matching and pose selecting means for calculating weighted distance values or similarity degrees between said normalized image and said reference images, using the pose information of said reference images and said pose- and variation-specific weighting coefficients corresponding to the estimated variation, and selecting one of the reference images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

18. An object pose estimating and matching program for enabling a computer to perform a process comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models;

determining pose candidates for an object;

generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

converting said reference three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

19. An object pose estimating and matching program for enabling a computer to perform a process comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, standard three-dimensional weighting coefficients;

storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

determining pose candidates for an object;

generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

determining a coordinate correspondence between said standard three-dimensional weighting coefficients and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

20. An object pose estimating and matching program for enabling a computer to perform a process comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models and image variations;

determining pose candidates for an object;

determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

converting said reference three-dimensional weighting coefficients corresponding to the estimated variation into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

21. An object pose estimating and matching program for enabling a computer to perform a process comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, standard three-dimensional weighting coefficients corresponding to image variations;

storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

determining pose candidates for an object;

determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

determining a coordinate correspondence between said standard three-dimensional weighting coefficients corresponding to the estimated variation and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

22. (canceled)

23. (canceled)

24. (canceled)

25. (canceled)

26. (canceled)

27. (canceled)

28. (canceled)

29. (canceled)

30. (canceled)

31. (canceled)

32. (canceled)

33. (canceled)

34. (canceled)

35. A method of object pose estimating and matching comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models;

determining pose candidates for an object;

generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

converting said reference three dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to match and estimate the pose of said object.

36. A method of object pose estimating and matching comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, standard three-dimensional weighting coefficients;

storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

determining pose candidates for an object;

generating comparative images close to an input image depending on said pose candidates, based on said reference three-dimensional object models;

determining a coordinate correspondence between said standard three-dimensional weighting coefficients and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

37. A method of object pose estimating and matching comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, reference three-dimensional weighting coefficients corresponding to said reference three-dimensional object models and image variations;

determining pose candidates for an object;

determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

converting said reference three-dimensional weighting coefficients corresponding to the estimated variation into two-dimensional weighting coefficients depending on said pose candidates, using said reference three-dimensional object models; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.

38. A method of object pose estimating and matching comprising the steps of:

storing, in advance, reference three-dimensional object models of objects;

storing, in advance, standard three-dimensional weighting coefficients corresponding to image variations;

storing, in advance, reference three-dimensional basic points corresponding to said reference three-dimensional object models;

storing, in advance, standard three-dimensional basic points corresponding to standard three-dimensional object models;

determining pose candidates for an object;

determining a correspondence between an area of a three-dimensional object model and an input image, using said pose candidates and said reference three-dimensional object models, and estimating a variation based on image information of a given area of said input image;

generating comparative images close to said input image depending on said pose candidates, based on said reference three-dimensional object models;

determining a coordinate correspondence between said standard three-dimensional weighting coefficients corresponding to the estimated variation and said reference three-dimensional object models, using said standard three-dimensional basic points and said reference three-dimensional basic points, and converting said standard three-dimensional weighting coefficients into two-dimensional weighting coefficients depending on said pose candidates; and

calculating weighted distance values or similarity degrees between said input image and said comparative images, using said two-dimensional weighting coefficients, and selecting one of the comparative images whose distance value up to said object is the smallest or whose similarity degree with respect to said object is the greatest, thereby to estimate and match the pose of said object.