Canonical correlation analysis of image/control-point location coupling for the automatic location of control points

• US 6,400,828 B2
• Filed: 02/13/2001
• Issued: 06/04/2002
• Est. Priority Date: 05/21/1996
• Status: Expired due to Fees

• Current Assignee: Hanger Solutions, LLC (IP Investments Group LLC)
• Original Assignee: Interval Research Corporation
• Inventors: Covell, Michele, Slaney, Malcolm
• Primary Examiner(s): MEHTA, BHAVESH M
• Application Number:
  US09/781,229
• Publication Number:
  US 20010028731A1
• Time in Patent Office: 476 Days
• Field of Search: 382/100, 382/107, 382/155, 382/156, 382/181, 382/203, 382/209, 382/218, 382/224, 382/225, 382/228, 382/276, 382/278, 382/282, 382/284, 382/285, 382/307, 382/308, 382/118, 382/243, 345/419, 345/420, 345/582, 345/648, 345/700, 345/863, 345/619, 348/575, 348/576, 348/578
• US Class Current: 382/100
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/476 using statistical shape modelling, e.g. point distribution models
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

The identification of hidden data, such as feature-based control points in an image, from a set of observable data, such as the image, is achieved through a two-stage approach. The first stage involves a learning process, in which a number of sample data sets, e.g. images, are analyzed to identify the correspondence between observable data, such as visual aspects of the image, and the desired hidden data, such as the control points. Two models are created. A feature appearance-only model is created from aligned examples of the feature in the observed data. In addition, each labeled data set is processed to generate a coupled model of the aligned observed data and the associated hidden data. In the second stage of the process, the modeled feature is located in an unmarked, unaligned data set, using the feature appearance-only model. This location is used as an alignment point and the coupled model is then applied to the aligned data, giving an estimate of the hidden data values for that data set.

Claims

1. A method for determining continuous-valued hidden data from observable data, comprising the steps of:

A) conducting a training stage which includes the steps of;

labelling a plurality of representative sets of unaligned observed data to identify correct alignment of the observed data and continuous-valued hidden data associated with each set of observed data;

analyzing the observed data to generate a first model which represents the aligned observed data;

generating a second model on the aligned and labeled data sets which explicitly represents the coupling between aligned observable data and the hidden data;

B) for each set of unlabeled data, conducting a labelling stage which includes the steps of;

analyzing the unlabeled set of unaligned observed data by means of the first model to determine alignment of the observable data associated therewith;

applying the second model to said unlabeled set of aligned observed data; and

determining hidden data for the unlabeled set of aligned data from said application of the second model.

2. The method of claim 1 wherein said second model is generated by means of canonical correlation.

3. The method of claim 1 wherein each set of unaligned observed data defines an image.

4. The method of claim 3 wherein said hidden data comprises control points which relate to fiduciary points on objects in an image.

5. The method of claim 4 wherein at least some of said control points relate to fiduciary points on obscured portions of objects in the images.

6. The method of claim 4 wherein control points are determined for at least two new images, and further including the step of morphing between said new images in accordance with the determined control points.

7. The method of claim 4 further including the step of creating a composite image by incorporating a new image into another image by means of the determined control points for each of the two images.

8. The method of claim 4 wherein said images include faces, and further including the step of analyzing the control points to recognize a known face in an image.

9. The method of claim 4 wherein said images comprise cartoons.

10. The method of claim 4 wherein said images include faces, and further including the step of analyzing the control points to recognize an expression on a face in an image.

11. The method of claim 4, further including the step of controlling a robot to grasp an object in accordance with the fiduciary points that are labeled in the image of the object.

12. The method of claim 1 wherein said sets of unaligned observed data comprise a sequence of video images.

13. The method of claim 12 further including the step of analyzing determined control points in said sequence of video images to recognize movement of an object in the images.

14. The method of claim 13 wherein said movement comprises nearly periodic motion.

15. The method of claim 1 wherein said sets of unaligned observed data comprise audio signals.

16. The method of claim 1 further including the steps of selecting a plurality of said representative sets of data, using hidden data in said plurality of data sets to automatically generate interpolated data sets that are based on said plurality of data sets and that include both observable and hidden data, and including said interpolated data sets in the plurality of representative data sets that are analyzed to generate said second model.

17. The method of claim 16 wherein said second model is a multifaceted model, and said interpolated data sets are at the boundaries of facets in said second model.

18. The method of claim 1 further including the steps of selecting a plurality of said representative sets of data, using hidden data in said plurality of data sets to automatically generate interpolated data sets that are based on said plurality of data sets and that contain observable data, and including said interpolated data sets in the plurality of representative data sets that are analyzed to generate said first model.

19. The method of claim 18 wherein said first model is a multifaceted model, and said interpolated data sets are at the boundaries of facets in said first model.

20. The method of claim 1 wherein said applying and determining steps are carried out in a non-iterative manner.

21. The method of claim 1 wherein said first model is a manifold model.

22. The method of claim 21 wherein said first model is an affine manifold model.

23. The method of claim 21 wherein the step of aligning the observed data in an unlabeled data set comprises the steps of:

i) selecting possible locations for the alignment of the data;

ii) for each possible location, determining a lower bound for the distance between the unlabeled data set aligned at that location and an expected appearance of aligned data, in accordance with an average appearance defined by the first model;

iii) removing the possible locations whose lower bound exceeds a threshold value;

iv) for each possible location, determining the coordinate value for a dimension of the first model;

v) for each possible location, determining a new lower bound by combining previously determined coordinate values with the distance between the data set aligned at that location and the appearance of the data set under said alignment in accordance with the previously determined coordinate values; and

vi) repeating steps iii), iv) and v) for all of the dimensions of the model.

24. The method of claim 23 wherein said lower bounds are determined in accordance with expected variances along each of the dimensions of the manifold model.

25. The method of claim 24 wherein said expected variances are progressively smaller on each successive repetition of said steps.

26. The method of claim 22 wherein the step of applying the second model to the unlabeled set of aligned observed data includes

projecting, with the use of an orthonormal transform, the aligned observed unlabeled data onto a subspace of the second model having fewer dimensions than said second model;

performing a general matrix multiplication within said subspace; and

projecting, with the use of an orthonormal transform, into a second space of the model to determine hidden data for the unlabeled data set.

27. The method of claim 26 wherein said general matrix multiplication is determined, in part, according to a gradual roll-off in manifold dimensions according to the coherence between the hidden and aligned observed data that is used to generate said second model.

28. The method of claim 1 further including the step defining the alignment of the observed data in the representative sets of data from an analysis of the hidden data with which the data sets are labeled.

29. The method of claim 28 wherein an analysis of the observed data is also employed in said alignment process.

30. The method of claim 28 wherein said defining step comprises dividing the hidden data into separate groups, and assigning a different definition of aligned observed data in each representative data set to the respective groups.

31. The method of claim 30 wherein the division of the hidden data into separate groups is determined in accordance with analysis of the hidden data.

32. The method of claim 30 wherein the definition of aligned observed data is determined in accordance with analysis of the hidden data.

33. The method of claim 32 wherein the definition of aligned observed data is also determined in accordance with analysis of the observed data.

34. The method of claim 31 wherein the observed data is also used to divide the hidden data into said groups.

35. The method of claim 32 wherein the division of hidden data into groups is carried out by measuring the coherence of the hidden data.