Apparatus for and method of constructing multi-view face database, and apparatus for and method of generating multi-view face descriptor

• US 20050013507A1
• Filed: 07/15/2004
• Published: 01/20/2005
• Est. Priority Date: 07/15/2003
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Won-sook, Sohn, Kyung-ah
• Granted Patent: US 7,643,684 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

An apparatus for generating a multi-view face descriptor includes a multi-view face database storing sets of training face images and sets of test face images, each set including a mosaic view of a single identity; a basis matrix generator generating a basis matrix of the training face images shown from respective viewpoints searched from the multi-view face database; a feature extractor extracting feature vectors of a test face image shown from each viewpoint using the basis matrix of the training face images searched from the multi-view face database; a to-be-registered view selector searching the test face images using the feature vectors extracted by the feature extractor and selecting a plurality of viewpoints as registered viewpoints according to the search results; and a face descriptor generator generating a multi-view face descriptor by synthesizing face descriptors having the plurality of selected viewpoints.

Claims

1. An apparatus for constructing a multi-view face database comprising:

an image capturer which captures a plurality of face images of a person shown from various viewpoints including a front view face image to be included in a mosaic view;

an eye position determiner which selects a plurality of key-frame images from the face images provided by the image capturer and positions of eyes corresponding to respective viewpoints to be included in the mosaic view using positions of eyes of the selected key-frame images;

a face localization unit which performs localization of the face images included in the mosaic view based on the eye positions of the respective viewpoints, and arranges sizes and the eye positions of the face images shown from the respective viewpoints; and

a mosaic view generator which generates the mosaic view of the person by synthesizing the localized face images having the respective viewpoints.

2. The apparatus of claim 1, wherein the face localization unit comprises:

an image cropping unit which crops face images having left rotation angles among face images shown from each viewpoint included in the mosaic view, based on a right eye position set in the front view face image and crops face images having right rotation angles, based on a left eye position set in the front view face image; and

a normalization unit which normalizes sizes of the face images provided by the image cropping unit to a predetermined size.

3. The apparatus of claim 1, wherein the mosaic view generator generates a mosaic view having views −

90°

through 90°

horizontal rotation angles and vertical rotation angles between −

30°

through 30° and

−

60°

through 60°

.

4. The apparatus of claim 3, wherein a view interval between views is at or greater than 10°

in the horizontal and vertical rotation angles.

5. A method of constructing a multi-view face database comprising:

capturing a plurality of face images of a person shown from various viewpoints including a front view face image to be included in a mosaic view;

selecting a plurality of key-frame images from the captured face images and determining positions of eyes corresponding to respective viewpoints included in the mosaic view using positions of eyes of the selected key-frame images;

performing localization of the face images included in the mosaic view based on the eye positions of the respective viewpoints, and arranging sizes and the eye positions of the face images shown from the respective viewpoints; and

generating the mosaic view of the person by synthesizing the localized face images having the respective viewpoints.

6. The method of claim 5, wherein performing localization of the face images comprises:

cropping face images having left rotation angles among face images shown from each viewpoint included in the mosaic view, based on a right eye position set in the front view face image and crops face images having right rotation angles, based on a left eye position set in the front view face image; and

normalizing sizes of the face images provided by the image cropping unit to a predetermined size.

7. An apparatus for generating a multi-view face descriptor comprising:

a multi-view face database which stores sets of training face images and sets of test face images, each set including a mosaic view of a single identity;

a basis matrix generator which generates a basis matrix of the training face images shown from respective viewpoints searched from the multi-view face database;

a feature extractor which extracts feature vectors of a test face image shown from each viewpoint using the basis matrix of the training face images searched from the multi-view face database;

a to-be-registered view selector which searches the test face images using the feature vectors extracted by the feature extractor and selects a plurality of the viewpoints as registered viewpoints according to the search results; and

a face descriptor generator which generates a multi-view face descriptor by synthesizing face descriptors having the plurality of selected viewpoints.

8. The apparatus of claim 7, wherein the basis matrix generator performs training of viewpoints adjacent to a predetermined viewpoint as well as the predetermined viewpoint.

9. The apparatus of claim 7, wherein the basis matrix generator designates different sub-regions on the face images, according to up and down rotation angles and left and right rotation angles of the face images.

10. The apparatus of claim 9, wherein the basis matrix generator designates different feature vectors in the sub-regions on the face images, according to up and down rotation angles and left and right rotation angles of the face images.

11. The apparatus of claim 7, wherein the feature extractor designates different sub-regions on the face images, according to up and down rotation angles and left and right rotation angles of the face images.

12. The apparatus of claim 11, wherein the feature extractor designates different feature vectors in the sub-regions on the face images, according to up and down rotation angles and left and right rotation angles of the face images.

13. The apparatus of claim 7, wherein the feature extractor generates an entire Fourier feature vector and sub-region Fourier feature vectors of the test face image and sets different sub-regions according to the viewpoints.

14. The apparatus of claim 7, wherein the to-be-registered view selector statistically analyzes face descriptors with a plurality of viewpoints, whose quasi view size is larger than a predetermined value, and viewpoints, which frequently appear in real life, and selects the registered viewpoints by combining at least one standard among the face descriptors with a plurality of viewpoints according to the analysis result and face descriptors with a plurality of viewpoints, with which quasi views corresponding to the given viewpoints can cover the mosaic view.

15. A method of generating a multi-view face descriptor comprising:

providing a multi-view face database storing sets of training face images and sets of test face images, each set including a mosaic view of a single identity;

generating a basis matrix of the training face images shown from respective viewpoints searched from the multi-view face database;

extracting feature vectors of a test face image shown from each viewpoint using the basis matrix of the training face images searched from the multi-view face database; and

searching the test face images using the feature vectors extracted by the feature extractor, selecting a plurality of viewpoints as registered viewpoints according to the search results and generating a multi-view face descriptor by synthesizing face descriptors having the plurality of selected viewpoints.

16. The method of claim 15, wherein, in the generating of the multi-view face descriptor, face descriptors with a plurality of viewpoints, whose quasi view size is larger than a predetermined value, are selected.

17. The method of claim 15, wherein, in the generating of the multi-view face descriptor, viewpoints, which frequently appear in real life, are statistically analyzed, and the face descriptors with a plurality of viewpoints are selected according to the analysis result.

18. The method of claim 15, wherein, in the generating of the multi-view face descriptor, face descriptors with a plurality of viewpoints, with which quasi views corresponding to the given viewpoints can cover the mosaic view, are selected.

19. The method of claim 15, wherein the generating of a basis matrix further comprises training the selected respective viewpoints and adjacent viewpoints together.

20. At least one computer readable medium having recorded thereon a computer readable program for performing a method of constructing a multi-view face database performed by one or more computers, the method comprising:

capturing a plurality of face images of a person shown from various viewpoints including a front view face image to be included in a mosaic view;

selecting a plurality of key-frame images from the captured face images and determining positions of eyes corresponding to respective viewpoints included in a mosaic view using positions of eyes of the selected key-frame images;

performing localization of the face images included in the mosaic view based on the eye positions of the respective viewpoints, and arranging sizes and the eye positions of the face images shown from the respective viewpoints; and

generating the mosaic view of the person by synthesizing the localized face images having the respective viewpoints.

21. At least one computer readable medium having recorded thereon a computer readable program for performing a method of generating a multi-view face descriptor performed by one or more computers, the method comprising:

providing a multi-view face database storing sets of training face images and sets of test face images, each set including a mosaic view of a single identity;

generating a basis matrix of the training face images shown from respective viewpoints searched from the multi-view face database;

extracting feature vectors of a test face image shown from each viewpoint using the basis matrix of the training face images searched from the multi-view face database; and

searching the test face images using the feature vectors extracted by the feature extractor, selecting a plurality of viewpoints as registered viewpoints according to the search results and generating a multi-view face descriptor by synthesizing face descriptors having the plurality of selected viewpoints.

22. An apparatus for constructing a multi-view face database using a plurality of images of an object shown from various viewpoints including a primary viewpoint to be included in a mosaic view, the apparatus comprising:

a feature position determiner which obtains positions of a predetermined feature in each of the images having respective viewpoints and which are to be included in the mosaic view;

a localization unit which performs localization of the images to be included in the mosaic view, and arranges sizes and the feature positions of the selected images based on the obtained feature positions of the respective viewpoints; and

a mosaic view generator generates the mosaic view of the object by synthesizing the localized images having the respective viewpoints.

23. The apparatus of claim 22, wherein the localization unit comprises a cropping unit which, for each image, adjusts the image as compared to the obtained position of the feature in the image at the primary view point according to a relative degree of rotation as compared to the primary view point.

24. The apparatus of claim 23, wherein the feature of image of the primary view point is located at a first coordinate, and the localization unit adjusts each image such that a coordinate of the feature is at the first coordinate.

25. The apparatus of claim 22, wherein each viewpoint of the images in the generated mosaic view is at least a predetermined rotational difference from another viewpoint of the images.

26. The apparatus of claim 25, wherein the predetermined rotational difference is at least a 10°

rotational difference.

27. The apparatus of claim 25, wherein the predetermined rotational difference for each image at a corresponding viewpoint is a difference between which a retrieval error for a query image at substantially the same viewpoint is at or less than 0.05 of an Average Normalized Modified Retrieval Rank (ANMRR).

28. The apparatus of claim 22, further comprising a feature extractor which, for selected ones of each viewpoint in the mosaic view, extracts feature vectors of the corresponding image and images adjacent the corresponding image in the mosaic view, and a descriptor generator which generates an object descriptor including the extracted feature vectors for use with the corresponding image for the viewpoint such that the selected viewpoint having the object descriptor allows identification of an image of an object not at the viewpoint.

29. The apparatus of claim 28, wherein:

the feature extractor which, for each viewpoint in the mosaic view, obtains at least one subregion of the corresponding image and the images adjacent the corresponding image in the mosaic view, extracts subregion feature vectors for the at least one subregion of the corresponding image and images adjacent the corresponding image in the mosaic view, and

the descriptor generator further includes the subregion feature vectors in the object descriptor.

30. An apparatus for generating a multi-view descriptor relative to a selected view using a multi-view database which stores sets of training images and sets of test images, each set including a mosaic view of a single identity, comprising:

a basis matrix generator which generates a basis matrix of the training images shown from respective viewpoints searched from the multi-view database;

a feature extractor which extracts feature vectors of a test image shown from each viewpoint using the basis matrix of the training images;

a view selector which searches the test images using the feature vectors extracted by the feature extractor and selects ones of the viewpoints as registered viewpoints according to the search results; and

a descriptor generator which generates a multi-view descriptor by synthesizing descriptors having the plurality of selected viewpoints.

31. The apparatus of claim 30, wherein:

the feature vectors comprise entire image feature vectors and subregion feature vectors, and

the feature extractor further selects sub-regions of the test images and obtains the subregion feature vectors from the subregions test images, and obtains the entire image feature vectors from the test images.

32. The apparatus of claim 31, wherein each of the subregion feature vectors and the entire image feature vector corresponds to one of the selected viewpoints and enable identification of the corresponding object when the viewpoint is other than the selected viewpoint.

33. The apparatus of claim 30, wherein each feature vector defines at least one quasi view of a corresponding one of the selected viewpoints and which is not included in the selected viewpoints, and the at least one quasi view allows identification of an object in a rotational range relative to the corresponding one selected viewpoint.

34. The apparatus of claim 33, wherein the rotational range of the quasi view relative to the corresponding one selected viewpoint varies as a function of rotational angle of the selected viewpoint relative to a primary viewpoint.

35. The apparatus of claim 31, wherein a number of subregions for one of the selected viewpoints is different from a number of subregions for another one of the selected viewpoints.

36. The apparatus of claim 31, wherein the feature extractor comprises:

a fourier transformer which, for each test image, fourier transforms the test image and at least one of the subregions of the test image,

an entire image projection unit which, for each test image, projects the fourier transformed test image to obtain the entire image feature vector for the test image, and

a subregion image projection unit which, for each subregion of the test image, projects the fourier transformed subregion image to obtain the subregion image feature vector for the subregion of the test image.

37. The apparatus of claim 36, wherein the entire image projection unit and the subregion image projection unit use at least one of a principal component analysis (PCA) method, a linear discriminant analysis (LDA) method, or a Principal Component Linear Discriminant Analysis (PCLDA) method to obtain the entire image and subregion image feature vectors.

38. An object recognition apparatus comprising:

a database of mosaic views of objects, with each mosaic view comprising selected images of one of the objects from different viewpoints, and quasi images of viewpoints not included in the selected images and which are described by descriptors in relation to corresponding selected images of the mosaic view; and

a comparison unit which compares a received image at a viewpoint with the mosaic views in the database in order to determine a correlation of the received image and the descriptors and selected images in order to determine an identity of an object in the received image.

39. The object recognition apparatus of claim 38, wherein the database is generated by:

for sets of training object images and sets of test object images, each set including a mosaic view of a single identity, generating a basis matrix of the training object images shown from respective viewpoints searched from the multi-view object database;

extracting feature vectors of a test object image shown from each viewpoint using the basis matrix of the training object images searched from the multi-view object database; and

searching the test object images using the feature vectors extracted by the feature extractor, selecting a plurality of viewpoints as registered viewpoints according to the search results and generating a multi-view object descriptor by synthesizing object descriptors having the plurality of selected viewpoints.