Method for video-based nose location tracking and hands-free computer input devices based thereon

• US 6,925,122 B2
• Filed: 07/25/2002
• Issued: 08/02/2005
• Est. Priority Date: 07/25/2002
• Status: Expired due to Fees

• Current Assignee: The National Research Council (The National Academies of Science)
• Original Assignee: The National Research Council (The National Academies of Science)
• Inventors: Gorodnichy, Dmitry O.
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US10/201,957
• Publication Number:
  US 20040017472A1
• Time in Patent Office: 1,104 Days
• Field of Search: H04/N00.7/12, G06/K00.9/00, 375/240.02, 375/240.16, 375/240.17, 375/240.03, 375/240.08, 375/240.21, 348/169, 348/396.1, 348/402.1, 348/406.1, 348/407.1, 348/414.1, 348/416.1, 348/417.1, 348/418.1, 382/181, 382/190, 382/194, 382/201, 382/220, 382/232, 382/239, 382/243, 382/249, 382251-253
• US Class Current: 375/240.16
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A method for tracking the location of the tip of the nose with a video camera, and a hands-free computer input device based thereon have been described. According to this invention, a convex shape such as the shape of the tip of the nose, is a robust object suitable for precise and smooth location tracking purposes. The disclosed method and apparatus are substantially invariant to changes in head pose, user preferred seating distances and brightness of the lighting conditions. The location of the nose can be tracked with pixel and sub-pixel accuracy.

Claims

1. A method for tracking the location of a moveable three-dimensional convex shape with a video camera, comprising the steps of:

step a;

defining the location of an X-spot on said convex shape, said location being selected from the group consisting of a point on said convex shape closest to the video camera, and a point on said convex shape closest to a fixed point in space, said location being moveable on said convex shape as said convex shape changes location and orientation in space;

step b;

storing a digitized video image of said convex shape in a vicinity of said X-spot, the size of said vicinity being defined by the surface area of said convex shape with a substantially constant spherical curvature, said digitized video image having a luminance pattern referred to as a stored X-luminance pattern, said stored X-luminance pattern being stored as a stored matrix;

step c;

defining the location of a reference point selected from the group consisting of the centre of said stored X-luminance pattern, and a location within said stored X-luminance pattern;

step d;

registering a plurality of digitized video images, each one of said plurality of video images containing a video image of said convex shape which is referred to as a registered X-luminance pattern, said registered X-luminance pattern being of the same size as said stored X-luminance pattern, each one of said plurality of video images being registered as a registered matrix;

step e;

comparing said stored matrix and said registered matrix in each one of said plurality of video images on a pixel-by-pixel basis to determine the two-dimensional location of said reference point in each one of said plurality of video images with pixel accuracy; and

step g;

producing the two-dimensional location of said reference point as information useable by a computer in each one of said plurality of video images;

said method being substantially invariant to rotation of said convex shape.

2. The method according to claim 1, further comprising a step f, said step f being performed after said step e and prior to said step g, said step f performing an average weighting operation in each one of said plurality of video images, said operation being performed in a neighbourhood of the two-dimensional location of said reference point determined with pixel accuracy on a pixel-by-pixel basis, said neighbourhood being comprised of the pixel location of said reference point determined with pixel accuracy and at least one of the immediate neighbour pixel locations, said operation giving a weight to each one of the two-dimensional pixel locations which is proportional to the correlation between the corresponding elements of said stored matrix and said registered matrix at that pixel location, said operation determining the two-dimensional location of said reference point in each one of said plurality of video images with sub-pixel accuracy within one pixel distance of the two-dimensional location of the reference point determined with pixel accuracy.

3. The method according to claim 1, wherein the modulus of the stored matrix and the modulus of the registered matrix are normalized to unity, resulting in said method being substantially invariant to changes in brightness in lighting conditions.

4. The method according to claim 1, wherein the comparison in step e is performed by performing a mathematical normalized correlation operation.

5. The method according to claims 1, wherein step d is further comprised of scaling each one of said plurality of video images to make said method invariant to the size of said convex shape and the distance of said convex shape from the video camera.

6. The method according to claim 1, wherein step d is further comprised of filtering each one of said plurality of video images with an averaging filter for the reduction of noise.

7. The method according to claim 5, wherein step d is further comprised of filtering each one of said plurality of video images with an averaging filter for the reduction of noise.

8. The method according to claim 1, wherein step e is further comprised of defining a search window in each one of said plurality of video images within which said comparison is performed, said search window being selected from the group consisting of:

a. a search window covering a substantially square area which is smaller than the video image area of a video frame, the centre of said area being substantially at the location of the reference point in the previous video image of said plurality of video images, if said location is known,

b. a search window covering an area determined by an automated object detection technique, and

c. a search window covering the entire video image area of a video frame.

9. The method according to claim 1, wherein said convex shape is the shape of the tip of a nose in a face, said step a being further comprised of positioning said tip of a nose at a substantially centre position in a video image with said face facing straight into a video camera.

10. The method according to claim 9, wherein step e is further comprised of defining a search window in each one of said plurality of video images within which said comparison is to be performed, said search window being selected from the group consisting of:

a. a search window covering a substantially square area having a side length ranging from a quarter the width of said face to the width of said face, said area having a centre at the location of the tip of the nose in the previous video image of said plurality of video images, if said location is known,

b. a search window covering a substantially rectangular area determined by an automated face detection technique, and

c. a search window covering the entire video image area of a video frame.

11. The method according to claim 9, wherein each one of the digitized video images is scaled such that the width of the face in each one of the scaled video images is between 50 and 80 pixels wide, and wherein the stored X-luminance pattern is stored as a square matrix with dimensions ranging from 7×

7 to 11×

11.

12. An apparatus for tracking the location of a moveable three-dimensional convex shape with a video camera, comprising:

a. means defining the location of an X-spot on said convex shape, said location being defined from the group consisting of a point on said convex shape closest to the video camera, and a point on said convex shape closest to a fixed reference point in space, said location being moveable on said convex shape as said convex shape changes location and orientation in space;

b. means storing a digitized video image of said convex shape in a vicinity of said X-spot, the size of said vicinity being defined by the surface area of said convex shape with a substantially constant spherical curvature;

c. means defining the location of a reference point selected from the group consisting of the centre of said stored video image, and a location within said stored video image;

d. means registering a plurality of digitized video images, each one of said plurality of video images containing a video image of said convex shape;

e. means comparing said stored video image to each one of said plurality of video images to determine the location of said reference point in each one of said plurality of video images with pixel accuracy; and

g. means producing the location of said reference point in each one of said plurality of video images as information useable by a computer;

said apparatus being substantially invariant to rotation of said convex shape.

13. The apparatus according to claim 12, further comprising means performing an average weighting operation on pixel locations in each one of said plurality of video images to determine the location of said reference point in each one of said plurality of video images with sub-pixel accuracy.

14. The apparatus according to claim 12, wherein the storing means and the registering means are further comprised of a normalizing means, resulting in said apparatus being substantially invariant to changes in brightness in lighting conditions.

15. The apparatus according to claim 12, wherein the comparing means performs a mathematical normalized correlation operation.

16. The apparatus according to claim 12, wherein the registering means is further comprised of a scaling means for scaling each one of said plurality of video images, resulting in said apparatus being invariant to the size of said convex shape and the distance of said convex shape from the video camera.

17. The apparatus according to claim 12, wherein said convex shape is the shape of the tip of a nose, said apparatus therefore being useable as a hands-free computer input device.