Image-based head position tracking method and system

• US 8,879,801 B2
• Filed: 10/03/2011
• Issued: 11/04/2014
• Est. Priority Date: 10/03/2011
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ragland, Richard R.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US13/251,530
• Publication Number:
  US 20130083976A1
• Time in Patent Office: 1,128 Days
• Field of Search: 382/103, 382/117, 382/128, 382/173, 382/294, 351/208, 351/209, 351/246
• US Class Current: 382/117
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Systems and methods for tracking a head position of a user include obtaining digital images of the user'"'"'s head, processing the images to locate anatomical structures beneath the visible surface, and using those determined locations as inputs to a computing device. In an embodiment, images of a user'"'"'s face are processed to identify the irises of the eyes, identify pixels along a boundary between each iris and the surrounding sclera, determine an ellipse defined by the identified iris-sclera boundary pixels, determine a distance-to-pixel ratio based on a pixel length of a long axis of such an ellipse compared to a known or presumed diameter of the iris, locating the iris in a three-axis coordinate system, determining an optical axis vector of the eye in the three-axis coordinate system, and calculating a center of the eyeball based on the optical axis vector and a known or presumed eyeball radius.

Claims

1. A method for controlling a computing device, comprising:

obtaining a first digital image of a face of a user of the computing device with a digital camera;

processing the first digital image to determine a position and orientation of an eyeball of the user beneath a surface of the user'"'"'s skin by;

determining an ellipse equation characterizing a first apparent iris disc in a plane of the first digital image by performing a least squares fit of iris-to-sclera boundary pixels in the first digital image; and

determining a spherical center point location of the eyeball using the determined ellipse equation characterizing the first apparent iris disc in the plane of the first digital image; and

translating the position determined and the orientation of the user'"'"'s eyeball into an input command;

wherein determining the spherical center point location of the eyeball comprises;

determining coordinates of a center point of an iris disc in a three-dimensional coordinate system based on the determined ellipse equation characterizing the first apparent iris disc;

calculating a first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc;

projecting the first optical axis vector into the user'"'"'s eyeball;

measuring one eyeball radius distance along the projected first optical axis vector from the surface of the iris disc in the user'"'"'s eyeball;

storing the projected first optical axis vector;

calculating at least a second optical axis vector of the iris disc by;

prompting the user to look in another direction;

obtaining at a second digital image of the user'"'"'s face;

determining an ellipse equation characterizing a second apparent iris disc in a plane of the second digital image; and

determining three-dimensional coordinates of a second center point of the iris disc based on the determined ellipse equation characterizing the second apparent iris disc;

projecting the at least second optical axis vector into the user'"'"'s eyeball; and

determining an intersection of the projected first and projected at least second optical axis vector.

2. The method of claim 1, further comprising:

repeating operations to determine a center point for both eyeballs; and

determining an orientation of the user'"'"'s head within the digital image based the determined eyeball center points and an inter-eyeball distance between the determined eyeball center points.

9. The method of claim 1, wherein determining the ellipse equation characterizing the first apparent iris disc comprises identifying ellipse defining parameters comprising:

an x-axis offset comprising a x-axis distance to the center of the first apparent iris disc in the first digital image plane;

a y-axis offset comprising a y-axis distance to the center of the first apparent iris disc;

a length of the major axis of an ellipse shape of the first apparent iris disc;

a length of the minor axis of the ellipse shape; and

a rotation angle of the ellipse shape with respect to the x-axis.

10. The method of claim 9, wherein calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc comprises:

identifying (x, y) coordinates of the center point of the first apparent iris disc in the first digital image plane using the x-axis offset and y-axis offset;

calculating (x, y) coordinates of at least two edge points of the first apparent iris disc in the first digital image plane based On the determined ellipse equation characterizing the first apparent iris disc; and

calculating a distance in pixels between an origin point (0,0) in the first digital image plane and each of the at least two edge points of the first apparent iris disc, wherein each of the distances comprises a hypotenuse of a triangle formed by the calculated (x, y) coordinates of the respective edge point.

11. The method of claim 10, wherein calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

calculating lengths of rays from an origin point (0,0,0) in the three-dimensional coordinate system to each of at least two points on a surface of the iris disc in the user'"'"'s eyeball that correspond to the at least two edge points, wherein the calculations utilize a spatial angle, between each ray and a z-axis, a function relating ray length to the spatial angle and a field of view of the image sensor'"'"'s optics, and iris radius; and

determining the three-dimensional coordinates of the at least two points on the surface of the iris disc based on the calculated length of the rays.

12. The method of claim 11, wherein calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

determining the three-dimensional coordinates of an iris center point in the user'"'"'s eyeball based on the determined three-dimensional coordinates of the at least two points on the surface of the iris disc; and

calculating the cross product of a first vector and a second vector, wherein the first vector comprises a vector from the iris center point to one of the at least two points on the surface of the iris disc, and wherein the second vector comprises a vector from the iris center point to another of the at least two points on surface of the iris disc.

3. A computing device, comprising:

a digital camera; and

a processor coupled to the camera and configured with processor-executable instructions to perform operations comprising;

obtaining a first digital image of a face of a user of the computing device with the digital camera;

processing the first digital image to determine a position and orientation of an eyeball of the user beneath a surface of the user'"'"'s skin by;

determining an ellipse equation characterizing a first apparent iris disc in a plane of the first digital image by performing a least squares fit of iris-to-sclera boundary pixels in the first digital image; and

determining a spherical center point location of the eyeball using the determined ellipse equation characterizing the first apparent iris disc in the plane of the first digital image; and

translating the position determined and the orientation of the user'"'"'s eyeball into an input command;

wherein determining the spherical center point location of the eyeball comprises;

determining coordinates of a center point of an iris disc in a three-dimensional coordinate system based on the determined ellipse equation characterizing the first apparent iris disc;

calculating a first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc;

projecting the first optical axis vector into the user'"'"'s eyeball;

measuring one eyeball radius distance along the projected first optical axis vector from the surface of the iris disc in the user'"'"'s eyeball;

storing the projected first optical axis vector;

calculating at least a second optical axis vector of the iris disc by;

prompting the user to look in another direction;

obtaining at a second digital image of the user'"'"'s face;

determining an ellipse equation characterizing a second apparent iris disc in a plane of the second digital image; and

determining three-dimensional coordinates of a second center point of the iris disc based on the determined ellipse equation characterizing the second apparent iris disc;

projecting the at least second optical axis vector into the user'"'"'s eyeball; and

determining an intersection of the projected first and projected at least second optical axis vector.

4. The computing device of claim 3, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

repeating operations to determine a center point for both eyeballs; and

determining an orientation of the user'"'"'s head within the digital image based the determined eyeball center points and an inter-eyeball distance between the determined eyeball center points.

13. The computing device of claim 3, wherein the processor is configured with processor-executable instructions to perform operations such that determining the ellipse equation characterizing the first apparent iris disc comprises identifying ellipse defining parameters comprising;

an x-axis offset comprising a x-axis distance to the center of the first apparent iris disc in the first digital image plane;

a y-axis offset comprising a y-axis distance to the center of the first apparent iris disc;

a length of the major axis of an ellipse shape of the first apparent iris disc;

a length of the minor axis of the ellipse shape; and

a rotation angle of the ellipse shape with respect to the x-axis.

14. The computing device of claim 13, wherein the processor is configured with processor-executable instructions to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc comprises:

identifying (x, y) coordinates of the center point of the first apparent iris disc in the first digital image plane using the x-axis offset and y-axis offset;

calculating (x, y) coordinates of at least two edge points of the first apparent iris disc in the first digital image plane based on the determined ellipse equation characterizing the first apparent iris disc; and

calculating a distance in pixels between an origin point (0,0) in the first digital image plane and each of the at least two edge points of the first apparent iris disc, wherein each of the distances comprises a hypotenuse of a triangle formed by the calculated (x, y) coordinates of the respective edge point.

15. The computing device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

calculating lengths of rays from an origin point (0,0,0) in the three-dimensional coordinate system to each of at least two points on a surface of the iris disc in the user'"'"'s eyeball that correspond to the at least two edge points, wherein the calculations utilize a spatial angle between each ray and a z-axis, a function relating ray length to the spatial angle and a field of view of the image sensor'"'"'s optics, and iris radius; and

determining the three-dimensional coordinates of the at least two points on the surface of the iris disc based on the calculated length of the rays.

16. The computing device of claim 15, wherein the processor is configured with processor-executable instructions to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

determining the three-dimensional coordinates of an iris center point in the user'"'"'s eyeball based on the determined three-dimensional coordinates of the at least two points on the surface of the iris disc; and

calculating the cross product of a first vector and a second vector, wherein the first vector comprises a vector from the iris center point to one of the at least two points on the surface of the iris disc, and wherein the second vector comprises a vector from the iris center point to another of the at least two points on surface of the iris disc.

5. A computing device, comprising:

means for obtaining a first digital image of a face of a user of the computing device;

means for processing the first digital image to determine a position and orientation of an eyeball of the user beneath a surface of the user'"'"'s skin comprising

means for determining an ellipse equation characterizing a first apparent iris disc in a plane of the first digital image by performing a least squares fit of iris-to-sclera boundary pixels in the first digital image; and

means for determining a spherical center point location of the eyeball using the determined ellipse equation characterizing the first apparent iris disc in the plane of the first digital image; and

means for translating the position determined and the orientation of the user'"'"'s eyeball into an input command;

wherein means for determining the spherical center point location of the eyeball comprises;

means for determining coordinates of a center point of an iris disc in a three-dimensional coordinate system based on the determined ellipse equation characterizing the first apparent iris disc;

means for calculating a first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc;

means for projecting the first optical axis vector into the user'"'"'s eyeball;

means for measuring one eyeball radius distance along the projected first optical axis vector from the surface of the iris disc in the user'"'"'s eyeball;

means for storing the projected first optical axis vector;

means for calculating at least a second optical axis vector of the iris disc comprising;

means for prompting the user to look in another direction;

means for obtaining at a second digital image of the user'"'"'s face;

means for determining an ellipse equation characterizing a second apparent iris disc in a plane of the second digital image; and

means for determining three-dimensional coordinates of a second center point of the iris disc based on the determined ellipse equation characterizing the second apparent iris disc;

means for projecting the at least second optical axis vector into the user'"'"'s eyeball; and

means for determining an intersection of the projected first and projected at least second optical axis vector.

6. The computing device of claim 5, further comprising:

means for repeating operations to determine a center point for both eyeballs; and

means for determining an orientation of the user'"'"'s head within the digital image based the determined eyeball center points and an inter-eyeball distance between the determined eyeball center points.

17. The computing device of claim 5, wherein means for determining the ellipse equation characterizing the first apparent iris disc comprises means for identifying ellipse defining parameters comprising:

an x-axis offset comprising a x-axis distance to the center of the first apparent iris disc in the first digital image plane;

a y-axis offset comprising a y-axis distance to the center of the first apparent iris disc;

a length of the major axis of an ellipse shape of the first apparent iris disc;

a length of the minor axis of the ellipse shape; and

a rotation angle of the ellipse shape with respect to the x-axis.

18. The computing device of claim 17, wherein means for calculating the first optical axis vector of the iris disc oriented in the thee-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc comprises:

means for identifying (x, y) coordinates of the center point of the first apparent iris disc in the first digital image plane using the x-axis offset and y-axis offset;

means for calculating (x, y) coordinates of at least two edge points of the first apparent iris disc in the first digital image plane based on the determined ellipse equation characterizing the first apparent iris disc; and

means for calculating a distance in pixels between an origin point (0,0) in the first digital image plane and each of the at least two edge points of the first apparent iris disc, wherein each of the distances comprises a hypotenuse of a triangle formed by the calculated (x, y) coordinates of the respective edge point.

19. The computing device of claim 18, wherein means for calculating the first optical axis vector of the iris disc oriented in the thee-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

means for calculating lengths of rays from an origin point (0,0,0) in the three-dimensional coordinate system to each of at least two points on a surface of the iris disc in the user'"'"'s eyeball that correspond to the at least two edge points, wherein the means for calculating utilize a spatial angle between each ray and a z-axis, a function relating ray length to the spatial angle and a field of view of the image sensor'"'"'s optics, and iris radius; and

means for determining the three-dimensional coordinates of the at least two points on the surface of the iris disc based on the calculated length of the rays.

20. The computing device of claim 19, wherein means for calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

means for determining the three-dimensional coordinates of an iris center point in the user'"'"'s eyeball based on the determined three-dimensional coordinates of the at least two points on the surface of the iris disc; and

means for calculating the cross product of a first vector and a second vector, wherein the first vector comprises a vector from the iris center point to one of the at least two points on the surface of the iris disc, and wherein the second vector comprises a vector from the iris center point to another of the at least two points on surface of the iris disc.

7. A non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of a computing device including a digital camera to perform operations comprising:

obtaining a first digital image of a face of a user of the non-transitory processor readable storage medium with the digital camera;

processing the first digital image to determine a position and orientation of an eyeball of the user a surface of the user'"'"'s skin by;

determining an ellipse equation characterizing a first apparent iris disc in a plane of the first digital image by performing a least a squares fit of iris-to-sclera boundary pixels in the first digital image; and

determining a spherical center point location of the eyeball using the determined ellipse equation characterizing the first apparent iris disc in the plane of the first digital image; and

translating the position determined and the orientation of the user'"'"'s eyeball anatomical structure into an input command;

wherein determining the spherical center point location of the eyeball comprises;

determining coordinates of a center point of an iris disc in a three-dimensional coordinate system based on the determined ellipse equation characterizing the first apparent iris disc;

calculating a first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc;

projecting the first optical axis vector into the user'"'"'s eyeball;

measuring one eyeball radius distance along the projected first optical axis vector from the surface of the iris disc in the user'"'"'s eyeball;

storing the projected first optical axis vector;

calculating at least a second optical axis vector of the iris disc by;

prompting the user to look in another direction;

obtaining at a second digital image of the user'"'"'s face;

determining an ellipse equation characterizing a second apparent iris disc in a plane of the second digital image; and

determining three-dimensional coordinates of a second center point of the iris disc based on the determined ellipse equation characterizing the second apparent iris disc;

projecting the at least second optical axis vector into the user'"'"'s eyeball; and

determining an intersection of the projected first and projected at least second optical axis vector.

8. The non-transitory processor-readable storage medium of claim 7, wherein the stored processor-executable instructions are configured to cause the processor to perform operations further comprising:

repeating operations to determine a center point for both eyeballs; and

determining an orientation of the user'"'"'s head within the digital image based the determined eyeball center points and an inter-eyeball distance between the determined eyeball center points.

21. The non-transitory processor-readable storage medium of claim 7, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that determining the ellipse equation characterizing the first apparent iris disc comprises identifying ellipse defining parameters comprising:

an x-axis offset comprising a x-axis distance to the center of the first apparent iris disc in the first digital image plane;

a y-axis offset comprising a y-axis distance to the center of the first apparent iris disc;

a length of the major axis of an ellipse shape of the first apparent iris disc;

a length of the minor axis of the ellipse shape; and

a rotation angle of the ellipse shape with respect to the x-axis.

22. The non-transitory processor-readable storage medium of claim 21, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc comprises:

identifying (x, y) coordinates of the center point of the first apparent iris disc in the first digital image plane using the x-axis offset and y-axis offset;

calculating (x, y) coordinates of at least two edge points of the first apparent iris disc in the first digital image plane based on the determined ellipse equation characterizing the first apparent iris disc; and

calculating a distance in pixels between an origin point (0,0) in the first digital image plane and each of the at least two edge points of the first apparent iris disc, wherein each of the distances comprises a hypotenuse of a triangle formed by the calculated (x, y) coordinates of the respective edge point.

23. The non-transitory processor-readable storage medium of claim 22, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

calculating lengths of rays from an origin point (0,0,0) in the three-dimensional coordinate system to each of at least two points on a surface of the iris disc in the user'"'"'s eyeball that correspond to the at least two edge points, wherein the calculations utilize a spatial angle between each ray and a z-axis, a function relating ray length to the spatial angle and a field of view of the image sensor'"'"'s optics, and iris radius; and

determining the three-dimensional coordinates of the at least two points on the surface of the iris disc based on the calculated length of the rays.

24. The non-transitory processor-readable storage medium of claim 23, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that calculating the first optical axis vector of the iris disc oriented in the three-dimensional coordinate system based on the center point of the iris disc and on the ellipse equation characterizing the first apparent iris disc further comprises:

determining the three-dimensional coordinates of an iris center point in the user'"'"'s eyeball based on the determined three-dimensional coordinates of the at least two points on the surface of the iris disc; and

calculating the cross product of a first vector and a second vector, wherein the first vector comprises a vector from the iris center point to one of the at least two points on the surface of the iris disc, and wherein the second vector comprises a vector from the iris center point to another of the at least two points on surface of the iris disc.