Eye tracking using image data
First Claim
1. A method of processing data in a processing apparatus to perform eye tracking across a sequence of images of a moving head, the method comprising processing each of at least some of the images in the sequence by:
- identifying pixels in the image corresponding to search regions defined relative to a three-dimensional computer model representative of at least part of a head in a three-dimensional space, the search regions being defined with different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model, and the identification of pixels being performed in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image;
determining image values of the identified pixels;
generating at least one respective representative image value for each search region using the determined image values; and
determining eye positions in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search regions.
2 Assignments
0 Petitions
Accused Products
Abstract
In a computer processing apparatus 8, frames of image data received from a camera 4 are processed to track the eyes of a user 2 in each image. A 3D computer model of a head is stored, and search regions are defined in the 3D space corresponding to the eyes and eyebrows. For each image, pixels within the projection of the search regions from the 3D space to the 2D image space are sampled to determine a representative intensity value for each of the search regions. Positions for the eyes in the 3D space are then calculated based on the determined values. The 3D computer model and search bands are moved within the 3D space to align the eyes with the calculated eye positions. In this way, when the next image is processed, the search bands project into the image from a head configuration determined from the previous image. This facilitates reliable and accurate eye tracking.
60 Citations
61 Claims
-
1. A method of processing data in a processing apparatus to perform eye tracking across a sequence of images of a moving head, the method comprising processing each of at least some of the images in the sequence by:
-
identifying pixels in the image corresponding to search regions defined relative to a three-dimensional computer model representative of at least part of a head in a three-dimensional space, the search regions being defined with different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model, and the identification of pixels being performed in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image; determining image values of the identified pixels; generating at least one respective representative image value for each search region using the determined image values; and determining eye positions in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search regions. - View Dependent Claims (2, 3, 4, 5, 6, 33, 34, 35, 36, 37, 38, 48, 49)
-
-
7. A method of processing data in a processing apparatus defining a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the method comprising processing each of at least some of the images in the sequence by:
-
determining image values of pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; generating at least one respective representative image value for each search band using the determined image values; determining eye positions in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search bands; and moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
8. A method of processing data in a processing apparatus to perform eye tracking across a sequence of images of a moving head, the method comprising processing each of at least some of the images in the sequence by:
-
identifying pixels in the image corresponding to search regions defined relative to a three-dimensional computer model representative of at least part of a head in a three-dimensional space, the search regions being defined with different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model, and the identification of pixels being performed in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image; determining image values of the identified pixels; generating at least one respective representative image value for each search region using the determined image values; generating an image value profile from the generated representative image values representative of the variation in the representative image values with variation in search region position; and comparing the generated image value profile with a reference profile to determine eye positions in the three-dimensional space. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A method of processing data in a processing apparatus defining a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the method comprising processing each of at least some of the images in the sequence by:
-
determining image values of pixels located in the image at positions defined by the projection of search bands or positions therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; generating at least one representative image value for each search band using the determined pixel image values; comparing the generated representative image values for different search bands with reference image values representative of image values for an eye and an eyebrow to calculate positions for the eyes in the three-dimensional space; and moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed. - View Dependent Claims (58)
-
-
21. A method of processing data in a processing apparatus to perform eye tracking across a sequence of images of a moving head, the method comprising processing each of at least some of the images in the sequence by:
-
identifying pixels in the image in dependence upon search regions defined relative to a three-dimensional computer model representative of at least part of a head in a three-dimensional space, the search regions dividing into parts a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model, and in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image; determining image values of the identified pixels; generating at least one respective representative image value for each search region using the determined image values; determining which of the search regions project onto dark regions in the image by comparing the generated representative image values of different search regions; and determining eye positions in the three-dimensional space in dependence upon the position of at least one search region determined to project onto a dark region in the image. - View Dependent Claims (22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 56, 57)
-
-
32. A method of processing data in a processing apparatus defining a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the method comprising processing each of least some of the images in the sequence by:
-
determining the image values of pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space so as to divide into parts a portion of the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; calculating a respective representative image value for each search band using the determined pixel image values; calculating positions for the eyes in the three-dimensional space by comparing the representative image values for different search bands to identify at least one search band corresponding to a dark region in the image; and moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
39. Image processing apparatus for processing image data of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
a data store configured to store data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions having different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; a pixel identifier operable to identify pixels in an image corresponding to the search regions, the pixel identifier being operable to identify pixels in dependence upon a transformation between the three-dimensional space and the two-dimensional image; an image value determinator operable to determine image values of the identified pixels; a representative value calculator operable to generate at least one respective representative image value for each search region using the determined image values; and an eye position determinator operable to determine eye positions in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search regions. - View Dependent Claims (40)
-
-
41. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
a pixel identifier operable to identify pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; an image value determinator operable to determine image values of the identified pixels; a representative value calculator operable to generate at least one respective representative image value for each search band using the determined image values; an eye position calculator operable to calculate positions for the eyes in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search bands; and a search band mover operable to move the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed. - View Dependent Claims (61)
-
-
42. Apparatus for processing image data of each of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
a data store configured to store data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions having different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; a pixel identifier operable to identify pixels in the image corresponding to the search regions, the pixel identifier being operable to identify pixels in dependence upon a transformation between the three-dimensional space and the two-dimensional image; an image value determinator operable to determine image values of the identified pixels; a representative value calculator operable to generate at least one respective representative image value for each search region using the determined image values; a profile generator operable to generate an image value profile from the generated representative image values representative of the variation in the representative image values with variation in search region position; and a profile comparer operable to compare the generated image value profile with a reference profile to determine eye positions in the three-dimensional space. - View Dependent Claims (43)
-
-
44. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
a pixel identifier operable to identify pixels located in the image at positions defined by the projection of search bands or positions therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; an image value determinator operable to determine image values of the identified pixels; a value generator operable to generate at least one representative image value for each search band using the determined pixel image values; a comparer operable to compare the generated representative image values for different search bands with reference image values representative of image values for an eye and an eyebrow to calculate positions for the eyes in the three-dimensional space; and a search band mover operable to move the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
45. Apparatus for processing image data of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
a data store configured to store data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions dividing into parts a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; a pixel identifier operable to identify pixels in the image in dependence upon the search regions and in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image; an image value determinator operable to determine image values of the identified pixels; a representative value calculator operable to generate at least one respective representative image value for each search region using the determined image values; a search region selector operable to determine which of the search regions project onto dark regions in the image by comparing the generated representative image values of different search regions; and an eye position determinator operable to determine eye positions in the three-dimensional space in dependence upon the position of at least one search region determined to project onto a dark region in the image. - View Dependent Claims (46, 59, 60)
-
-
47. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
a pixel identifier operable to identify pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space so as to divide into parts a portion of the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; an image value determinator operable to determine image values of the identified pixels; a value calculator operable to calculate a respective representative image value for each search band using the determined pixel image values; an eye position calculator operable to calculate positions for the eyes in the three-dimensional space by comparing the representative image values for different search bands to identify at least one search band corresponding to a dark region in the image; and a search band mover operable to move the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
50. Image processing apparatus for processing image data of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
means for storing data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions having different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; pixel identifying means for identifying pixels in an image corresponding to the search regions, the pixel identifying means being operable to identify pixels in dependence upon a transformation between the three-dimensional space and the two-dimensional image; means for determining image values of the identified pixels; representative value calculating means for generating at least one respective representative image value for each search region using the determined image values; and means for determining eye positions in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search regions.
-
-
51. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
means for identifying pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; means for determining image values of the identified pixels; representative value calculating means for generating at least one respective representative image value for each search band using the determined image values; means for calculating positions for the eyes in the three-dimensional space in dependence upon the relationship between the generated representative image values for different search bands; and means for moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
52. Apparatus for processing image data of each of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
means for storing data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions having different positions in a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; pixel identifying means for identifying pixels in the image corresponding to the search regions, the pixel identifying means being operable to identify pixels in dependence upon a transformation between the three-dimensional space and the two-dimensional image; means for determining image values of the identified pixels; representative value calculating means for generating at least one respective representative image value for each search region using the determined image values; means for generating an image value profile from the generated representative image values representative of the variation in the representative image values with variation in search region position; and means for comparing the generated image value profile with a reference profile to determine eye positions in the three-dimensional space.
-
-
53. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
means for identifying pixels located in the image at positions defined by the projection of search bands or positions therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; means for determining image values of the identified pixels; means for generating at least one representative image value for each search band using the determined pixel image values; means for comparing the generated representative image values for different search bands with reference image values representative of image values for an eye and an eyebrow to calculate positions for the eyes in the three-dimensional space; and means for moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
-
54. Apparatus for processing image data of at least some images in a sequence of images of a moving head to perform eye tracking across the sequence, the apparatus comprising:
-
means for storing data defining a three-dimensional computer model representative of at least part of a head in a three-dimensional space, and data defining search regions dividing into parts a portion of the three-dimensional space corresponding to the eyes and eyebrows of the model; pixel identifying means for identifying pixels in the image in dependence upon the search regions and in dependence upon data defining a transformation between the three-dimensional space and the two-dimensional image; means for determining image values of the identified pixels; representative value calculating means for generating at least one respective representative image value for each search region using the determined image values; search region selection means for determining which of the search regions project onto dark regions in the image by comparing the generated representative image values of different search regions; and eye position determining means for determining eye positions in the three-dimensional space in dependence upon the positions of at least one search region determined to project onto a dark region in the image.
-
-
55. Apparatus for processing image data of each of at least some images in a sequence of images of a person as the person'"'"'s head moves, to track the eyes of the person in the images, the apparatus comprising:
-
means for identifying pixels located in the image at positions defined by the projection of search bands or points therein from a three-dimensional space into the two-dimensional image, wherein the search bands have different positions in the three-dimensional space so as to divide into parts a portion of the three-dimensional space, and wherein the projection is dependent upon the position and orientation of the search bands in the three-dimensional space resulting from the processing of the previous image in the sequence to be processed; means for determining image values of the identified pixels; means for calculating a respective representative image value for each search band using the determined pixel image values; means for calculating positions for the eyes in the three-dimensional space by comparing the representative image values for different search bands to identify at least one search band corresponding to a dark region in the image; and means for moving the search bands within the three-dimensional space in dependence upon the calculated positions for the eyes, so as to move the search bands for subsequent processing of the next image in the sequence to be processed.
-
Specification