Device for detecting line of sight

• US 5,543,887 A
• Filed: 10/27/1993
• Issued: 08/06/1996
• Est. Priority Date: 10/29/1992
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Akashi, Akira
• Primary Examiner(s): Adams, Russell E.
• Application Number:
  US08/141,613
• Time in Patent Office: 1,014 Days
• Field of Search: 354/410, 354/400, 354/402, 354/195.1, 354/62
• US Class Current: 396/51
• CPC Class Codes: G03B 13/02 Viewfinders
• CPC Class Codes: G03B 2213/025 Sightline detection

Abstract

A device is provided device for detecting the line of sight of an observer, based on the positions of the Purkinje'"'"'s image and of the center of the pupil circle. The device is capable, in determining the pupil circle from the pupil edge data collected from the image sensor, of preventing erroneous estimation of the pupil circle resulting from false noise data, by partially excluding or adding the edge data in the estimation process of the pupil circle and evaluating the reliability of the obtained estimated circle.

Claims

1. A device for detecting a shape of a region of an eyeball, said device comprising:

a) detection means for detecting boundary position information for a boundary between a first region and a second region of the eyeball, said boundary divided into areas that each include plural boundary position information, wherein the boundary between the first and second regions is substantially circular;

b) selection means for selecting at least one area among the plurality of areas of the boundary detected by said detection means based on a predetermined threshold; and

c) calculation means for estimating an external shape of the first region by utilizing the boundary position information for the area of the boundary selected by said selection means.

2. A device according to claim 1, wherein said first organism is a pupil, and said second organism is an iris.

3. A device according to claim 1, wherein said detection means is adapted to detect said boundary in directionally different plural groups.

4. A device according to claim 3, wherein said first organism is a pupil, and said second organism is an iris.

5. A device according to claim 3, wherein said calculation means is adapted to estimate said external shape of said first organism, by preferentially employing, among said boundary position information of said plural groups detected by said detection means, the boundary position information of the horizontal direction.

6. A device according to claim 4, wherein said calculation means is adapted to estimate said external shape of said pupil, by preferentially employing, among said boundary position information of said plural groups detected by said detection means, the boundary position information of the horizontal direction.

7. A device according to claim 1, wherein said calculation means is adapted to estimate said external shape by the least square method.

8. A device according to claim 6, wherein said calculation means is adapted to estimate said external shape by the least square method.

9. A device according to claim 1, wherein said detection means includes light receiving means for receiving an eyeball image, and is adapted to detect said boundary between said first and second organisms in said plural groups, by the output of said light receiving means.

10. A device according to claim 4, wherein said detection means includes light receiving means for receiving an eyeball image, and is adapted to detect said boundary between said pupil and said iris in said plural groups, by the output of said light receiving means.

11. A device according to claim 1, wherein said calculation means is adapted to estimate the outline of said first organism.

12. A device according to claim 2, wherein said calculation means is adapted to estimate the outline of said pupil.

13. A device according to claim 5, wherein said calculation means is adapted to estimate the outline of said first organism.

14. A device according to claim 6, wherein said calculation means is adapted to estimate the outline of said pupil.

15. A device according to claim 1, further comprising:

sight line detecting means for detecting the line of sight of said eyeball, based on the information on the external shape of said pupil, estimated by said calculation means.

16. A device according to claim 2, further comprising:

sight line detecting means for detecting the line of sight of said eyeball, based on the information on the external shape of said pupil, estimated by said calculation means.

17. A device according to claim 5, further comprising:

sight line detecting means for detecting the line of sight of said eyeball, based on the information on the external shape of said first organism, estimated by said calculation means.

18. A device according to claim 6, further comprising:

sight line detecting means for detecting the line of sight of said eyeball, based on the information on the external shape of said pupil estimated by said calculation means.

19. A device according to claim 15, further comprising:

display means controlled according to the sight line information obtained by said sight line detecting means.

20. A device according to claim 16, further comprising:

display means controlled according to the sight line information obtained by said sight line detecting means.

21. A device according to claim 15, further comprising:

electromechanical conversion means controlled according to the information on the line of sight, obtained by said sight line detecting means.

22. A device according to claim 17, further comprising:

electromechanical conversion means controlled according to the information on the line of sight, obtained by said sight line detecting means.

23. A device for detecting a shape of a region of an eyeball, said device comprising:

a) detection means for detecting data indicative of a boundary between a first region and a second region of the eyeball, wherein the boundary is substantially circular;

b) first calculation means for calculating a respective estimated error of each of said plurality of boundary data;

c) determining means for determining which boundary data has the greatest effect on an overall estimated error associated with an estimated shape of the first region; and

d) second calculation means for calculating an estimated external shape of the first region utilizing a portion of the boundary data excluding the boundary data having the greatest effect on the overall estimated error from said boundary data.

24. A device according to claim 23, wherein said determining means comprises means for selectively excluding different portions of the boundary data to form respective subsets of boundary data, means for calculating a plurality of shape estimates of the first region respectively based on the subsets of boundary data, means for determining an estimated error for each shape estimate, and means for selecting a subset of boundary data corresponding to the shape estimate having the lowest estimated error.

25. A device according to claim 24, wherein said means for calculating a plurality of shape estimates comprises means for reducing the estimated error for shape estimates that are based on a subset of boundary data including data from predetermined regions of the boundary.

26. A device according to claim 24, wherein said means for excluding selected portions of boundary data comprises means for excluding a portion of boundary data corresponding to one of an upper region, lower region, left region, and right region of the boundary.

27. A device according to claim 24, wherein the first region of the eyeball is a pupil region and the second region of the eyeball is an iris region.

28. A device according to claim 24, wherein said means for calculating the plurality of shape estimates uses a least square method.

29. A device according to claim 24, wherein said determining means further comprises means for selecting an excluded portion of boundary data corresponding to the selected subset of boundary data and means for designating the selected excluded portion as the boundary data having the greatest effect on the overall estimated error.

30. A device according to claim 24, wherein said determining means further comprises means for selecting an excluded portion of boundary data corresponding to the selected subset of boundary data, means for determining whether each data value in the excluded portion of boundary data satisfies a predetermined condition, and means for designating the data values that fail to satisfy the predetermined condition as the boundary data having the greatest effect on the overall estimated error.

31. A device according to claim 30, wherein said determining means further comprises means for selecting a portion of the selected subset of boundary data, means for determining whether each data value in the selected portion of the selected subset satisfies a different predetermined condition, and means for designating the data values that satisfy the predetermined condition and the data values in the excluded portion of boundary data as the boundary data having the greatest effect on the overall estimated error.

32. A device for detecting a line of sight of an eyeball, comprising:

a) detection means for detecting data indicative of a boundary between a first region and a second region of the eyeball, wherein the boundary is substantially circular;

b) first calculation means for calculating a respective estimated error of each of said plurality of boundary data;

c) estimating means for estimating an external shape of the first region by selectively utilizing a portion of the boundary data so that an overall estimated error associated with the shape estimate is below a predetermined threshold; and

d) second calculation means for calculating a line of sight using the shape estimated by said estimating means.

33. A device according to claim 32, wherein said estimating means comprises determining means for determining the boundary data having the greatest effect on the overall estimated error and means for calculating a shape estimate utilizing the boundary data excluding the boundary data having the greatest effect on the overall estimated error.

34. A device according to claim 33, wherein said determining means comprises means for selectively excluding different portions of the boundary data to form respective subsets of boundary data, means for calculating a plurality of shape estimates of the first region respectively based on the subsets of boundary data, means for determining an estimated error for each shape estimate, and means for selecting a subset of boundary data corresponding to the shape estimate having the lowest estimated error.

35. A device according to claim 34, wherein said means for calculating a plurality of shape estimates comprises means for reducing the estimated error for shape estimates that are based on a subset of boundary data including predetermined portions of data.

36. A device according to claim 34, wherein said means for excluding selected portions of boundary data comprises means for excluding a portion of boundary data corresponding to one of an upper region, lower region, left region, and right region of the boundary.

37. A device according to claim 34, wherein the first region of the eyeball is a pupil region and the second region of the eyeball is an iris region.

38. A device according to claim 34, wherein said means for calculating the plurality of shape estimates uses a least square re method.

39. A device according to claim 34, wherein said determining means further comprises means for selecting an excluded portion of boundary data corresponding to the selected subset of boundary data and means for designating the selected excluded portion as the boundary data having the greatest effect on the overall estimated error.

40. A device according to claim 34, wherein said determining means further comprises means for selecting an excluded portion of boundary data corresponding to the selected subset of boundary data, means for determining whether each data value in the excluded portion of boundary data satisfies a predetermined condition, and means for designating the data values that fail to satisfy the predetermined condition as the boundary data having the greatest effect on the overall estimated error.

41. A device according to claim 40, wherein said determining means further comprises means for selecting a portion of the selected subset of boundary data, means for determining whether each data value in the selected portion of the selected subset satisfies a different predetermined condition, and means for designating the data values that satisfy the predetermined condition and the data values in the excluded portion of boundary data as the boundary data having the greatest effect on the overall estimated error.

42. A device according to claim 32, further comprising means for detecting a position of an image reflected from the eyeball, wherein said calculation means comprises means for calculating the line of sight based on the shape estimated by said estimating means and the detected position of the reflected image.

43. A device according to claim 32, wherein said means for detecting a reflected image comprises means for detecting a Purkinje'"'"'s image.

44. A device for detecting a shape of a region of an eyeball, comprising:

a) detection means for detecting a plurality of boundary position information between a first region and a second region of the eyeball, wherein a boundary between the first region and the second region is substantially circular;

b) grouping means for dividing the detected plural boundary position information into a plurality of groups, each group consisting of a plurality of boundary position information;

c) selection means for selecting a group which satisfies a predetermined condition from among the plural groups; and

d) calculation means for estimating an external shape of the first region only based on the plurality of boundary position information included in the group selected by said selection means.

45. A device according to claim 44, wherein the first region is a pupil portion of the eyeball and the second region is an iris portion of the eyeball.

46. A device according to claim 44, wherein said grouping means divides the boundary between the first region and the second region into a horizontal direction and a vertical direction, and performs group dividing.

47. A device according to claim 46, wherein the predetermined condition of said selection means is set in order to select by preferentially employing a group in the horizontal direction among the plural groups.

48. A device according to claim 44, wherein said calculation means estimates the external shape by utilizing a least square method.

49. A device according to claim 44, wherein said detection means have light receiving means for receiving an eyeball image and detects the boundary position information dividing the first region and the second region according to an output from the light receiving means.

50. A device according to claim 44, wherein said calculation means estimates an outline of a pupil portion of the eyeball.

51. A device according to claim 45, wherein said calculation means estimates an outline of the pupil portion.

52. A device according to claim 44, further comprising line of sight detection means for detecting line of sight of the eyeball based on information of the external shape of the first region estimated by said calculation means.

53. A device according to claim 52, further comprising display means to be controlled according to line of sight information obtained by said line of sight detection means.

54. A device according to claim 52, further comprising electro-mechanical energy conversion means to be controlled by line of sight information obtained by said line of sight detection means.

55. A device for detecting a shape of a region of an eyeball, comprising:

a) detection means for detecting plural boundary position information between a first region and a second region of the eyeball, wherein a boundary between the first region and the second region is substantially circular;

b) grouping means for dividing the detected plural boundary position information into a plurality of groups, each group consisting of plural boundary position information;

c) first calculation means for estimating an external shape of the first region by selectively excluding one of the groups among the plurality of groups, said first calculation means changing the group to be excluded and performing the calculation a plurality of times; and

d) first evaluation means for evaluating a reliability of the respective external shape of the first region estimated by said first calculation means for each calculation;

wherein a shape of the region of the eyeball is detected by utilizing at least the plural boundary position information included in the plural groups which said first evaluation means evaluates as having the highest reliability.

56. A device according to claim 55, wherein the first region is a pupil portion of the eyeball and the second region is an iris portion of the eyeball.

57. A device according to claim 55, wherein said calculation means estimates the external shape by utilizing a least square method.

58. A device according to claim 55, wherein said calculation means estimates an outline of a pupil portion of the eyeball.

59. A device according to claim 56, wherein said calculation means estimates an outline of the pupil portion.

60. A device according to claim 55, wherein said detection means have light receiving means for receiving an eyeball image and detects the boundary position information between the first region and the second region according to an output from the light receiving means.

61. A device according to claim 55, further comprising second evaluation means for evaluating again the reliability of the respective plural boundary position information included in the excluded group when said first evaluation means evaluates the highest reliability;

and

second calculation means for estimating again the external shape of the first region by utilizing the plural boundary position information of the plural groups of which said first evaluation means evaluates the highest reliability and by utilizing the boundary position information of the excluded groups of which said second evaluation means evaluates high reliability among the excluded groups.

62. A device according to claim 61, further comprising line of sight detection means for detecting line of sight of the eyeball based on the external shape of the first region estimated by said second calculation means.

63. A device according to claim 62, further comprising display means to be controlled according to line of sight information obtained by said line of sight detection means.

64. A device according to claim 62, further comprising electro-mechanical energy conversion means to be controlled according to the line of sight information obtained by said line of sight detection means.

65. A device for detecting a shape of a region of an eyeball, comprising:

a) detection means for detecting plural boundary position information between a first region and a second region of the eyeball, wherein a boundary between the first region and the second region is substantially circular;

b) calculation means for estimating an external shape of the first region based on the plural boundary position information;

c) determining means for determining a reliability of the plural boundary position information based on whether the respective boundary position information included in the plural boundary position information is located at the same circumference; and

d) changing means for changing determination criterion of said determining means based on a size of the external shape of the first region estimated by said calculation means.

66. A device according to claim 65, wherein the first region is a pupil portion and the second region is an iris portion.

67. A device according to claim 65, wherein said calculation means estimates the external shape by utilizing a least square method.

68. A device according to claim 65, wherein said detection means divide the plural boundary position information into plural groups and detects the information, and said determination means determines a reliability of each group.

69. A device according to claim 68, wherein said calculation means excludes the group of which reliability is determined as low by said determination means and estimates again the external shape of the first region.

70. A device according to claim 65, wherein said detection means have light receiving means for receiving an eyeball image and detects the boundary position information between the first region and the second region according to an output from the light receiving means.

71. A device according to claim 65, wherein said second calculation means estimates an outline of a pupil region of the eyeball.

72. A device according to claim 66, wherein said second calculation means estimates an outline of the pupil portion.

73. A device according to claim 65, further comprising line of sight detection means for detecting line of sight of the eyeball based on information of the external shape of the first region estimated by said calculation means.

74. A device according to claim 73, further comprising display means to be controlled according to line of sight information obtained by said line of sight detection means.

75. A device according to claim 73, further comprising electro-mechanical energy conversion means to be controlled according to line of sight information obtained by said line of sight detection means.