Method and apparatus for associating pupils with subjects

• US 6,539,100 B1
• Filed: 01/27/1999
• Issued: 03/25/2003
• Est. Priority Date: 01/27/1999
• Status: Expired due to Fees

• Current Assignee: IPG Healthcare 501 Limited (Coller IP Capital Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Flickner, Myron Dale, Morimoto, Carlos Hitoshi, Koons, David Bruce, Amir, Arnon
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/238,979
• Time in Patent Office: 1,518 Days
• Field of Search: 382/117, 382/199, 382/128, 382/173, 382/286, 351/206, 351/221, 351/208, 351/210, 434/40, 434/42
• US Class Current: 382/117
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A method and apparatus analyzes a scene to determine which pupils correspond to which subjects. First, a machine-readable representation of the scene, such as a camera image, is generated. Although more detail may be provided, this representation minimally depicts certain visually perceivable characteristics of multiple pupil candidates corresponding to multiple subjects in the scene. A machine such as a computer then examines various features of the pupil candidates. The features under analysis include (1) visually perceivable characteristics of the pupil candidates at one given time (“spatial cues”), and (2) changes in visually perceivable characteristics of the pupil candidates over a sampling period (“temporal cues”). The spatial and temporal cues may be used to identify associated pupil pairs. Some exemplary spatial cues include interocular distance, shape, height, and color of potentially paired pupils. In addition to features of the pupils themselves, spatial cues may also include nearby facial features such as presence of a nose/mouth/eyebrows in predetermined relationship to potentially paired pupils, a similarly colored iris surrounding each of two pupils, skin of similar color nearby, etc. Some exemplary temporal cues include motion or blinking of paired pupils together, etc. With the foregoing examination, each pupil candidate can be associated with a subject in the scene.

Claims

1. A machine-implemented method of associating pupils present in a scene to corresponding subjects, comprising the operations of:

receiving a machine-readable representation of visually perceivable characteristics of multiple pupil candidates corresponding to multiple subjects in a scene;

associating each pupil candidate with one of the multiple subjects in the scene by examining at least one of the following;

(1) one or more of the visually perceivable characteristics of the pupil candidates at one sample time, and (2) any changes in one or more of the visually perceivable characteristics of the pupil candidates during a sampling period; and

providing a machine-readable output representing the association of pupil candidates with subjects in the scene.

2. The method of claim 1, where the representation comprises position coordinates of each pupil candidate in at least two dimensions.

3. The method of claim 1, where the subjects include humans.

4. The method of claim 1, where the subjects include non-humans.

5. The method of claim 1, where the representation comprises a machine-readable image representing the scene.

6. The method of claim 1, where the representation comprises a machine-readable image depicting visual appearance of each pupil candidate.

7. The method of claim 1, further comprising operations of generating the machine-readable representation of the pupil candidates.

8. The method of claim 1, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time includes identifying any associated pairs of pupil candidates using one or more of the following features:

horizontal alignment of two pupil candidates with each other, presence of a predetermined interocular distance between two pupil candidates, similar size of two pupil candidates, and similar shape of two pupil candidates.

9. The method of claim 1, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time further includes identifying any associated pairs of pupil candidates using facial features external to the pupil candidates, the external features including one or more of the following:

presence of a nose in predetermined relationship between two pupil candidates presence of a mouth in predetermined relationship between two pupil candidates, presence of irises of substantially similar color surrounding two pupil candidates, presence of skin of substantially similar color surrounding two pupil candidates.

10. The method of claim 1, where the operation of examining any changes in the visually perceivable characteristics of the pupil candidates during a sampling period further includes identifying any associated pairs of pupil candidates using one or more of the following features:

substantially identical motion of two pupil candidates without relative motion therebetween, and eye blinking indicated by substantially concurrent disappearance and reappearance of the pupil candidates.

11. The method of claim 1, where the representation includes an image representing visual appearance of the scene, and the examining operation further includes identifying any associated pairs of pupil candidates, and the operations further include:

using distance between associated pupils to estimate extent of an associated face in the scene.

12. The method of claim 1, the examining operation further comprising applying a predetermined filtering process to the pupil candidates to eliminate pupil candidates exhibiting certain non-pupil traits.

13. The method of claim 1, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene; and

employing the updated representations to track movement of one or more pupil candidates.

14. The method of claim 1, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to track movement of one or more pupil candidates; and

responsive to losing track of a pupil candidate, repeating the receiving and associating operations.

15. The method of claim 1, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to detect presence of any new pupil candidates; and

responsive to detecting a new pupil candidate, repeating the associating operation.

16. The method of claim 1, the operations further comprising employing the association of pupil candidates with subjects to identify faces in the scene.

17. The method of claim 16, the operations further comprising monitoring changes in the expressions of the faces.

18. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing machine to perform operations for correlating pupils present in a scene to corresponding persons, the operations comprising:

receiving a machine-readable representation of visually perceivable characteristics of multiple pupil candidates corresponding to multiple subjects in a scene;

associating each pupil candidate with one of the multiple subjects in the scene by examining at least one of the following;

(1) one or more of the visually perceivable characteristics of the pupil candidates at one sample time, and (2) any changes in one or more of the visually perceivable characteristics of the pupil candidates during a sampling period; and

providing a machine-readable output representing the association of pupil candidates with subjects in the scene.

19. The medium of claim 18, where the representation comprises position coordinates of each pupil candidate in at least two dimensions.

20. The medium of claim 18, where the representation comprises a machine-readable image representing the scene.

21. The medium of claim 18, where the representation comprises a machine-readable image depicting visual appearance of each pupil candidate.

22. The medium of claim 18, further comprising operations of generating the machine-readable representation of the pupil candidates.

23. The medium of claim 18, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time includes identifying any associated pairs of pupil candidates using one or more of the following features:

horizontal alignment of two pupil candidates with each other, prescence of a predetermined interocular distance between two pupil candidates, similiar size of two pupil candidates, and similiar shape of two pupil candidates.

24. The medium of claim 18, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time further includes identifying any associated pairs of pupil candidates using facial features external to the pupil candidates, the external features including one or more of the following:

presence of a nose in predetermined relationship between two pupil candidates, presence of a mouth in predetermined relationship between two pupil candidates, presence of irises of substantially similar color surrounding two pupil candidates, presence of skin of substantially similar color surrounding two pupil candidates.

25. The medium of claim 18, where the operation of examining any changes in visually perceivable characteristics of the pupil candidates during a sampling period further includes identifying any associated pairs of pupil candidates using one or more of the following features:

substantially identical motion of two pupil candidates without relative motion therebetween, and eye blinking indicated by substantially concurrent disappearance and reappearance of the pupil candidates.

26. The medium of claim 18, where the representation includes an image representing visual appearance of the scene, and the examining operation further includes identifying any associated pairs of pupil candidates, and the operations further include:

using distance between associated pupils to estimate extent of an associated face in the scene.

27. The medium of claim 18, the examining operation further comprising applying a predetermined filtering process to the pupil candidates to eliminate pupil candidates exhibiting prescribed non-pupil characteristics.

28. The medium of claim 18, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene; and

employing the updated representations to track movement of one or more pupil candidates.

29. The medium of claim 18, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to track movement of one or more pupil candidates; and

responsive to losing track of a pupil candidate, repeating the associating operation.

30. The medium of claim 18, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to detect presence of any new pupil candidates; and

responsive to detecting a new pupil candidate, repeating associating operation.

31. The medium of claim 18, the operations further comprising employing the association of pupil candidates with subjects to identify faces in the scene.

32. The method of claim 31, the operations further comprising monitoring changes in expressions of the faces.

33. A logic circuit of multiple interconnected electrically conductive elements configured to perform operations to correlate pupils present in a scene to corresponding persons, the operations comprising:

receiving a machine-readable representation of visually perceivable characteristics of multiple pupil candidates corresponding to multiple subjects in a scene;

associating each pupil candidate with one of the multiple subjects in the scene by examining at least one of the following;

(1) one or more of the visually perceivable characteristics of the pupil candidates at one sample time, and (2) any changes in one or more of the visually perceivable characteristics of the pupil candidates during a sampling period; and

providing a machine-readable output representing the association of pupil candidates with subjects in the scene.

34. The circuit of claim 33, where the representation comprises position coordinates of each pupil candidate in at least two dimensions.

35. The circuit of claim 33, where the representation comprises a machine-readable image representing the scene.

36. The circuit of claim 33, where the representation comprises a machine-readable image depicting visual appearance of each pupil candidate.

37. The circuit of claim 33, further comprising operations of generating the machine-readable representation of the pupil candidates.

38. The circuit of claim 33, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time includes identifying any associated pairs of pupil candidates using one or more of the following features:

horizontal alignment of two pupil candidates with each other, presence of a predetermined interocular distance between two pupil candidates, similar size of two pupil candidates, and similar shape of two pupil candidates.

39. The circuit of claim 33, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time further includes identifying any associated pairs of pupil candidates using facial features external to the pupil candidates, the external features including one or more of the following:

presence of a nose in predetermined relationship between two pupil candidates, presence of a mouth in predetermined relationship between two pupil candidates, presence of irises of substantially similar color surrounding two pupil candidates, presence of ski n of substantially similar color surrounding two pupil candidates.

40. The circuit of claim 33, where the operation of examining any changes in visually perceivable characteristics of the pupil candidates during a sampling period further includes identifying any associated pairs of pupil candidates using one or more of the following features:

substantially identical motion of two pupil candidates without relative motion therebetween, and eye blinking indicated by substantially concurrent disappearance and reappearance of the pupil candidates.

41. The circuit of claim 33, where the representation includes an image representing visual appearance of the scene, and the examining operation further includes identifying any associated pairs of pupil candidates, and the operations further include:

using distance between associated pupils to estimate extent of an associated face in the scene.

42. The circuit of claim 33, the examining operation further comprising applying a predetermined filtering process to the pupil candidates to eliminate pupil candidates exhibiting prescribed non-pupil characteristics.

43. The circuit of claim 33, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene; and

employing the updated representations to track movement of one or more pupil candidates.

44. The circuit of claim 33, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to track movement of one or more pupil candidates; and

responsive to losing track of a pupil candidate, repeating the associating operation.

45. The circuit of claim 33, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to detect presence of any new pupil candidates; and

responsive to detecting a new pupil candidate, repeating the associating operation.

46. The circuit of claim 33, the operations further comprising employing the association of pupil candidates with subjects to identify faces in the scene.

47. The circuit of claim 46, the operations further comprising monitoring changes in expressions of the faces.

48. A human-machine interface, comprising:

a camera to produce one or more machine-readable images of a scene; and

a digital data processing machine, coupled to the camera, and programmed to perform operations to correlate any pupils present in the scene to corresponding persons, the operations comprising;

receiving a machine-readable representation of visually perceivable characteristics of multiple pupil candidates corresponding to multiple subjects in a scene;

associating each pupil candidate with one of the multiple subjects in the scene by examining at least one of the following;

(1) one or more of the visually perceivable characteristics of the pupil candidates at one sample time, and (2) any changes in one or more of the visually perceivable characteristics of the pupil candidates during a sampling period; and

providing a machine-readable output representing the association of pupil candidates with subjects in the scene.

49. The system of claim 48, the camera comprising a video camera.

50. The system of claim 48, further comprising a light source.

51. The system of claim 48, further comprising an output device coupled to the digital data processing machine, the digital data processing machine being programmed to provide the output to the output device.

52. The system of claim 48, where the representation comprises position coordinates of each pupil candidate in at least two dimensions.

53. The system of claim 48, where the representation comprises a machine-readable image representing the scene.

54. The system of claim 48, where the representation comprises a machine-readable image depicting visual appearance of each pupil candidate.

55. The system of claim 48, the digital data processing machine being further programmed to operate the camera to generate the machine-readable representation by creating a machine-readable image of the pupil candidates.

56. The system of claim 48, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time includes identifying any associated pairs of pupil candidates using one or more of the following features:

horizontal alignment of two pupil candidates with each other, presence of a predetermined interocular distance between two pupil candidates, similar size of two pupil candidates, and similar shape of two pupil candidates.

57. The system of claim 48, where the operation of examining the visually perceivable characteristics of the pupil candidates at one sample time further includes identifying any associated pairs of pupil candidates using facial features external to the pupil candidates, the external features including one or more of the following:

presence of a nose in predetermined relationship between two pupil candidates, presence of a mouth in predetermined relationship between two pupil candidates, presence of substantially similar iris surrounding two pupil candidates, presence of skin of substantially similar color surrounding two pupil candidates.

58. The system of claim 48, where the operation of examining any changes in visually perceivable characteristics of the pupil candidates during a sampling period further includes identifying any associated pairs of pupil candidates using one or more of the following features:

substantially identical motion of two pupil candidates without any relative motion, therebetween, and eye blinking indicated by substantially concurrent disappearance and reappearance of the pupil candidates.

59. The system of claim 48, where the representation includes an image representing visual appearance of the scene, and the examining operation further includes identifying any associated pairs of pupil candidates, and the operations further include:

using distance between associated pupils to estimate extent of an associated face in the scene.

60. The system of claim 48, the examining operation further comprising applying a predetermined filtering process to the pupil candidates to eliminate pupil candidates exhibiting prescribed non-pupil characteristics.

61. The system of claim 48, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene; and

employing the updated representations to track movement of one or more pupil candidates.

62. The system of claim 48, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to track movement of one or more pupil candidates; and

responsive to losing track of a pupil candidate, repeating the associating operation.

63. The system of claim 48, the operations further comprising:

receiving updated machine-readable representations of the visually perceivable characteristics of pupil candidates in the scene;

employing the updated representations to detect presence of any new pupil candidates; and

responsive to detecting a new pupil candidate, repeating the associating operation.

64. The system of claim 48, the operations further comprising employing the association of pupil candidates with subjects to identify faces in the scene.

65. The method of claim 64, the operations further comprising monitoring changes in expressions of the faces.