System and method for detecting an occupant and head pose using stereo detectors

• US 20070183651A1
• Filed: 11/18/2004
• Published: 08/09/2007
• Est. Priority Date: 11/21/2003
• Status: Active Grant

• Current Assignee: Siemens Healthcare GMBH (Siemens AG)
• Original Assignee: Siemens Corp. (Siemens AG)
• Inventors: Zhu, Ying, Comaniciu, Dorin, Xie, Binglong, Kohler, Thorsten
• Granted Patent: US 7,508,979 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A system and method for detecting an occupant and head pose using stereo detectors is disclosed. In the training stage, pairs of images taken simultaneously from a pair of stereo cameras are received and components in each pair of images are identified. Features are associated with each component and a value is associated with each feature. Among them, a subset of features with best discriminative capabilities is selected and forms a strong classifier for a given component in a pair of images. In the detection stage, the strong classifiers are used in the incoming image pairs to detect components. Identified components are used to detect occupants and head pose of the occupants. Stereo detection is combined with stereo matching in a systematic way to improve occupant detection and localization.

Claims

1. A method for detecting head pose of an individual comprises the steps of:

capturing a pair of images of a person from a pair of stereo cameras;

scanning the pair of images simultaneously to locate at least one component in the pair of images;

computing three dimensional (3D) locations for each of the at least one components; and

estimating head pose using the 3D locations.

2. The method of claim 1 wherein the pair of stereo cameras are positioned at a known angle and position relative to one another.

3. The method of claim 1 wherein the step of scanning the pair of images further comprises using a trained joint detector to scan the pair of images.

4. The method of claim 1 wherein the step of computing 3D locations comprises using triangulation.

5. The method of claim 1 wherein head pose is detected of an occupant of a vehicle.

6. The method of claim 5 wherein the occupant is a driver of the vehicle.

7. The method of claim 1 wherein the step of scanning the pair of images further comprises the steps of:

using a window to analyze a section of each of the images;

calculating a predetermined set of features and a classification score for the window;

determining if the classification score is above a predefined threshold;

if the classification score is above the predefined threshold, determining that a component has been located; and

if a feature score is below the predefined threshold, sliding the window in the image to an adjacent position and calculating a feature score in the adjacent position.

8. The method of claim 1 wherein the at least one component is a left eye.

9. The method of claim 1 wherein the at least one component is a right eye.

10. The method of claim 1 wherein the at least one component is a nose.

11. The method of claim 1 wherein the at least one component is right ear.

12. The method of claim 1 wherein the at least one component is a left ear.

13. The method of claim 1 wherein the at least one component is a mouth.

14. The method of claim 1 wherein the at least one component is a head.

15. The method of claim 1 wherein the at least one component is a shoulder.

16. The method of claim 1 wherein the computing step further comprises using disparity maps to determine the location of the at least one component.

26. The method of claim 15 wherein the step of determining a classification function further comprises combining the strengths of the subset of features.

17. A method for training stereo detectors for use in real-time head pose detection comprises the steps of:

collecting pairs of images of an individual'"'"'s head, each pair of images depicting a different head pose;

collecting pairs of images of additional individuals'"'"' heads, each pair of images for each additional individual depicting different head poses;

identifying and labeling a set of components for each pair of images;

indexing data pertaining to each identified component;

storing the indexed data in a database;

defining a plurality of features that are associated with each identified component in stereo image pairs; and

for each component, determining a subset of features that best discriminate positive component examples.

for each component, determining a classification function which generates a classification score from the set of discriminative features.

18. The method of claim 17 wherein each head pose is defined by a yaw angle.

19. The method of claim 18 wherein each head pose is defined by a pitch angle.

20. The method of claim 18 wherein each head pose is defined by a roll angle.

21. The method of claim 17 wherein a same set of head poses is received for each individual imaged.

22. The method of claim 17 wherein the set of components are identified manually.

23. The method of claim 17 further comprising the step of:

collecting pairs of non-components.

24. The method of claim 17 wherein each feature is assigned a value that determines strength of the feature.

25. The method of claim 17 wherein the step of determining a subset of features uses boosting.

27. The method of claim 17 wherein the step of training component detectors uses disparity maps.

28. A method for testing a pair of images to identify stereo components in the images comprising the steps of:

using a window of a predetermined size to sample a portion of a first image;

calculating a subset of discriminative features and a classification score for the window;

if the classification score is above a predetermined threshold, determining presence of a component in the window; and

sliding the window to an adjacent position and calculating the classification score for the window in the adjacent position.

29. The method of claim 28 wherein the window scans through all of an image region of interest by sliding to adjacent positions relative to a previous positioning of the window.

30. The method of claim 28 wherein if the classification score is above a predetermined threshold, the method further comprising the steps of:

applying a second window to the second image;

calculating a classification score for the second window;

if the classification score is above a predetermined threshold, determining the presence of a component in the second window; and

if the feature score is below a predetermined threshold, sliding the second window to an adjacent position and calculating the classification score for the second window in the adjacent position.

31. The method of claim 30 wherein placement of the second window is constrained based on the location of the first window in the first image in which the component was identified.

32. The method of claim 28 wherein the set of stereo components corresponds to a left eye.

33. The method of claim 28 wherein the set of stereo components corresponds to a right eye.

34. The method of claim 28 wherein the set of stereo components corresponds to a nose.

35. The method of claim 28 wherein the set of stereo components corresponds to a right ear.

36. The method of claim 28 wherein the set of stereo components corresponds to a left ear.

37. The method of claim 28 wherein the set of stereo components corresponds to a mouth.

38. The method of claim 28 wherein the set of stereo components corresponds to a head

39. The method of claim 28 wherein the set of stereo components corresponds to a shoulder.

40. A method for detecting an occupant of a vehicle comprising the steps of:

a). capturing a pair of images of a person from a pair of stereo cameras;

b). applying a head detector and shoulder detector to identify head and shoulder from one image;

c). locating positions with high classification scores in a first image of the pair of images;

d). if the classification score at a location in the first image is above a predetermined threshold, locating a corresponding region in a second image of the pair of images;

e). determining if a head and shoulder component is detected in the second image;

f). if a head and shoulder component is detected at corresponding positions in stereo image pair, using the head and shoulder component as an initial matching point for stereo matching;

g). performing stereo matching on the head and shoulder component to obtain correspondence information;

h). feeding back the correspondence information to a detection module;

i). using the correspondence information to refine the detection region in the pair of images and to generate new detection information;

j). repeating steps e), f), g), h) and until convergence between the correspondence information and detection information is obtained; and

k). determining if an occupant is present based on the detection information.

41. The method of claim 40 wherein said step of using the head and shoulder component for stereo matching further comprises the step of:

producing a disparity map for the head and shoulder component.

42. The method of claim 40 wherein the correspondence information induces constraints on the detection module.

43. The method of claim 40 further comprising the step of:

if a occupant is detected, using the detection information to determine body position and pose.

44. A system for detecting head pose of an individual comprises:

a pair of camera for capturing images of a person, the pair of cameras being situated in a known position relative to one another;

a processor associated with the pair of cameras, the processor performing the following steps;

i). simultaneously scanning the images captured by the pair of cameras to locate at least one component in the images;

ii). computing three dimensional (3D) locations for each of the at least one components; and

iii). estimating head pose using the 3D locations.

45. The system of claim 44 wherein the processor further comprises a trained joint detector to scan the images captured by the pair of cameras.

46. The system of claim 44 wherein the step of computing 3D locations comprises using triangulation.

47. The system of claim 44 wherein head pose is detected of an occupant of a vehicle.

48. The system of claim 47 wherein the occupant is a driver of the vehicle.

49. The system of claim 44 wherein the step of scanning the pair of images further comprises the steps of:

using a window to analyze a section of each of the images;

calculating a feature score for the window;

determining if the feature score is above a predefined threshold;

if the feature score is above the predefined threshold, determining that a component has been located; and

if a feature score is below the predefined threshold, sliding the window in the image to an adjacent position and calculating a feature score in the adjacent position.

50. The system of claim 44 wherein the at least one component is a left eye.

51. The system of claim 44 wherein the at least one component is a right eye.

52. The system of claim 44 wherein the at least one component is a nose.

53. The system of claim 44 wherein the at least one component is right ear.

54. The system of claim 44 wherein the at least one component is a left ear.

55. The system of claim 44 wherein the computing step further comprises using disparity maps to determine the location of the at least one component.

56. A database system for use in real-time head pose detection comprises:

means for receiving pairs of images of an individual'"'"'s head, each pair of images depicting a different head pose;

means for receiving pairs of images of additional individuals'"'"' heads, each pair of images for each additional individual depicting different head poses;

means for identifying a set of components for each pair of images;

means for indexing data pertaining to each identified component;

means for storing the indexed data in a database;

means for defining a plurality of features that are associated with each identified component; and

means for determining a subset of features that best discriminate positive component examples for each component.

57. The database system of claim 56 wherein each head pose is defined by a yaw angle.

58. The database system of claim 57 wherein each head pose is defined by a pitch angle.

59. The database system of claim 58 wherein each head pose is defined by a roll angle.

60. The database system of claim 56 wherein a same set of head poses is received for each individual imaged.

61. The database system of claim 56 wherein the set of components are identified manually.

62. The database system of claim 56 further comprising:

means for receiving pairs of non-components.

63. The database system of claim 56 wherein each feature is assigned a value that determines a strength of the feature.

64. The database system of claim 56 wherein means for determining a subset of features uses boosting.

65. The database method of claim 56 wherein the means for identifying a set of components uses disparity maps.

66. The database system of claim 56 further comprising:

means for testing each pair of images to identify stereo components in the images.

67. The database system of claim 56 wherein the set of components corresponds to a left eye.

68. The database system of claim 56 wherein the set of components corresponds to a right eye.

69. The database system of claim 56 wherein the set of components corresponds to a nose.

70. The database system of claim 56 wherein the set of components corresponds to a right ear.

71. The database system of claim 56 wherein the set of components corresponds to a left ear.