System and method of alerting a driver that visual perception of pedestrian may be difficult

• US 9,070,023 B2
• Filed: 09/23/2013
• Issued: 06/30/2015
• Est. Priority Date: 09/23/2013
• Status: Active Grant

• Current Assignee: Indiana University Research & Technology Corporation (Indiana University), Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Indiana University Research & Technology Corporation (Indiana University), Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation), Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Du, Eliza Y., Yang, Kai, Jiang, Pingge, Takahashi, Hiroyuki, Sherony, Rini
• Primary Examiner(s): Seth, Manav
• Application Number:
  US14/034,103
• Publication Number:
  US 20150086077A1
• Time in Patent Office: 645 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06V 10/755 Deformable models or variational models, e.g. snakes or active contours
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A pedestrian perception alert system configured to issue a warning during real-time when a driver'"'"'s visual detection of a pedestrian is difficult, and a method thereof is provided. The system includes a video camera, an alert for issuing a warning, a processor, and a Pedestrian Detection Unit (“PDU”). The PDU analyzes the video camera image to detect a pedestrian. A Global Clutter Analysis Unit (“GCAU”) generates a global clutter score. A Local Pedestrian Clutter Analysis Unit (“LPCAU”) generates a local pedestrian clutter score. The processer processes the global clutter score and local pedestrian clutter score so as to generate a pedestrian detection score. The alert is actuated when the pedestrian detection score is outside of a predetermined threshold so as to notify the driver that perception of a pedestrian is difficult at that time.

Claims

1. A pedestrian perception alert system configured to issue a warning in real-time when driver visual perception of a pedestrian is difficult, the pedestrian perception alert system comprising:

a video camera configured to capture video image;

an alert for issuing the warning;

a processor in electrical communication with the video camera;

a Pedestrian Detection Unit (“

PDU”

) configured to analyze the video image to detect the pedestrian;

a Global Clutter Analysis Unit (“

GCAU”

) configured to generate a global clutter score, the global clutter score measuring the clutter of the entire video image; and

a Local Pedestrian Clutter Analysis Unit (“

LPCAU”

) configured to generate a local pedestrian clutter score, the local pedestrian clutter score measuring the clutter of each of the pedestrians detected in the video image, wherein when the PDU detects a pedestrian in the video image the processor initiates both the GCAU and the LPCAU, the processer processes the global clutter score and local pedestrian clutter score so as to generate a pedestrian detection score, and processor further actuating the alert when the pedestrian detection score is outside of a predetermined threshold so as to warn a driver that it is difficult to visually perceive a pedestrian.

2. The pedestrian perception alert system as set forth in claim 1, further including a Pedestrian Contour Generation Unit (“

PCGU”

) configured to generate a pedestrian mask, the LCPAU further processing the pedestrian mask so as to compute the local pedestrian clutter score.

3. The pedestrian perception alert system as set forth in claim 2, wherein the GCAU processes edge density, luminance variation and chrominance variation of the video image to calculate the global clutter score and the LPCAU processes edge density of the detected pedestrian, edge distribution, local luminance variation, local chrominance variation, mean luminance intensity, and mean chrominance intensity to calculate the local pedestrian clutter score.

4. The pedestrian perception alert system as set forth in claim 3, wherein pedestrian detection score is the difference between the global clutter score and local pedestrian score.

5. The pedestrian perception alert system as set forth in claim 3, wherein the GCAU is further configured to remove high frequency image components and subsequently calculating a ratio between a number of edge pixels and a total number of pixels within a video frame so as to calculate the edge density.

6. The pedestrian perception alert system as set forth in claim 3, wherein the GCAU includes a sliding window and a luminance variation matrix dimensioned the same size as the video frame, the GCAU configured to slide the sliding window across the entire video frame so as to calculate a standard deviation of luminance value within the sliding window, the standard deviation for a particular area of the video frame is entered into a corresponding position of the luminance variation matrix, and wherein the luminance variation is calculated as the mean value of the luminance matrix.

7. The pedestrian perception alert system as set forth in claim 3, wherein the chrominance variation is calculated using two chrominance channels.

8. The pedestrian perception alert system as set forth in claim 3, wherein the edge density, luminance variation, and chrominance variation are evenly weighted when calculating the global clutter score.

9. The pedestrian perception alert system as set forth in claim 2, wherein the PCGU generates a contour of the detected pedestrian and generates a deformable model which is applied to the contour, wherein the PCGU applies energy minimization function to further refine the contour so as to generate the pedestrian mask.

10. The pedestrian perception alert system as set forth in claim 9, wherein PCGU is further configured to segment a cloth region from a background image of the video image so as to further generate a cloth mask.

11. The pedestrian perception alert system as set forth in claim 9, wherein the LPCAU is further configured to generate a background window and a detected pedestrian window, the background window being at least twice the area of the detected pedestrian window, and wherein the background window includes the video image surrounding the detected pedestrian.

12. The pedestrian perception alert system as set forth in claim 11, wherein the LPCAU is further configured to determine the ratio between the number of edge pixels and the total number of pixels within both (1) the detected pedestrian window and (2) the background window and absent the detected pedestrian window, so as to calculate an edge density for a pedestrian.

13. The pedestrian perception alert system as set forth in claim 12, wherein the LPCAU is configured to calculate the edge distribution of the background window and the detected pedestrian by determining the histogram of edge magnitude binned by the edge orientation for both (1) the detected pedestrian window and (2) the background window and absent the detected pedestrian window.

14. The pedestrian perception alert system as set forth in claim 11, wherein the LPCAU is configured to calculate the local luminance variation within the pedestrian mask and also within a region defined by the subtraction of the pedestrian mask from the background window.

15. The pedestrian perception alert system as set forth in claim 11, wherein the LPCAU calculates the local chrominance variation within the pedestrian mask and also within a region defined by the subtraction of the pedestrian mask from the background window.

16. The pedestrian perception alert system as set forth in claim 11, wherein the LPCAU calculates the mean luminance intensity within the cloth mask and a region generated by subtracting the cloth mask from the background window.

17. The pedestrian perception alert system as set forth in claim 11, wherein the LPCAU is configured to calculate the mean chrominance intensity within the cloth mask and a region generated by subtracting the cloth mask from the background window.

18. The pedestrian perception alert system as set forth in claim 1, wherein the PDU is configured to execute a first detection method or a second detection method based upon the probability of pedestrian appearance within the video image, wherein the first detection method is executed in instances where there is a low chance of pedestrian appearance and the second detection method is executed in instances where there is a high chance of pedestrian appearance.

19. The pedestrian perception alert system as set forth in claim 18, wherein the PDU determines a probability of a pedestrian appearance based upon at least one of the following criteria:

time, geographic location, or traffic scene.

20. The pedestrian perception alert system as set forth in claim 19, wherein the first detection method is configured to identify a region of interest within the video image by determining the variation between sequential frames of the video image, and identifies a region of interest in instances where the variation exceeds a predetermined threshold, the first detection method further applying a set of constraints to each of the regions of interest, wherein each region of interest having a requisite number of constraints is labeled as having a pedestrian.

21. The pedestrian perception alert system as set forth in claim 20, wherein the second detection method is configured to determine the region of interest within the video image by detecting vertical edges within the frame, and identifies the region of interests in instances where the vertical edge has a predetermined characteristic, the second detection method further applying a feature filter to each region of interest, wherein each region of interest having a requisite number of features is labeled as having a pedestrian.

22. The pedestrian perception alert system as set forth in claim 1, further including a Saliency Map Generating Unit (“

SMGU”

), the SMGU is configured to process the video image and extract salient features from the video image, wherein the processor is further configured to actuate the LPCAU wherein the extracted salient features are processed so as to generate the local pedestrian clutter score.

23. The pedestrian perception alert system as set forth in claim 22, wherein the salient features include pedestrian behavior.

24. The pedestrian perception alert system as set forth in claim 23, wherein the pedestrian behavior is pedestrian motion.

25. The pedestrian perception alert system as set forth in claim 23, wherein the pedestrian behavior is based upon an environment surrounding the pedestrian.

26. The pedestrian perception alert system as set forth in claim 22, further including a Pedestrian Group Analysis Unit (“

PGAU”

) configured to detect a group of pedestrians and assign a perception difficulty value to the group of pedestrians, wherein the PGAU analyzes individual pedestrian interaction within the group of pedestrians, and the interaction of one group of pedestrians with respect to another group of pedestrians.

27. A method for issuing an alert in real-time when a driver'"'"'s visual detection of a pedestrian is difficult, the method comprising the steps of:

providing a video camera configured to capture video image;

providing an alert for issuing a warning that the pedestrian in a driving environment is visually difficult to perceive;

providing a processor in electrical communication with the camera;

analyzing the video image to detect a pedestrian;

measuring a clutter of an entire video image and calculating a global clutter score;

measuring the clutter of each pedestrian detected in the video image and calculating a local pedestrian clutter score;

processing the global clutter score and local pedestrian clutter score to calculate a pedestrian detection score; and

issuing a warning when the pedestrian detection score is outside of a predetermined threshold so as to notify the driver that visual perception of a pedestrian is difficult.

28. The method as set forth in claim 27, further including the step of providing a processor, Pedestrian Detection Unit (“

PDU”

), a Global Clutter Analysis Unit (“

GCAU”

), and a Local Pedestrian Clutter Analysis Unit (“

LPCAU”

), the PDU analyzes the video image to detect a pedestrian, the GCAU analyzes the video image to measure the clutter of the entire video image and calculate a global clutter score, and the LPCAU analyzes the detected pedestrians to measure the clutter of the detected pedestrians and calculate a local pedestrian clutter score, the processor processing the global clutter score and local pedestrian clutter score so as to calculate the pedestrian detection score.

29. The method as set forth in claim 28, further including the step of generating a pedestrian mask, the LPCAU processing the pedestrian mask to calculate the local pedestrian clutter score.

30. The method as set forth in claim 29, including a Pedestrian Contour Generation Unit (“

PCGU”

) configured to generate the pedestrian mask.

31. The method as set forth in claim 30, wherein the GCAU processes edge density, luminance variation and chrominance variation of the video image to calculate the global clutter score and the LPCAU processes edge density of the detected pedestrian, edge distribution, local luminance variation, local chrominance variation, mean luminance intensity, and mean chrominance intensity to calculate the local pedestrian clutter score.

32. The method as set forth in claim 31, wherein pedestrian detection score is the difference between the global clutter score and local pedestrian score.

33. The method as set forth in claim 32, wherein the GCAU is further configured to remove high frequency image components and subsequently determine a ratio between a number of edge pixels and a total number of pixels within a video frame so as to calculate the edge density.

34. The method as set forth in claim 33, wherein the GCAU includes a sliding window and a luminance variation matrix dimensioned the same size as the video frame, the GCAU configured to slide the sliding window across the entire video frame so as to calculate a standard deviation of luminance value within the sliding window, the standard deviation for a particular area of the video frame is entered into the corresponding position of the luminance variation matrix, and wherein the luminance variation is calculated as the mean value of the luminance variation matrix.

35. The method as set forth in claim 31, wherein the chrominance variation is calculated using two chrominance channels.

36. The method as set forth in claim 31, wherein the edge density, luminance variation, and chrominance variation are evenly weighted when calculating the global clutter score.

37. The method as set forth in claim 30, wherein the PCGU generates a contour of the detected pedestrian and generates a deformable model which is applied to the contour, wherein the PCGU applies energy minimization function to further refine the contour so as to generate the pedestrian mask.

38. The method as set forth in claim 37, wherein PCGU is further configured to segment a cloth region from a background image of the video image so as to further generate a cloth mask.

39. The method as set forth in claim 38, wherein the LPCAU is further configured to generate a background window and a detected pedestrian window, the background window being at least twice the area of the detected pedestrian window, and wherein the background window includes the video image surrounding the detected pedestrian.

40. The method as set forth in claim 39, wherein the LPCAU is further configured to determine the ratio between the number of edge pixels and the total number of pixels within both (1) the detected pedestrian window and (2) the background window and absent the detected pedestrian window, so as to calculate an edge density for a pedestrian.

41. The method as set forth in claim 40, wherein the LPCAU is configured to calculate an edge distribution of the background window and the detected pedestrian by determining the histogram of edge magnitude binned by the edge orientation for both (1) the detected pedestrian window and (2) the background window and absent the detected pedestrian window.

42. The method as set forth in claim 39, wherein the LPCAU is configured to calculate the local luminance variation within the pedestrian mask and also within a region defined by the subtraction of the pedestrian mask from the background window.

43. The method as set forth in claim 39, wherein the LPCAU is configured to calculate the local chrominance variation within the pedestrian mask and also within a region defined by the subtraction of the pedestrian mask from the background window.

44. The method as set forth in claim 39, wherein the LPCAU is configured to calculate the mean luminance intensity within the cloth mask and a region generated by subtracting the cloth mask from the background window.

45. The method as set forth in claim 39, wherein the LPCAU is configured to calculate the mean chrominance intensity within the cloth mask and a region generated by subtracting the cloth mask from the background window.

46. The method as set forth in claim 27, wherein the PDU is configured to execute a first detection method or a second detection method based upon the probability of a pedestrian appearance within the video image, wherein the first detection method is executed in instances where there is a low chance of pedestrian appearance and the second detection method is execute in instances where there is a high chance of pedestrian appearance.

47. The method as set forth in claim 44, wherein the PDU determines a probability of a pedestrian appearance based upon at least one of time, geographic location, or traffic scene.

48. The method as set forth in claim 45, wherein the first detection method is configured to identify regions of interest within the video image by determining the variation between sequential frames of the video image, and identifies a region of interest in instances where the variation exceeds a predetermined threshold, the first detection method further applying a set of constraints to each of the regions of interest, wherein each region of interest having a requisite number of constraints is labeled as having a pedestrian.

49. The method as set forth in claim 46, wherein the second detection method is configured to determine regions of interest within the video image by detecting vertical edges within the frame, and identifies a region of interest in instances where the vertical edge has a predetermined characteristic, the second detection method further applying a feature filter to each region of interest, wherein each region of interest having a requisite number of features is labeled as having a pedestrian.

50. The method as set forth in claim 27, further including the step of utilizing pedestrian behavior to calculate the pedestrian detection score.

51. The method as set forth in claim 50, including a Saliency Map Generating Unit (“

SMGU”

), the SMGU is configured to process the video image and extract salient features from the video image, wherein the processor is further configured to process the extracted salient features with LPCAU so as to generate a local pedestrian clutter score, and wherein the salient features include pedestrian behavior.

52. The method as set forth in claim 51, wherein the pedestrian behavior is pedestrian motion.

53. The method as set forth in claim 51, wherein the pedestrian behavior is based upon an environment surrounding the pedestrian.

54. The method as set forth in claim 27, further including the step of analyzing individual pedestrian interaction within a group of pedestrians, and the interaction of one group of pedestrians with respect to another group of pedestrians to calculate the pedestrian detection score.

55. The method as set forth in claim 54, further including a Pedestrian Group Analysis Unit (“

PGAU”

) configured to detect the group of pedestrians and assign a perception difficulty value to the group of pedestrians, the PGAU analyzes individual pedestrian interaction within the group of pedestrians, and the interaction of one group of pedestrians with respect to another group of pedestrians.