Systems and methods for detecting pedestrians in the vicinity of a powered industrial vehicle

• US 7,576,639 B2
• Filed: 03/13/2007
• Issued: 08/18/2009
• Est. Priority Date: 03/14/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation), Taylor Machine Works Incorporated (The Taylor Group Incorporated)
• Original Assignee: Mobileye Technologies Limited (Intel Corporation), Taylor Machine Works Incorporated (The Taylor Group Incorporated)
• Inventors: Benady, Meny, Herzberg, Eran, Dagan, Erez, Gdalyahu, Yoram, Hillyer, Matthew Craig, Boyles, Tonnie Michael
• Primary Examiner(s): Crosland; Donnie L
• Application Number:
  US11/685,208
• Publication Number:
  US 20070229238A1
• Time in Patent Office: 889 Days
• Field of Search: 340/435, 348/143, 348/135, 348/153
• US Class Current: 340/435
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system mounted on a powered industrial vehicle for detecting classifying and tracking in real time at least one obstruction in the scene around the vehicle. The vehicle is capable of moving in multiple directions. The system includes a multiple cameras mounted on the vehicle, wherein the viewing zones viewed respectively by the cameras preferably encompass 360° horizontally around the vehicle. Each of the cameras is operatively attached to an image processor, which processes the image frames acquired respectively by the camera. When a pedestrian is present in the viewing zone viewed by one of the cameras, the image processor attached to the one camera identifies in at least one of the image frames at least a portion of an image of the detected pedestrian.

Claims

1. A system mounted on a powered industrial vehicle for detecting and classifying in real time at least one obstruction in the scene around the vehicle, the vehicle capable of moving in multiple directions, the system comprising:

(a) a plurality of cameras mounted on the vehicle; and

(b) a plurality of image processors operatively attached to each of said cameras, wherein said image processors each processes a plurality of image frames acquired respectively by said cameras, and when a pedestrian is present in the viewing zone viewed by one of said cameras, the image processor attached to said one camera identifies in at least one of the image frames at least a portion of an image of the detected pedestrian, thereby producing a detected pedestrian data object, and wherein said detected pedestrian data object is transferred from said one camera to another one of the cameras, thereby tracking the pedestrian.

2. The system of claim 1, wherein the viewing zones viewed respectively by said cameras encompass 360°

horizontally around the vehicle.

3. The system of claim 1, wherein said tracking is performed horizontally over 360°

.

4. The system of claim 1, wherein said image processor attached to said one camera identified said detected pedestrian, further processes said at least one of the image frames to track said detected pedestrian.

5. The system of claim 1, further comprising:

(c) a system processor which controls said plurality of cameras and notifies the vehicle operator on each detected pedestrian.

6. The system of claim 5, wherein said system processor further processes said at least one of the image frames to track said detected pedestrian identified by said image processor.

7. The system of claim 5, wherein said system processor and said image processor share a common bus, wherein said transfer of said detected pedestrian data object from said one camera to another one of the cameras is performed over said bus and wherein said transfer of said detected pedestrian data object is also transferred to said system processor.

8. The system of claim 1, wherein said data object comprises features selected from the group of features including distance, azimuth angle, size, time, color.

9. The system of claim 5, further comprising:

(d) a vehicle operator interface for receiving signals from said system processor.

10. The system of claim 9, wherein said vehicle operator interface comprises indicators selected from the group of indicators including lights, colored lights, display, speakers.

11. The system of claim 10, wherein said speakers are configured in a stereophonic configuration or in a surround configuration.

12. The system of claim 11, wherein the stereo sound produced from said configuration is indicating to the vehicle operator the direction to said detected pedestrian.

13. The system of claim 5, further comprising:

(e) an activation mechanism operatively attached to said system processor, said activation mechanism causing vehicle to slow down or stop, to avoid an accident.

14. The system of claim 1, wherein said viewing zones as viewed by adjacent pairs of said cameras angularly overlap horizontally.

15. The system of claim 5, wherein said system processor and one of said image processors are operated from a single processor.

16. In a system mounted on a powered industrial vehicle for detecting and classifying in real time at least one obstruction in the scene around the vehicle, the vehicle capable of moving in multiple directions, a method comprising the steps of:

(a) providing plurality of cameras mounted on the vehicle; and

(b) processing said at least one image by image processors operatively attached to each of said cameras, wherein said image processors each processes a plurality of image frames acquired respectively by said cameras, and when a pedestrian is present in the viewing zone viewed by one of said cameras, the image processor attached to said one camera identifies in at least one of the image frames at least a portion of an image of the detected pedestrian, thereby producing a detected pedestrian data object, and wherein said detected pedestrian data object is transferred from said one camera to another one of the cameras, thereby tracking the pedestrian.

17. The system of claim 16, further comprising:

(c) providing a system processor which controls said plurality of cameras and notifies the vehicle operator on each detected pedestrian.

18. The method of claim 17, further comprising the step of:

(d) notifying the system processor by said image processor about said detected pedestrian, wherein the notification also includes said image processor identification code (ID).

19. The method of claim 17, wherein said system processor and said image processor share a common bus, and wherein said transferring of said detected pedestrian data object from said one camera to another one of the cameras is performed over said bus and wherein said transfer of said detected pedestrian data object is also transferred to said system processor.

20. The method of claim 16, wherein the viewing zones viewed respectively by said cameras encompass 360°

horizontally around the vehicle.

21. The method of claim 16, wherein said tracking is performed horizontally over 360°

.

22. The method of claim 16, wherein the distortion of each camera of said plurality of cameras is corrected.

23. The method of claim 16, wherein said data object comprises features selected from the group of features including distance, azimuth angle, size, time, color.

24. The method of claim 16, further comprising the steps of:

(e) computing the distance from the vehicle to said detected pedestrian; and

(f) computing the azimuth of said detected pedestrian relative to the track longitudinal axis.

25. The method of claim 24, wherein said computing the distance and said computing the azimuth are performed by said image processor.

26. The method of claim 24, wherein said computing the distance and said computing the azimuth are performed by said system processor.

27. The method of claim 16, further comprising a step of:

(g) tracking said detected pedestrian by said image processor.

28. The method of claim 24, further comprising a step of:

(g) tracking said detected pedestrian by said system processor.

29. The method of claim 26, further comprising the steps of:

(h) notifying the system processor by said image processor about said distance of each of said detected pedestrian, the notification also including said image processor ID; and

(i) notifying the system processor by said image processor about said azimuth of each of said detected pedestrian, the notification also including said image processor ID.

30. The method of claim 26, further comprising:

(j) activating a mechanism operatively attached to said system processor, said mechanism causing vehicle to slow down or stop, to avoid an accident.

31. The method of claim 24, wherein the position of each camera of said plurality of cameras with reference to the local external surface of vehicle is known in said computing of distance.

32. The method of claim 24, wherein the initial orientation of each camera of said plurality of cameras relative to the track longitudinal axis is known in said computing of azimuth.

33. The method of claim 17, wherein when a pedestrian is detected by two adjacent cameras of said plurality of cameras, stereo analysis is performed by said system processor to refine the distance computation to said detected pedestrian.

34. The method of claim 17, wherein when said notifying the vehicle operator on each of said detected pedestrian by said system processor comprise an audible notification.

35. The method of claim 34, wherein the volume of said audible notification is indicates the alarm level to the vehicle operator.

36. The method of claim 34, wherein the frequency of said audible notification indicates the alarm level to the vehicle operator.

37. The method of claim 34, wherein said audible notification is sound by a mono speaker, stereo speakers or surround speakers.

38. The method of claim 37, wherein said stereo speakers or surround speakers play said audible notification such that said audible notification is in the direction of said computed azimuth.

39. The method of claim 17, wherein when said notifying the vehicle operator on each of said detected pedestrian by said system processor comprise a visual notification.

40. The method of claim 39, wherein said visual notification is an indicator selected from the group of indicators including a light, colored lights, display.

41. The method of claim 40, wherein said colored lights indicate the alarm level to the vehicle operator.

42. The method of claim 40, wherein said display indicates the distance and azimuth of said detected pedestrian relative to the vehicle.

43. A system mounted on a powered industrial vehicle for detecting and classifying in real time at least one obstruction in the scene around the vehicle, the vehicle capable of moving in multiple directions, the system comprising:

(a) a plurality of cameras mounted on the vehicle; and

(b) a plurality of image processors operatively attached to each of said cameras, wherein said image processors each processes a plurality of image frames acquired respectively by said cameras, said image processors attach a respective camera unit identification code to each said image frames; and

(c) a system processor which controls said cameras and processes each of said image frames acquired respectively by said cameras and transferred to said system processor, and when a pedestrian is present in the viewing zone viewed by one of said cameras, said system processor identifies in at least one of the image frames at least a portion of an image of the detected pedestrian, thereby producing a detected pedestrian data object, and notifies the vehicle operator on each detected pedestrian, and wherein said detected pedestrian data object is transferred from said one camera to another one of the cameras, thereby tracking the pedestrian.

44. In a system mounted on a powered industrial vehicle for detecting and classifying in real time at least one obstruction in the scene around the vehicle, the vehicle capable of moving in multiple directions, a method comprising the steps of:

(a) a plurality of cameras mounted on the vehicle; and

(b) processing said at least one image by image processors operatively attached to each of said cameras, wherein said image processors each processes a plurality of image frames acquired respectively by said cameras, said image processors attach a respective camera unit identification code (ID) to each said image frames; and

(c) transferring said image frames with said respective ID to a system processor, wherein said system processor processes each of said image frames acquired respectively by said cameras and transferred to said system processor, and when a pedestrian is present in the viewing zone viewed by one of said cameras, said system processor identifies in at least one of said image frames at least a portion of an image of the detected pedestrian, thereby producing a detected pedestrian data object, and notifies the vehicle operator on each detected pedestrian.

45. The method of claim 44, wherein the viewing zones viewed respectively by said cameras encompass 360°

horizontally around the vehicle.

46. The method of claim 44, further comprising the steps of:

(d) computing the distance from the vehicle to said detected pedestrian by said system processor; and

(e) computing the azimuth of said detected pedestrian relative to the track longitudinal axis by said system processor.

47. The method of claim 44, further comprising the steps of:

(f) tracking said detected pedestrian by said system processor.

48. The method of claim 47, wherein said tracking is performed horizontally over 360°

.

49. The method of claim 44, further comprising the steps of:

(g) activating a mechanism operatively attached to said system processor, said mechanism causing vehicle to slow down or stop, to avoid an accident.