Multi-camera vision system and method of monitoring

• US 20180330175A1
• Filed: 05/10/2017
• Published: 11/15/2018
• Est. Priority Date: 05/10/2017
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: FotoNation Limited (Adeia, Inc.)
• Inventors: CORCORAN, Peter
• Granted Patent: US 10,740,627 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A multi-camera vision system and method of monitoring. In one embodiment imaging systems provide object classifications with cameras positioned to receive image data from a field of view to classify an object among multiple classifications. A control unit receives classification or position information of objects and (ii) displays an image corresponding to a classified object relative to the position of the structure. An embodiment of a related method monitors positions of an imaged object about a boundary by continually capturing at least first and second series of image frames, each series comprising different fields of view of a scene about the boundary, with some of the image frames in the first series covering a wide angle field of view and some of the image frames in the second series covering no more than a narrow angle field of view.

Claims

1. A multi-camera vision processing system which provides multiple fields of view exterior to a structure, comprising:

a plurality of imaging systems, each including a camera positionable about a peripheral surface of the structure, the imaging systems each configured to provide object classifications over a depth of field providing a range of camera focus distances extending away from the peripheral surface of the structure, with each camera configured or positioned about the peripheral surface to receive image data from a field of view characterized by a field of view angle, each camera system including a processor, memory and a non-transitory computer readable medium containing program instructions representing software executable on the processor, which instructions, when executed by the processor, cause the camera system to perform a sequence of steps which classify an object among multiple classifications based on an image of the object present within a camera FOV; and

a central control unit comprising a programmable processor, memory and a non-transitory computer readable medium, the central control unit coupled (i) to receive classification or position information of objects from the imaging systems and (ii) to display an image corresponding to a classified object relative to the position of the structure.

2. The multi-camera vision processing system of claim 1 wherein the structure is a moving vehicle and imaging systems are configured in groups, with imaging systems in each of a plurality of the groups configured to acquire images of objects positioned in different ranges of distance from the peripheral surface of the structure.

3. The multi-camera vision processing system of claim 1 wherein:

cameras in a first group are configured to acquire data comprising images of objects positioned in a first range of distances from the peripheral surface of the structure; and

cameras in a second group are configured to acquire data comprising images of objects positioned in a second range of distances from the peripheral surface of the structure where the second range of distances extends farther from the peripheral surface of the structure than the first range of distances extends from the peripheral surface of the structure.

4. The multi-camera vision processing system of claim 1 wherein:

cameras in a first group are configured with relatively wide angle fields of view to acquire data comprising images of objects positioned in a first range of distances from the peripheral surface of the structure; and

cameras in a second group are configured with relatively narrow angle fields of view to acquire data comprising images of objects positioned in a second range of distances from the peripheral surface of the structure.

5. The multi-camera vision processing system of claim 4 wherein cameras in the first group are configured to identify or classify images of objects positioned in the first range of distances without identifying or classifying an image of an object positioned in the second range of distances.

6. The multi-camera vision processing system of claim 1 wherein cameras in a first group are configured to identify or classify images of objects positioned in the first range of distances without identifying or classifying images of objects positioned in the second range of distances.

7. The multi-camera vision processing system of claim 1 wherein:

cameras in a first group are configured to have relatively wide field of view angles and acquire data comprising images of objects positioned in a first range of camera focus distances from the peripheral surface of the structure; and

cameras in a second group are configured to have relatively narrow field of view angles and acquire data comprising images of objects positioned in a second range of camera focus distances from the peripheral surface of the structure where the second range of distances extends farther from the peripheral surface of the structure than the first range of distances extends from the peripheral surface of the structure.

8. The multi-camera vision processing system of claim 1, where:

imaging systems in a first group are configured to receive image data from different fields of view each having the same first field of view angle; and

imaging systems in a second group are configured to receive image data from different fields of view each having the same second field of view angle.

9. The multi-camera vision processing system of claim 8 where some of the imaging systems in the first group provide to the central control unit classification or position information of objects based on overlapping fields of view provided by the cameras in the first group.

10. The multi-camera vision processing system of claim 8 where:

some of the imaging systems in the first group provide to the processor unit classification or position information of objects based on overlapping fields of view provided by the cameras in the first group; and

the second field of view angle is smaller than the first field of view angle;

11. The multi-camera vision processing system of claim 1 where:

some of the imaging systems in a first group provide to the processor unit classification or position information of objects based on overlapping fields of view provided by cameras in the first group.

12. The multi-camera vision processing system of claim 10 where:

cameras in the first group are configured to acquire data comprising images of objects positioned in a first range of distances from the peripheral surface of the structure; and

cameras in the second group are configured to acquire data comprising images of objects positioned in a second range of distances from the peripheral surface of the structure where the second range of distances extends farther from the peripheral surface of the structure than the first range of distances extends from the peripheral surface of the structure and the first field of view angle is greater than the second field of view angle.

13. The processing system of claim 2 where, during operation, the system acquires images of objects with cameras in the second group each having a range of camera focus distances extending farther from the peripheral surface than the range of focus distances of cameras in the first group extends from the peripheral surface.

14. The processing system of claim 1 where, during operation:

cameras in the plurality of imaging systems include cameras with one or more relatively wide field of view angles, cameras with one or more relatively narrow field of view angles and cameras with one or more field of view angles intermediate the wide and narrow angles;

imaging systems in a first group provide the one or more relatively wide angle fields of view and a first range of focus distances relative to the peripheral surface;

imaging systems in a second group provide the one or more relatively narrow angle fields of view and a second range of focus distances, relative to the peripheral surface, which second range of focus distances extends farther from the peripheral surface than focus distances in the first range extend from the peripheral surface; and

imaging systems in a third group have the one or more intermediate field of view angles and a third range of focus distances, where;

the third range of focus distances extends farther from the peripheral surface than the first range of focus distances extends from the peripheral surface, and the second range of focus distances extends farther from the peripheral surface than the third range of focus distances extends from the peripheral surface.

15. The processing system of claim 14 where, during operation:

imaging systems in the first group are configured to identify images of objects positioned in the first range of focus distances but not images of objects positioned in the second range of focus distances, and

imaging systems in the second group are configured to identify images of objects positioned in the second range of focus distances but not images of objects positioned in the first range of focus distances.

16. The multi-camera vision processing system of claim 1 wherein the sequence of steps performed by the camera system also provides position or displacement information of the object relative to the structure.

17. The processing system of claim 1 configured to generate a street map when the structure is a moving vehicle, wherein the non-transitory computer readable medium of the central control unit includes a program containing instructions representing software executable on the control unit processor, which instructions, when executed by the control unit processor, cause the control unit to perform a sequence of steps which generate the street map displaying movement or position of the vehicle on the map and overlay on the street map position of an image, corresponding to a classified object, relative to the vehicle position.

18. The multi-camera vision processing system of claim 1 wherein objects are classified with parallel classifier cascades based on one or more of a Haar measure, local binary pattern features, a histogram of oriented gradients, and a census transform, and the objects classified include vehicles and pedestrians.

19. A method of monitoring for the presence of or monitoring positions of an imaged object, in a set of possible object types, about a boundary, comprising:

continually capturing at least a first series of image frames and a second series of image frames, each series comprising multiple different fields of view of a scene about the boundary,

some of the image frames in the first series at least covering a wide angle field of view; and

some of the image frames in the second series at most covering no more than a narrow angle field of view relative to the wide angle field of view covered by images in the first series.

20. The method of claim 19 where:

image frames in the first series are captured with the one or more cameras each providing a first range of focus distances relative to the boundary of the structure; and

image frames in a second series are captured with the one or more cameras each providing a second range of focus distances relative to the boundary of the structure.

21. The method of claim 19 wherein the multiple series of image frames are captured with a plurality of imaging systems each having a processing unit containing program instructions representing software executable therein, the method including deriving information from the image frames according to the program instructions which, when executed, cause the imaging system to perform a sequence of steps which result in classification of an object among multiple object classes based on an image of the object present within one of the fields of view.

22. The method of claim 21 further including:

transmitting classification information derived by one of the imaging systems to a central control unit; and

combining the classification information with object position or movement information to provide situational awareness about the boundary.

23. The method of claim 19 where:

image frames in the first series are captured with one or more cameras in a first series positioned along the boundary and covering a first range of focus distances relative to the boundary; and

image frames in the second series are captured with one or more cameras in a second series covering a second range of focal distances relative to the boundary.

24. The method of claim 23 where the second range of focus distances extends farther from the boundary than the first range of focus distances.

25. The method of claim 19 applied to a boundary which is a peripheral surface of a moving vehicle with capturing of the image frames performed by:

placing one or more cameras along the peripheral surface to capture the image frames in the first series which cover at least the wide angle field of view; and

placing one or more additional cameras along the peripheral surface to capture the image frames in the second series which cover at most no more than the narrow angle field of view.

26. The method of claim 19 further including classifying the object based on presence of an image of a region of the object in a field of view by:

performing a complete scan of an image frame in the first series with a scanning window to match a region in the image frame with an object type characteristic;

or

performing a complete scan of an image frame in the second series with a scanning window to match a region in the image frame with an object type characteristic.

27. The method of claim 19 where classifying the object based on presence of an image of the object region in a field of view includes:

performing a complete scan of an image frame in the first series with multiple scanning windows of differing sizes to match a region in the image frame with an object type characteristic; and

performing a complete scan of an image frame in the second series with multiple scanning windows of differing sizes to match a region in the image frame with an object type characteristic.

28. The method of claim 27 where the image of the object region is larger than one of the scanning windows and the image of the object region fits within one of the scanning windows.

29. The method of claim 27, where the complete scan of an image frame in one of the series is performed with no more than ten scanning windows of differing sizes to match the region in the image frame with an object type characteristic.

30. The method of claim 27, where the complete scan of an image frame in one of the series is performed with no more than five scanning windows of differing sizes to match the region in the image frame with an object type characteristic.

31. A method of classifying and tracking an imaged object, among a group of possible object types positioned about a moving vehicle, for display to a driver of the vehicle, the method comprising:

with the object positioned about a moving vehicle, simultaneously operating multiple cameras to image portions of the same scene in which the object is located with the cameras capturing different field of view angles so that cameras in a first series each acquire frames of image data covering relatively large fields of view of scene portions and cameras in a second series each acquire frames of image data covering relatively small fields of view of scene portions; and

classifying the imaged object according to a determination of object type based on similarity matching among a set of possible object type characteristics by applying multiple scan windows through frames of image data generated by cameras in both the first series and the second series and determining whether a region in an image frame matches an object type characteristic.

32. The method of claim 31 applied to identify and classify objects taken from a group comprising automobiles, medium trucks, heavy trucks, motor cycles, pedestrians, and bicycles.

33. The method of claim 31 where some of the cameras in at least one of the series are configured to provide a continuous sequence of adjoining fields of view about the vehicle and two fields of view next to one another in the sequence overlap with one another.

34. The method of claim 31 where cameras in at least one series each acquire frames of image data covering overlapping fields of view.

35. The method of claim 31 where:

an object positioned relatively far from the vehicle is identified or classified based on image data generated by a camera in the second series;

or

an object positioned relatively close to the vehicle is identified or classified based on image data generated by a camera in the first series.

36. The method of claim 31 where:

some of the cameras in the first series each have a relatively short focal length suitable for imaging an object positioned relatively close to the vehicle and classifying the object by applying some of the multiple scan windows; and

some of the cameras in the second series each have a relatively long focal length suitable for imaging an object positioned relatively far from the vehicle and classifying the object by applying some of the multiple scan windows.

37. The method of claim 31 where the first series of cameras and the second series of cameras both comprise cameras each having a fixed focus and a fixed field of view angle.

38. The method of claim 31 where object detection or classification is performed by applying multiple scan windows through entire frames of image data generated by cameras in both the first series and the second series to determine whether a region in an image frame matches an object type characteristic.

39. The method of claim 31 where some of the cameras in the first and second series are each part of an image acquisition device which processes a sequence of image frame data to classify objects, the method further including transmitting object type determinations and object location information from one or more of the image acquisition devices for display of object type and object location.

40. The method of claim 31 where the similarity matching is performed with cascading classifiers.

41. The method of claim 31 where classifying includes simultaneously applying multiple classifier sets to digital image data for parallel classification processing of the image data in each frame for multiple object types with cascading classifiers.

42. The method of claim 31 including, in lieu of displaying captured portions of the scene containing the imaged object, transmitting only the object type determination and location information for display.

44. A method of identifying an object of interest in a zone within a region about a structure based on sizes of object images, comprising:

acquiring a scene image in a field of view for each in a series of at least two zones of the region about the structure where a first of the zones is relatively close to the structure and a second of the zones, separate and distinct from the first zone, extends farther away from the structure than the first zone, each scene image being a frame of image data of a given size acquired at a given field of view angle; and

applying a first criterion to identify an image of a first object present in a first of the acquired scene images as a first object type, while the first object is positioned in the first zone and wherein a first scene image subtends a first field of view angle,

the first criterion requiring, in order for the first object to be identified as an object of interest in the first zone, that the image of the first object in the first of the acquired scene images be within a predetermined first range of sizes relative to the given size of the frame of image data based on the first field of view angle, said first criterion enabling identification of images of objects positioned within the first zone without identifying an image of a second object visible in the first scene but positioned farther away from the structure than the first zone extends from the structure based, in part, on size of the image of the second object in the first scene relative to the given size of the frame of image data based on the first field of view angle.

45. The method of claim 44 further including:

applying second criteria to identify the image of the second object when present in a second of the acquired scene images as the first object type, while the second object is positioned in the second zone and wherein the second scene image subtends a second field of view angle,

the second criteria requiring, in order for the second object to be identified as an object of interest in the second zone, that the image of the second object in the second of the acquired scene images be within a predetermined second range of sizes relative to the given size of the frame of image data based on the second field of view angle, said second criterion enabling identification of the image of the second object in the second scene image when the second object is positioned within the second zone based, in part, on size of the image of the second object in the second scene relative to the given size of the frame of image data based on the second field of view angle.

46. The method of claim 45 where the first and second criteria are based on the same range of image sizes relative to the image frame size for the first and second field of view angles.

47. The method of claim 44 where field of view angles with which scene images in adjacent zones are acquired decrease as distance from the structure increases.

48. The method of claim 44 where the field of view angle with which every scene image in each of the zones is acquired decreases as distance from the structure increases.

49. The method of claim 44 where the first criterion do not differ from the second criterion.

50. The method of claim 44 where each zone subtends an angle of at least 90°

about a point on the structure.

51. The method of claim 50 where multiple ones of the zones are bands extending around a point on the structure.

52. The method of claim 50 including acquiring a sequence of scene images from each of the first and second zones and determining positions of the first and second objects relative to the structure as a function of time.

53. The method of claim 52 where the series extends away from the structure and field of view angles of all of the multiple scene images among the first and second zones decrease as a function of increasing distance from the structure.

54. The method of claim 53 where application of identification and classification criteria includes operations which iteratively apply a series of different size scan windows in a field of view that limit object detection to a predetermined range of image sizes in the field of view.

55. The method of claim 54 where position of an object image relative to the structure is based on (i) the distance between the portion of the zone in which an object image appears and the structure, or (ii) the size of the object image relative to the given size of the frame of image data in which the object image is captured.

56. The method of claim 44 where, compared to the field of view angle of the first scene image acquired from the first of the zones relatively close to the structure, field of view angles decrease for multiple image scenes acquired from the second of the zones with multiple cameras, thereby increasing sizes of images of objects positioned in the second of the zones to limit the number of scan window sizes needed for detection of images according to the second criterion.

57. The method of claim 44 where the structure is a vehicle in motion and the zones correspond to a series of detection zones positioned varied distances from the vehicle.