Multi-camera vehicle vision system and method

• US 20180330526A1
• Filed: 05/10/2017
• Published: 11/15/2018
• Est. Priority Date: 05/10/2017
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: CORCORAN, Peter
• Granted Patent: US 11,615,566 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A multi-camera vehicle vision system and method. In one embodiment a map is generated about a moving vehicle. Frames of image data are provided with a series of cameras extending along a surface of the vehicle. The image data frames are processed to identify an object of interest. An object of interest is classified among a set of object types and location of an identified object of interest is determined. Object type and location information is provided to a control unit spaced apart from the cameras via a data link. Road map data is generated to illustrate changes in position of the moving vehicle along a roadway based on data other than the image data provided by the cameras. A display of the road map data is generated with the object type and location information overlaid on the road map data to indicate object location relative to the vehicle.

Claims

1. A method of generating a map about a moving vehicle comprising:

providing frames of image data with a series of cameras extending along a surface of the vehicle;

processing the image data frames to identify an object of interest;

classifying an object of interest among a set of object types and determining location of an identified object of interest as a function of time;

providing object type and location information to a central control unit spaced apart from the cameras via a data link;

generating road map data to illustrate changes in position of the moving vehicle along a roadway based on data other than the image data provided by the cameras; and

generating a display of the road map data with the object type and location information overlaid on the road map data to indicate object location relative to the vehicle.

2. The method of claim 1 where the step of classifying an object of interest is performed by template matching with multiple scan windows of different sizes applied through entire image data frames generated by cameras to determine whether a region in an image frame matches an object type characteristic.

3. The method of claim 1 where an object is identified in one among multiple detection zones about the vehicle and, for a given field of view angle capturing an image data frame in the detection zone, position information for the object is approximated based on the distance, that the portion the detection zone, in which the object is positioned, is from the vehicle or the relative size of the object image in the given field of view to the size of the image data frame.

4. A network based vision system operable to monitor a plurality of object detection zones in a region about a vehicle traveling over a roadway for presence or location of objects about the vehicle, the system comprising:

a plurality of image acquisition devices for acquiring multiple fields of view of a scene surrounding the vehicle, each device having programmable settings, connected to transmit data and receive instructions, the image acquisition devices configurable into multiple series each comprising one or more of the image acquisition devices,

a control unit connected through the network to receive data from the image acquisition devices and control configuration of the object detection zones,

the control unit including a programmable microprocessor, memory, and data storage containing data and executable software which runs on the microprocessor, the memory and storage operatively connected to the microprocessor, the storage including a non-transitory computer readable medium containing program instructions representing software executable on the processor, which instructions, when executed by the processor, cause the control unit to perform a sequence of steps based on an association between each object detection zone and a series of the image acquisition devices by which the control unit;

receives and applies information indicative of a detected object type or detected location from one or more of the image acquisition devices, and

automatically reassigns one or more of the image acquisition devices from one series to another series based on one or more predetermined criteria.

5. The vision system of claim 4 where the program instructions base reassignment of the one or more image acquisition devices to a different series on a criterion taken from the group consisting of vehicle speed, location of a detected object relative to the vehicle, and change in vehicle direction of motion.

6. The vision system of claim 4 where the program instructions cause the control unit to apply the information to generate an overlay indicative of object type or position on a real time map of the roadway.

7. The vision system of claim 4 wherein the system operates each series of image acquisition devices to identify and classify one or more objects or the position of the one or more objects.

8. The vision system of claim 4 where reassignment of the one or more image acquisition devices from one series to another series modifies a characteristic of a detection zone while the vehicle is operating, and the characteristic is taken from the group consisting of size or shape of the detection zone, distance between a portion of the detection zone and the vehicle, captured field of view of a scene in the detection zone, field of view angle of an image acquisition device assigned to the detection zone, depth of field of an image acquisition device assigned to the detection zone, and ability to detect an object positioned in the detection zone based on image size of the object.

9. The vision system of claim 4 where the program instructions cause the control unit to reassign image acquisition devices among different ones of the series to modify ability of the system to detect, classify or track an object in response to a change in vehicle operation or a change in roadway situation about the vehicle.

10. The vision system of claim 4 where the vision system is configured to operate the image acquisition devices in four detection zones.

11. The vision system of claim 4 where the program instructions cause the system to adjust one or more settings of the image acquisition devices taken from the group comprising field of view, depth of field and pointing direction for image acquisition.

12. The vision system of claim 4 where the image acquisition devices are grouped in two dimensional array units, devices in each array unit deployable based on execution of the program instructions to automatically reassign the image acquisition devices from one series to another series.

13. A network based vision system operable to monitor a plurality of object detection zones about a vehicle traveling over a roadway for presence or location of objects about the vehicle, the system comprising:

a plurality of image acquisition devices for acquiring multiple fields of view of a scene surrounding the vehicle, each device having programmable settings, connected to transmit data and receive instructions, the image acquisition devices configurable into multiple series to detect, classify, or determine the position of, an object in one of the detection zones,

a control unit connected through the network to receive data from the image acquisition devices and control the shape of a first of the object detection zones based on operation of image acquisition devices in a first of the series,

the control unit including a programmable microprocessor, memory, and data storage containing data and executable software which runs on the microprocessor, the memory and storage operatively connected to the microprocessor, the storage including a non-transitory computer readable medium containing program instructions representing software executable on the processor, which instructions, when executed by the processor, cause the control unit to perform a sequence of steps based on an association between each object detection zone and a series of the image acquisition devices by which the control unit;

receives and applies information indicative of a detected object type or detected location from one or more of the image acquisition devices, and automatically adjusts an optical setting or a pointing direction of one or more of

the image acquisition devices in the first series to alter an object detection response characteristic of the system in a portion of the first object detection zone and thereby modify the range of distances that a portion of the first object detection zone extends away from the vehicle.

14. The vision system of claim 13 where the adjustment alters the size and range of object detection in the portion of the first object detection zone.

15. The vision system of claim 13 where the adjustment in the optical setting or pointing direction of an image acquisition device in the first series is responsive to a change in vehicle operation or a change in roadway situation about the vehicle.

16. The vision system of claim 13 where the adjustment in the optical setting or pointing direction of an image acquisition device in the first series modifies the ability of the system to detect, classify or track an object.

17. The vision system of claim 13 where the adjustment in the optical setting or pointing direction of an image acquisition device in the first series modifies a characteristic of the system response due to presence of an object in the portion of the first object detection zone.

18. The vision system of claim 17 where the adjustment changes an optical setting which results in a change, in the portion of the first object detection zone, of:

captured field of view of a scene in the first object detection zone,

field of view angle of an image acquisition device assigned to the first object detection zone,

depth of field of an image acquisition device assigned to the detection zone, or

ability to detect an object positioned in the zone based on image size of the object.

19. The vision system of claim 13 where the program instructions cause the control unit to apply the information to generate an overlay indicative of object type or position on a real time map of the roadway.

20. A network based vision system operable to monitor a plurality of object detection zones about a vehicle traveling over a roadway for presence or location of objects about the vehicle, the system comprising:

to detect, classify, or determine the position of, an object in one of the detection zones, and

a control unit connected through the network to receive data from the image acquisition devices via a serial bus network or a video data bus,

the control unit including a multi-channel digital video receiver, a programmable microprocessor, memory, and data storage containing data and executable software which runs on the microprocessor, the memory and storage operatively connected to the microprocessor, the storage including a non-transitory computer readable medium containing program instructions representing software executable on the processor, which instructions, when executed by the processor, cause the control unit to perform a sequence of steps based on an association between each object detection zone and a series of the image acquisition devices by which the control unit;

receives and applies non-video information indicative of a detected object type or detected location from one or more of the image acquisition devices to generate an overlay indicative of object type or position on a real time map of the roadway; and

,

when a determination is made, based on execution of the program instructions, that selection criteria are met, the input selection is changed from a first mode, in which non-video information is received, to a second mode to begin assimilating multi-channel frames of camera video data.

21. The vision system of claim 20 where the data feed comprising non-video information is received via a CAN transceiver in the control unit and the multi-channel frames of camera video data are received via a High Speed video data bus compliant with an IEEE1394 serial bus architecture.