System And Method For Object And Event Identification Using Multiple Cameras

• US 20140333775A1
• Filed: 05/09/2014
• Published: 11/13/2014
• Est. Priority Date: 05/10/2013
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Naikal, Nikhil, Lajevardi, Pedram
• Granted Patent: US 9,665,777 B2
• Time in Patent Office: Days
• US Class Current: 348/159
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system for identifying objects and events of interest uses one or more cameras with image processing capabilities. The system is includes multiple cameras configured to perform image processing of a scene from multiple angles to extract and transmit meta-data corresponding to objects or people in the scene. The cameras transmit the meta-data to a processing station that is configured to process the stream of data over time to detect objects and events of interest to alert monitoring personnel of objects or events in the scene.

Claims

1. A surveillance system comprising:

a plurality of cameras located in a plurality of positions to record a scene, each camera further comprising;

a sensor configured to generate video data of the scene comprising a series of frames;

a first network device configured to transmit the video data and feature vectors associated with the video data to a processing station; and

a feature extraction processor operatively connected to the sensor and the network device, the feature extraction processor being configured to;

identify a plurality of feature vectors in video data generated by the sensor;

transmit only the plurality of feature vectors to the processing station with the first network device in a first operating mode; and

transmit the video data to the processing station with the first network device in a second operating mode only in response to a request for the video data from the processing station; and

the processing station further comprising;

a second network device;

a video output device; and

a processor operatively connected to the second network device and the video output device, the processor being configured to;

receive the plurality of feature vectors generated by each camera in the plurality of cameras with the second network device;

identify an object and motion of the object in the scene with reference to the plurality of feature vectors received from at least two of the plurality of cameras;

identify an event corresponding to the motion of the object in the scene with reference to a predetermined database of events;

generate a request for transmission of the video data from at least one camera in the plurality of cameras; and

generate a graphical display of the video data from the at least one camera with the video output device to display the object associated with the event.

2. The surveillance system of claim 1, feature extraction processor in each of the plurality of cameras being further configured to:

identify a portion of one frame in the video data that corresponds to a background of the scene; and

identify the plurality of features in the video data only in portions of the one frame that do not correspond to the background of the scene.

3. The surveillance system of claim 2, the feature extraction processor in each camera being further configured to:

generate a bounding box corresponding a portion of the one frame that does not correspond to the background of the scene;

resize the portion of the one frame in the bounding box to a generate a fixed resolution image of the portion of the one frame included in the bounding box; and

identify at least one feature vector in the plurality of feature vectors with reference to the fixed resolution image.

4. The surveillance system of claim 3, the feature extraction processor being further configured to:

identify a plurality of image gradients in the fixed resolution image; and

generate the feature vector including a histogram of gradients corresponding to the plurality of image gradients.

5. The surveillance system of claim 1, the feature extraction processor in a first camera in the plurality of cameras being further configured to:

identify a first feature vector for a first frame in the video data;

transmit the first feature vector to the processing station with the first network device;

identify a second feature vector for a second frame in the video data, the second frame occurring after the first frame; and

transmit a sparse feature vector corresponding to only a portion of the second feature vector that is different than the first feature vector to the processing station with the first network device.

6. The surveillance system of claim 4, the processor in the processing station being further configured to:

receive the first feature vector from the first camera;

receive a sparse feature vector from a second camera in the plurality of cameras, the sparse feature vector including only portions of a feature vector generated in the second camera that have changed since generation of an earlier feature vector, the earlier feature vector being generated approximately concurrently to the first feature vector from the first camera; and

generate a full feature vector corresponding to the sparse feature vector from the second camera with reference to the first feature vector from the first camera.

7. The surveillance system of claim 1, the processor in the processing station being further configured to:

identify occurrence or non-occurrence of an event in each plurality of feature vectors from each of the plurality of cameras individually with reference to the plurality of feature vectors corresponding to a plurality of key-frames from each of the plurality of cameras corresponding to the motion of the object during a single time period;

identify the event in response to an identification of an occurrence of the event from a majority of the plurality of cameras.

8. The surveillance system of claim 1, the processor in the processing station being further configured to:

receive a first plurality of feature vectors from a first camera corresponding to a first frame of video data;

identify that the first plurality of feature vectors from the first camera correspond to a first key-frame for an event in the predetermined database of events;

receive a second plurality of feature vectors from a second camera corresponding to a second frame of video data; and

identify that the second plurality of feature vectors from the second camera correspond to a second key-frame from the event; and

generate a score corresponding to a deformation between the first plurality of feature vectors in the first frame and the second plurality of feature vectors in the second frame; and

identify occurrence or non-occurrence of the event with reference to the score.

9. The surveillance system of claim 8, the processor in the processing station being further configured to:

identify a first location of an object corresponding to the first plurality of feature vectors that are extracted from the first frame of video data;

identify a second location of the object corresponding to the second plurality of feature vectors that are extracted from the second frame of video data;

perform a homographic transformation to identify a distance between the first location of the object and the second location of the object with reference to a ground plane that is present in both the first frame of video data and the second frame of video data; and

generate the score corresponding to the deformation between the first plurality of feature vectors in the first frame and the second plurality of feature vectors in the second frame with reference to the distance between the first location and the second location.

10. A method for surveillance of a scene comprising:

generating with a sensor in a first camera first video data of the scene, the first video data comprising a first series of frames;

identifying with a feature extraction processor in the first camera a first plurality of feature vectors in the first video data;

transmitting with a network device in the first camera only the first plurality of feature vectors to a processing station in a first operating mode;

transmitting with the network device in the first camera the first video data to the processing station in a second operating mode only in response to a request for the first video data from the processing station;

generating with another sensor in a second camera second video data of the scene, the second video data comprising a second series of frames and the second camera generating the second video data of the scene from a different position than the first camera;

identifying with another feature extraction processor in the second camera a second plurality of feature vectors in the second video data;

transmitting with another network device in the second camera only the second plurality of feature vectors to the processing station in the first operating mode;

transmitting with the other network device in the second camera the second video data to the processing station in the second operating mode only in response to a request for the second video data from the processing station;

receiving with another network device in the processing station the first plurality of feature vectors from the first camera and the second plurality of feature vectors from the second camera;

identifying with an event processor in the processing station an object and motion of the object in the scene with reference to the first and second plurality of feature vectors;

identifying with the event processor in the processing station an event corresponding to the motion of the object in the scene with reference to a predetermined database of events;

generating with the event processor in the processing station a request for transmission of the video data from at least one of the first camera and the second camera; and

generating with a video display device a graphical display of video data received from at least one of the first camera and the second camera with the video output device to display the object associated with the event.

11. The method of claim 10 further comprising:

identifying with the feature extraction processor in the first camera a portion of one frame in the first video data that corresponds to a background of the scene; and

identifying with the feature extraction processor in the first camera the first plurality of features in the first video data only in portions of the one frame that do not correspond to the background of the scene.

12. The method of claim 11 further comprising:

generating with the feature extraction processor in the first camera a bounding box corresponding a portion of the one frame that does not correspond to the background of the scene;

resizing with the feature extraction processor in the first camera the portion of the one frame in the bounding box to a generate a fixed resolution image of the portion of the frame included in the bounding box; and

identifying with the feature extraction processor in the first camera at least one feature vector in the first plurality of feature vectors with reference to the fixed resolution image.

13. The method of claim 12, the identification of the at least one feature vector further comprising:

identifying with the feature extraction processor in the first camera a plurality of image gradients in the fixed resolution image; and

generating with the feature extraction processor in the first camera the at least one feature vector including a histogram of gradients corresponding to the plurality of image gradients.

14. The method of claim 10 further comprising:

identifying with the feature extraction processor in the first camera a first feature vector for a first frame in the video data;

transmitting with the network device in the first camera the first feature vector to the processing station;

identifying with the feature extraction processor in the first camera a second feature vector for a second frame in the video data, the second frame occurring after the first frame; and

transmitting with the network device in the first camera a sparse feature vector corresponding to only a portion of the second feature vector that is different than the first feature vector to the processing station.

15. The method of claim 14 further comprising:

receiving with the event processor in the processing station the first feature vector from the first camera;

receiving with the event processor in the processing station another sparse feature vector from the second camera, the other sparse feature vector including only portions of a feature vector generated in the second camera that have changed since generation of an earlier feature vector in the second camera, the earlier feature vector being generated approximately concurrently to the first feature vector from the first camera; and

generating with the event processor in the processing station a full feature vector corresponding to the other sparse feature vector from the second camera with reference to the first feature vector from the first camera.

16. The method of claim 10, the identification of the event further comprising:

identifying with the event processor in the processing station occurrence or non-occurrence of an event in the first plurality of feature vectors and the second plurality of feature vectors individually with reference to the first plurality of feature vectors corresponding to a plurality of key-frames and the second plurality of feature vectors corresponding to the plurality of key-frames corresponding to the motion of the object during a single time period;

identify the event in response to an identification of an occurrence of the event from one or both of the first camera and second camera.

17. The method of claim 10, the identification of the event further comprising:

receiving with the event processor in the processing station the first plurality of feature vectors from the first camera corresponding to a first frame of the first video data;

identifying with the event processor that the first plurality of feature vectors correspond to a first key-frame for an event in the predetermined database of events;

receiving with the event processor the second plurality of feature vectors from the second camera corresponding to a second frame of the second video data; and

identifying with the event processor that the second plurality of feature vectors correspond to a second key-frame for the event; and

generating with the event processor in the processing station a score corresponding to a deformation between the first plurality of feature vectors in the first frame and the second plurality of feature vectors in the second frame; and

identifying with the event processor in the processing station occurrence or non-occurrence of the event with reference to the score.

18. The method of claim 17, the generation of the score further comprising:

identifying with the event processor in the processing station a first location of an object corresponding to the first plurality of feature vectors that are extracted from the first frame of video data;

identifying with the event processor in the processing station a second location of the object corresponding to the second plurality of feature vectors that are extracted from the second frame of video data;

performing with the event processor in the processing station a homographic transformation to identify a distance between the first location of the object and the second location of the object with reference to a ground plane that is present in both the first frame of video data and the second frame of video data; and

generating with the event processor in the processing station the score corresponding to the deformation between the first plurality of feature vectors in the first frame and the second plurality of feature vectors in the second frame with reference to the distance between the first location and the second location.