Real-time object re-identification in a multi-camera system using edge computing

• US 10,579,880 B2
• Filed: 08/31/2017
• Issued: 03/03/2020
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Original Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Inventors: Khorsandi, Rahman, Amano, Jun
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Brumfield, Shanika M
• Application Number:
  US15/693,339
• Publication Number:
  US 20190065858A1
• Time in Patent Office: 915 Days
• Field of Search: None
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/431 Frequency domain transformation; Autocorrelation
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A video surveillance system with real-time object re-identification capabilities, which employs an object re-identification algorithm and an edge computing architecture. An operator monitors video images from the multiple cameras, and when a target object is observed, a target image containing the object is transmitted to all video cameras for object re-identification. Each video camera has dedicated processing circuitry that performs an object re-identification algorithm to identify the target in video images captured by that camera in real time. The algorithm calculates a frequency domain similarity measure between the target image and test images captured by the camera. The similarity measure in the frequency domain is calculated as a dot product of the 1D discrete Fourier transforms of the target image data and of the test image data. The multiple cameras also transmit object re-identification results to each other to achieve more efficient and intelligent object re-identification.

Claims

1. A video surveillance system comprising:

a plurality of video cameras located at different physical locations, each video camera comprising;

an image capture device;

dedicated image data processing circuitry;

a memory; and

a communication interface device;

a monitoring device connected to the plurality of video cameras via a communication network;

wherein each video camera is configured to transmit image data captured by the image capture device of the video camera to the monitoring device via the communication interface device of the video camera,

wherein the dedicated image data processing circuitry of each video camera is configured to perform an object re-identification process which includes;

receiving, from the monitoring device via the communication interface device, target image data x representing an image containing a target object, wherein the target image data is image data that has been transmitted to the monitoring device from another one of the plurality of video cameras;

obtaining test image data y from a frame of image captured by the image capture device of the video camera;

computing a one-dimensional discrete Fourier transform {circumflex over (x)} of the target image data x;

computing a one-dimensional discrete Fourier transform ŷ

of the test image data y;

computing a spatial frequency domain similarity measure S between the target image data x and the test image data y which is proportional to a dot product of the discrete Fourier transform {circumflex over (x)} of the target image data x and the discrete Fourier transform ŷ

of the test image data y;

when the spatial frequency domain similarity measure S is greater than a predetermined threshold value, determining that the target object has been re-identified; and

when the object is re-identified, transmitting, via the communication interface device, an object re-identification result to the monitoring device.

2. The system of claim 1, wherein each video camera is further configured to transmit the object re-identification result to all other video cameras.

3. The system of claim 1, wherein the dedicated image data processing circuitry includes an FPGA (field programmable gate array) or ASIC (application specific integrated circuit) circuitry.

4. A method implemented in a video surveillance system, comprising:

capturing video image data by each of a plurality of video cameras located at different physical locations;

transmitting the video image data from each video camera to a monitoring device;

at the monitoring device, monitoring the video image data to identify a target object in the video image data from one of the video cameras;

generating target image data x representing an image containing the target object using the monitoring device;

transmitting the target image data x from the monitoring device to each of the plurality of video cameras;

each of the video cameras, using its dedicated image data processing circuitry, performing an object re-identification process which includes;

continuing to capture video images;

obtaining test image data y from a frame of captured video image;

computing a one-dimensional discrete Fourier transform {circumflex over (x)} of the target image data x;

computing a one-dimensional discrete Fourier transform ŷ

of the test image data y;

computing a spatial frequency domain similarity measure S between the target image data x and the test image data y which is proportional to a dot product of the discrete Fourier transform {circumflex over (x)} of the target image data x and the discrete Fourier transform ŷ

of the test image data y;

when the spatial frequency domain similarity measure S is greater than a predetermined threshold value, determining that the target object has been re-identified; and

when the object is re-identified, transmitting an object re-identification result to the monitoring device.

5. The method of claim 4, wherein the step of obtaining test image data y from a frame of captured video image includes using a sliding window on the image frame to obtain a plurality of test image data y, and

wherein the object re-identification process further comprises:

computing a one-dimensional discrete Fourier transform ŷ

of each of the test image data y;

computing a plurality of spatial frequency domain similarity measures S between the target image data x and the corresponding plurality of test image data y, each spatial frequency domain similarity measures S being proportional to a dot product of the discrete Fourier transform {circumflex over (x)} of the target image data x and the discrete Fourier transform ŷ

of the corresponding test image data y; and

determining whether each of the spatial frequency domain similarity measure S is greater than a predetermined threshold value.

6. The method of claim 5, wherein the object re-identification process is performed in real time as capture of the video images.

7. The method of claim 4, further comprising, by each of the video cameras:

when the object is re-identified, transmitting the object re-identification result to at least some other ones of the plurality of video cameras.

8. The method of claim 7, wherein each of the video cameras that have received the object re-identification result from another video camera performs the object re-identification process based on the received object re-identification result.