Secure and private tracking across multiple cameras

• US 10,181,090 B2
• Filed: 04/20/2018
• Issued: 01/15/2019
• Est. Priority Date: 12/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chu, Chun-Te, Jung, Jaeyeon, Liu, Zicheng, Mahajan, Ratul
• Primary Examiner(s): Beck, Leron
• Application Number:
  US15/958,820
• Publication Number:
  US 20180239985A1
• Time in Patent Office: 270 Days
• Field of Search: 348159
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 7/02 Comparing digital values G06F7/06, G06F7/22, G06F7/38 take precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A technique for multi-camera object tracking is disclosed that preserves privacy of imagery from each camera or group of cameras. This technique uses secure multi-party computation to compute a distance metric across data from multiple cameras without revealing any information to operators of the cameras except whether or not an object was observed by both cameras. This is achieved by a distance metric learning technique that reduces the computing complexity of secure computation while maintaining object identification accuracy.

Claims

1. A system, comprising:

computer-readable storage media having stored thereon instructions executable to cause the system to perform operations;

one or more hardware processors coupled to the computer-readable storage media and configured to read instructions from the computer-readable storage media media to cause the system to perform operations comprising;

obtaining by a remote server a plurality of images from a first video camera within a first privacy boundary, the plurality of images being provided to the remote server through a first communications network, the first privacy boundary being maintained by a secure configuration of the remote server, wherein the first privacy boundary prevents a user associated with the first privacy boundary from obtaining a plurality of images captured by a second video camera within a second privacy boundary;

detecting, by the remote server, an indication of at least one object in the plurality of video frames from the first video camera;

generating a first feature vector for the at least one object in the plurality of video frames from the first video camera;

receiving a request for a match evaluation for a second object at the remote server from an issuing computing system; and

determining if the at least one object in the plurality of video frames from the first video camera matches the second object using a distance function based upon the first feature vector and a second feature vector for the second object.

2. The system of claim 1, wherein the distance function comprises determining a histogram intersection function for the first feature vector and the second feature vector.

3. The system of claim 1, wherein the operations further comprise transforming the first feature vector of the at least one object in the plurality of video frames from the first video camera to a lower dimensionality vector.

4. The system of claim 3, wherein the transformation comprises Principal Component Analysis (PCA) and linear mapping.

5. The system of claim 1, wherein the operations further comprise:

sending, as a part of a joint computing for the distance function, a Boolean circuit of at least a portion of the distance function to a peer computing system; and

waiting for the peer computing system to compute an output to the Boolean circuit.

6. The system of claim 1, wherein the first feature vector and the second feature vector comprise at least one of color histograms or texture histograms.

7. The system of claim 1, wherein the operations further comprise generating an alert if the at least one object in the plurality of video frames from the first video camera is the same as the second object.

8. A computer-implemented method, comprising:

obtaining by a remote server a plurality of images from a first video camera within a first privacy boundary, the plurality of images being provided to the remote server through a first communications network, the first privacy boundary being maintained by a secure configuration of the remote server, wherein the first privacy boundary prevents a user associated with the first privacy boundary from obtaining a plurality of images captured by a second video camera within a second privacy boundary;

detecting, by the remote server, an indication of at least one object in the plurality of video frames from the first video camera;

generating a first feature vector for the at least one object in the plurality of video frames from the first video camera;

receiving a request for a match evaluation for a second object at the remote server from an issuing computing system; and

determining if the at least one object in the plurality of video frames from the first video camera matches the second object using a distance function based upon the first feature vector and a second feature vector for the second object.

9. The computer-implemented method of claim 8, wherein the distance function comprises determining a histogram intersection function for the first feature vector and the second feature vector.

10. The computer-implemented method of claim 8, further comprising transforming the first feature vector of the at least one object in the plurality of video frames from the first video camera to a lower dimensionality vector.

11. The computer-implemented method of claim 10, wherein the transformation comprises Principal Component Analysis (PCA) and linear mapping.

12. The computer-implemented method of claim 8, further comprising:

sending, as a part of a joint computing for the distance function, a Boolean circuit of at least a portion of the distance function to a peer computing system; and

waiting for the peer computing system to compute an output to the Boolean circuit.

13. The computer-implemented method of claim 8, wherein the first feature vector and the second feature vector comprise at least one of color histograms or texture histograms.

14. The computer-implemented method of claim 8, further comprising generating an alert if the at least one object in the plurality of video frames from the first video camera is the same as the second object.

20. The computer-implemented method of claim 8, further comprising generating an alert if the at least one object in the plurality of video frames from the first video camera is the same as the second object.

15. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause machine to perform operations comprising:

obtaining by a remote server a plurality of images from a first video camera within a first privacy boundary, the plurality of images being provided to the remote server through a first communications network, the first privacy boundary being maintained by a secure configuration of the remote server, wherein the first privacy boundary prevents a user associated with the first privacy boundary from obtaining a plurality of images captured by a second video camera within a second privacy boundary;

detecting, by the remote server, an indication of at least one object in the plurality of video frames from the first video camera;

generating a first feature vector for the at least one object in the plurality of video frames from the first video camera;

receiving a request for a match evaluation for a second object at the remote server from an issuing computing system; and

determining if the at least one object in the plurality of video frames from the first video camera matches the second object using a distance function based upon the first feature vector and a second feature vector for the second object.

16. The computer-implemented method of claim 15, wherein the distance function comprises determining a histogram intersection function for the first feature vector and the second feature vector.

17. The computer-implemented method of claim 15, further comprising transforming the first feature vector of the at least one object in the plurality of video frames from the first video camera to a lower dimensionality vector.

18. The computer-implemented method of claim 17, wherein the transformation comprises Principal Component Analysis (PCA) and linear mapping.

19. The computer-implemented method of claim 15, further comprising:

sending, as a part of a joint computing for the distance function, a Boolean circuit of at least a portion of the distance function to a peer computing system; and

waiting for the peer computing system to compute an output to the Boolean circuit.