Secure and private tracking across multiple cameras

• US 9,977,991 B2
• Filed: 04/22/2015
• Issued: 05/22/2018
• Est. Priority Date: 12/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chu, Chun-Te, Jung, Jaeyeon, Liu, Zicheng, Mahajan, Ratul
• Primary Examiner(s): Beck, Leron
• Application Number:
  US14/693,791
• Publication Number:
  US 20150227814A1
• Time in Patent Office: 1,126 Days
• Field of Search: 348159
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 7/02 Comparing digital values G06F7/06, G06F7/22, G06F7/38 take precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A technique for multi-camera object tracking is disclosed that preserves privacy of imagery from each camera or group of cameras. This technique uses secure multi-party computation to compute a distance metric across data from multiple cameras without revealing any information to operators of the cameras except whether or not an object was observed by both cameras. This is achieved by a distance metric learning technique that reduces the computing complexity of secure computation while maintaining object identification accuracy.

Claims

1. A system comprising:

one or more processing elements;

an object detection module communicatively coupled to or integrated with the one or more processing elements and configured to detect a first object when the first object is in view of a first video camera associated with a first site having a first privacy boundary that encompasses the first video camera;

a preprocessing module communicatively coupled to or integrated with the one or more processing elements and configured to perform local processing of at least a portion of the first video to form a first secure input;

a circuit generator communicatively coupled to or integrated with the one or more processing elements and configured to generate a function, wherein inputs to the function comprise the first secure input from the first site and a second secure input from a second site having a second privacy boundary that encompasses a second video camera and that does not overlap with the first privacy boundary, wherein the first privacy boundary prevents a user at the first site from obtaining video captured by the second video camera at the second site and the second privacy boundary prevents a user at the second site from obtaining video captured by the first video camera at the first site, the second secure input generated by processing a second object that was in view of the second video camera associated with the second site;

a match query module communicatively coupled to or integrated with the one or more processing elements and configured to send a query to the second site to request participation in solving function; and

a joint computation module communicatively coupled to or integrated with the one or more processing elements and configured to solve the function with the second site to determine if the first object is the same as the second object.

2. The system of claim 1, wherein a first feature vector is generated from the first object, and a second feature vector is generated from the second object, the first and second feature vectors comprising at least one of color histograms or texture histograms.

3. The system of claim 1, wherein the function comprises a Boolean circuit.

4. The system of claim 1, further comprising a database communicatively coupled to or integrated with the one or more processing elements and configured to store a first feature vector generated from the first object for later comparison.

5. The system of claim 1, further comprising an alert module communicatively coupled to or integrated with the one or more processing elements and configured to generate an alert if the first object is the same as the second object.

6. The system of claim 1, wherein at least one of the preprocessing module, the circuit generator, the match query module, or the joint computation module is implemented in a hardware logic component.

7. A system comprising:

one or more processors; and

computer-readable storage including a plurality of programming instructions that are executable by the one or more processors to;

obtain an indication of a first object viewed by a first video camera within a first privacy boundary;

process at least a portion of the indication of the first object to form a first input;

issue a match evaluation request to a peer computing system having a second privacy boundary that encompasses a second video camera and does not overlap with the first privacy boundary, wherein the first privacy boundary prevents a user within the first privacy boundary from obtaining video captured by the second video camera at the within the second privacy boundary, and the second privacy boundary prevents a user within the second privacy boundary from obtaining video captured by the first video camera within the first privacy boundary;

jointly compute, by the computing system and the peer computing system, at least a portion of a distance function over the first input and a second input derived by the peer computing system from a second object viewed by the second video camera; and

characterize, based at least in part on an output of the distance function, the first object and the second object as the same or different.

8. The system of claim 7, wherein the distance function comprises determining a histogram intersection function for the first input and the second input.

9. The system of claim 7, wherein the first input comprises a transformation of a first feature vector of the first object to a lower dimensionality vector.

10. The system of claim 9, wherein the transformation comprises Principal Component Analysis (PCA) and linear mapping.

11. The system of claim 7, wherein the programming instructions are further executable by the one or more processors to issue the match evaluation request when the first object matches an object of interest, enters the view of the first video camera, or leaves the view of the first video camera.

12. The system of claim 7, wherein the programming instructions are further executable by the one or more processors to send, as part of jointly computing the distance function, a Boolean circuit of at least a portion of the distance function to the peer computing system and wait for the peer computing system to compute an output to the Boolean circuit.

13. The system of claim 7, wherein the joint computation is performed without disclosing either input to another computing system.

14. A computer-implemented method for jointly computing a distance metric using secure two-party computing to improve tracking of an object between independent computing systems, each computing comprising a computer-readable memory storing a plurality of machine-readable instructions, and one or more hardware processors coupled to the computer-readable memory and configured to read instructions from the computer-readable memory to perform operations comprising:

obtaining, by a first computing system, a first private input;

solving, by the first computing system, a local computation portion of a distance function, the local computation portion computed within a first privacy boundary of a first entity that provided the first private input, the distance function comprising a Metric learning by Collapsing Classes (MCC) distance function transformed to a different vector space by a linear projection matrix and having a Euclidean distance function replaced with a histogram intersection function;

solving, jointly with a second entity having a second computing system that has also independently solved a same local computation portion of the distance function for a second private input, a joint computation portion of the distance function that takes the first private input and the second private input as inputs, the second entity being within a second privacy boundary that does not overlap with the first privacy boundary, wherein the first privacy boundary prevents a user of the first computing system from obtaining the second private input of the second computing system and the second computing system prevents a user of the second computing system from obtaining the first private input;

comparing a computed distance determined by the distance function to a threshold distance; and

when the computed distance is smaller than the threshold distance, categorizing the first private input as matching the second private input.

15. The method of claim 14, wherein the first private input and second private input both comprise one or more of video, still images, images of human faces, handwritten text, or biometric features.

16. The method of claim 14, wherein the joint computation portion of the distance function is constructed as a circuit.

17. The method of claim 14, wherein a first secure input is determined from the first private input, and is quantized into an integer value.

18. The method of claim 17, wherein a second secure input is determined from the second private input, and the joint computation portion takes the first secure input and the second secure input as inputs.

19. The method of claim 14, wherein the joint solving comprising a single process using a transmission control protocol (TCP) port opened by the second entity exclusively for comparisons against the second private input.