System and method for appearance search

• US 10,726,312 B2
• Filed: 12/05/2017
• Issued: 07/28/2020
• Est. Priority Date: 12/05/2016
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Corporation (Motorola Solutions, Inc.)
• Inventors: Butt, Richard, Chau, Alexander, Doumbouya, Moussa, Glozman, Levi, He, Lu, Lipchin, Aleksey, Marlatt, Shaun P., Sadanand, Sreemanananth, Saha, Mitul, Saptharishi, Mahesh, Hu, Yanyan
• Primary Examiner(s): Nguyen, Tuan H
• Application Number:
  US15/832,654
• Publication Number:
  US 20180157939A1
• Time in Patent Office: 966 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61

Abstract

There is provided an appearance search system comprising one or more cameras configured to capture video of a scene, the video having images of objects. The system comprises one or more processors and memory comprising computer program code stored on the memory and configured when executed by the one or more processors to cause the one or more processors to perform a method. The method comprises identifying one or more of the objects within the images of the objects. The method further comprises implementing a learning machine configured to generate signatures of the identified objects and generate a signature of an object of interest. The system further comprises a network configured to send the images of the objects from the camera to the one or more processors. The method further comprises comparing the signatures of the identified objects with the signature of the object of interest to generate similarity scores for the identified objects, and transmitting an instruction for presenting on a display one or more of the images of the objects based on the similarity scores.

Claims

1. An appearance search system comprising:

one or more cameras configured to capture video of a scene, the video having images of objects, wherein at least one of the one or more cameras is further configured to identify, using a first learning machine at the camera, one or more of the objects within the images of the objects;

one or more processors and memory comprising computer program code stored on the memory; and

a network configured to send images comprising the one or more identified objects from the camera to the one or more processors,

wherein the computer program code is configured when executed by the one or more processors to cause the one or more processors to perform a method comprising;

generating, as output from a second learning machine, one or more signatures of the respective one or more identified objects and a signature of an object of interest;

comparing the one or more signatures of the respective one or more identified objects with the signature of the object of interest to generate one or more similarity scores for the respective one or more identified objects; and

transmitting an instruction for presenting on a display one or more of the images of the one or more identified objects based on the one or more similarity scores.

2. The system of claim 1, wherein the first and second learning machines comprise neural networks.

3. The system of claim 2, wherein the first and second learning machines comprise convolutional neural networks.

4. The system of claim 1, further comprising one or more graphics processing units for running the first and second learning machines.

5. The system of claim 1, wherein the second learning machine is implemented by one or more processors located on a server.

6. A non-transitory computer-readable medium having stored thereon computer program code executable by one or more processors and configured when executed by the one or more processors to cause the one or more processors to perform a method comprising:

receiving images of one or more identified objects, the one or more identified objects having been identified by a first learning machine at a video camera that captured video of a scene, the video having the one or more images;

generating, as output from a second learning machine, one or more signatures of the respective one or more identified objects, and a signature of an object of interest;

generating one or more similarity scores for the respective one or more identified objects by comparing the one or more signatures of the respective one or more identified objects with the signature of the object of interest; and

presenting on a display one or more of the images of the one or more identified objects based on the one or more similarity scores.

7. The non-transitory computer-readable medium of claim 6, wherein the first and second learning machines comprise neural networks.

8. The non-transitory computer-readable medium of claim 7, wherein the neural networks comprise convolutional neural networks.

9. The non-transitory computer-readable medium of claim 6, wherein the second learning machine is implemented by one or more processors located on a server.

10. A system comprising:

one or more cameras configured to capture video of a scene; and

one or more processors and memory comprising computer program code stored on the memory and configured when executed by the one or more processors to cause the one or more processors to perform a method comprising;

extracting chips from the video, wherein the chips comprise images of objects;

for each of at least one of the chips;

determining a confidence level for the chip; and

if the confidence level does not meet a confidence requirement, then;

identifying, using a first learning machine, multiple objects within the chip;

dividing, using the first learning machine, the chip into multiple divided chips, each of the divided chips comprising at least a portion of one of the identified objects; and

generating, using a second learning machine, respective feature vectors from the multiple divided chips;

or

if the confidence level meets the confidence requirement, then using the second learning machine to generate a feature vector from the chip.

11. The system of claim 10, wherein the at least one chip comprises at least one padded chip, wherein each padded chip comprises a first image portion of an image frame of the video.

12. The system of claim 11, wherein the at least one chip further comprises at least one non-padded chip, wherein each non-padded chip comprises a second image portion of an image frame of the video, the second image portion being smaller than the first image portion.

13. A method comprising:

capturing video of a scene, the video having images of objects;

identifying, using a first learning machine at a video camera, one or more of the objects within the images of the objects;

generating, as output from a second learning machine, one or more signatures of the respective one or more identified objects and a signature of an object of interest;

generating one or more similarity scores for the respective one or more identified objects by comparing the one or more signatures of the respective one or more identified objects with the signature of the object of interest; and

presenting on a display one or more of the images of the one or more identified objects based on the one or more similarity scores.

14. The method of claim 13, wherein the first and second learning machines comprise neural networks.

15. The method of claim 14, wherein the neural networks comprise convolutional neural networks.

16. The method of any one of claim 13, wherein the second learning machine is implemented by one or more processors located on a server.