Video based matching and tracking by analyzing one or more image abstractions

• US 8,600,172 B2
• Filed: 03/16/2011
• Issued: 12/03/2013
• Est. Priority Date: 03/16/2011
• Status: Active Grant

• Current Assignee: Johnson Controls Tyco IP Holdings LLP (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Inventors: Sengupta, Kuntal, Westmacott, Ian, Ince, Serdar
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): CHEN, XUEMEI G
• Application Number:
  US13/049,527
• Publication Number:
  US 20120237082A1
• Time in Patent Office: 993 Days
• Field of Search: 340/573.1, 345/419, 348/207.99, 375/346, 382/103, 382/107, 382/118, 382/131, 382/170, 382/199, 382/209, 382/294, 704/251, 706/12, 707/741, 707/748, 713/186
• US Class Current: 382/218
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

An analytical device is disclosed that analyzes whether a first image is similar to (or the same as) as a second image. The analytical device analyzes the first image by combining at least a part (or all) of the first image with at least a part (or all) of the second image, and by analyzing at least a part (or all) of the combined image. Part or all of the combination may be analyzed with respect to the abstraction of the first image and/or the abstraction of the second image. The abstraction may be based on a Bag of Features (BoF) description, based on a histogram of intensity values, or based on other types of abstraction methodologies. The analysis may involve comparing one or more aspects of the combination (such as the entropy or randomness of the combination) with the one or more aspects of the abstracted first image and/or abstracted second image. Based on the comparison, the analytical device may determine whether the first image is similar to or the same as the second image. The analytical device may work with a variety of images in a variety of applications including a video tracking system, a biometric analytic system, or a database image analytical system.

Claims

1. A method of analyzing a known image abstraction and a query image abstraction to determine whether a query image is similar to a known image, the method comprising:

accessing the known image abstraction, the known image abstraction being based on part or all of the known image;

accessing the query image abstraction, the query image abstraction being based on part or all of the query image, the known image and the query image being of a same type;

combining at least a part of the known image abstraction with at least a part of the query image abstraction to generate a combined image abstraction;

comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction; and

determining whether the query image is similar to the known image based on comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction,

wherein the known image abstraction comprises a known covariance matrix;

wherein the query image abstraction comprises a query covariance matrix; and

wherein combining at least a part of the known image abstraction with at least a part of the query image abstraction comprises combining the known covariance matrix with the query covariance matrix.

2. The method of claim 1, wherein the known image abstraction comprises a known Bag of Features (BoF) description, the known BoF description representing a cloud of points in multidimensional feature space;

and

the query image abstraction comprises a query BoF description.

3. The method of claim 1, wherein comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction includes comparing entropy of the combined image abstraction with entropy of one or both of the known image abstraction and the query image abstraction.

4. The method of claim 1, wherein the known image abstraction comprises an object abstraction associated with an object being tracked by a video tracking system;

and

wherein the query image abstraction is associated with an image patch in a video frame.

5. The method of claim 4, further comprising:

performing foreground/background segmentation to generate a list of multiple image patches in the video frame;

generating an abstraction for each of the multiple image patches;

iteratively combining the object abstraction with each of the abstractions for the multiple image patches to generate combined abstractions; and

analyzing the combined abstractions to determine which of the multiple image patches is most similar to the object abstraction.

6. The method of claim 5, wherein the object abstraction is updated based on the image patch most similar to the object abstraction.

7. The method of claim 6, wherein the object abstraction is fused with the image patch most similar to the object abstraction.

8. The method of claim 6, wherein the object abstraction comprises a multimodal abstraction;

and

wherein one mode of the multimodal abstraction is modified using the image patch most similar to the object abstraction.

9. The method of claim 4, wherein the object abstraction comprises an object cloud of points;

wherein each of the multiple image patches comprise a respective cloud of points; and

wherein generating the combined image abstractions comprises fusing the object cloud of points with each of the respective clouds of points.

10. The method of claim 1, wherein comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction comprises generating a score indicative of randomness of the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction;

and

wherein determining whether the query image is similar to the known image comprises comparing the score with a predetermined amount.

11. The method of claim 3, wherein comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction further comprises:

determining entropy of the combined image abstraction;

determining entropy of the known image abstraction and entropy of the query image abstraction;

combining the entropy of the known image abstraction and the entropy of the query image abstraction to generate an entropy of underlying images; and

generating a dissimilarity metric by subtracting the entropy of the underlying images from the entropy of the combined image abstraction,

wherein determining whether the query image is similar to the known image comprises using the dissimilarity metric to determine whether the query image is similar to the known image.

12. The method of claim 1, wherein the query image abstraction comprises a first query image abstraction based on a first part of the query image;

wherein a second query image abstraction is based on a second part of the query image, the first part of the query image being different from the second part of the query image;

wherein combining at least a part of the known image abstraction with at least a part of the query image abstraction to generate a combined image abstraction comprises;

generating a first combined image abstraction by combining at least a part of the known image abstraction with at least a part of the first query image abstraction;

generating a second combined image abstraction by combining at least a part of the known image abstraction with at least a part of the second query image abstraction;

wherein comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction comprises;

generating a first score indicative of randomness of the first combined image abstraction with at least a part of one or both of the known image abstraction and the first query image abstraction;

generating a second score indicative of randomness of the second combined image abstraction with at least a part of one or both of the known image abstraction and the second query image abstraction;

wherein determining whether the query image is similar to the known image comprises;

comparing the first score with the second score; and

determining whether the first part of the query image or the second part of the query image is similar to the known image based on comparing the first score with the second score.

13. A method for tracking an object in multiple image frames, the method comprising:

accessing a current image frame;

performing foreground/background segmentation to generate at least one image patch in the current image frame;

abstracting the at least one image patch into a query covariance matrix, the query covariance matrix representing a cloud of points in multidimensional feature space;

accessing a known covariance matrix indicative of the object, the known covariance matrix being based, at least in part on one or more image frames previous to the current image frame;

combining at least a part of the query covariance matrix with the known covariance matrix; and

analyzing the combined covariance matrix in order to determine whether the at least one image patch is similar to the object.

14. The method of claim 13, wherein the multiple image frames comprises successive frames in a video;

and

wherein the query covariance matrix is computed with an interleaved integral image approach, with interleaving of the M(M−

1)/2 integral images being performed, and computing (k.

     1)th location value of two integral images independently of each other, where M is a number of features.

15. The method of claim 13, wherein performing foreground/background segmentation generates image patches in the current image frame;

wherein analyzing the combined covariance matrix determines which of the image patches is most similar to the object; and

wherein the known covariance matrix is updated based on the image patch most similar to the object.

16. The method of claim 15, wherein the known covariance matrix is fused with the image patch most similar to the object abstraction.

17. The method of claim 15, wherein the known covariance matrix is based on a multimodal distribution;

and

wherein one mode of the multimodal distribution is modified using the image patch most similar to the known covariance matrix.

18. An apparatus for analyzing a known image abstraction and a query image abstraction to determine whether a query image is similar to a known image, the apparatus comprising:

at least one memory configured to store the known image abstraction and the query image abstraction, the known image abstraction being based on part or all of the known image, the query image abstraction being based on part or all of the query image, the known image and the query image being of a same type; and

a controller in communication with the memory and configured to;

combine at least a part of the known image abstraction with at least a part of the query image abstraction to generate a combined image abstraction;

compare the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction; and

determine whether the query image is similar to the known image based on comparing the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction,

wherein the known image abstraction comprises a known covariance matrix;

wherein the query image abstraction comprises a query covariance matrix; and

wherein the controller is configured to combine at least a part of the known image abstraction with at least a part of the query image abstraction by combining the known covariance matrix with the query covariance matrix.

19. The apparatus of claim 18, wherein the known image abstraction comprises a known Bag of Features (BoF) description, the known BoF description representing a cloud of points in multidimensional feature space;

and

the query image abstraction comprises a query BoF description.

20. The apparatus of claim 18, wherein the controller is configured to compare the combined image abstraction with at least a part of one or both of the known image abstraction and the query image abstraction by comparing entropy of the combined image with entropy of one or both of the known image abstraction and the query image abstraction.

21. The apparatus of claim 18, wherein the known image abstraction comprises an object abstraction associated with an object being tracked by a video tracking system;

wherein the query image abstraction is associated with an image patch in a video frame; and

wherein the controller is further configured to

perform foreground/background segmentation to generate a list of multiple image patches in the video frame;

generate an abstraction for each of the multiple image patches;

iteratively combine the object abstraction with each of the abstractions for the multiple image patches to generate combined abstractions; and

analyze the combined abstractions to determine which of the multiple image patches is most similar to the object abstraction.

22. The apparatus of claim 21, wherein the object abstraction is updated based on the image patch most similar to the object abstraction.

23. An apparatus for tracking an object in multiple image frames, the apparatus comprising:

at least one memory configured to store a current image frame and an object abstraction of the object, the object abstraction comprising an object covariance matrix based, at least in part on one or more image frames previous to the current image frame, the object covariance matrix representing a cloud of points in multidimensional feature space; and

a controller in communication with the memory and configured to;

access the current image frame;

perform foreground/background segmentation to generate at least one image patch in the current image frame;

abstract the at least one image patch into a query covariance matrix to generate abstracted at least one image patch;

combine at least a part of the query covariance matrix with the known covariance matrix; and

analyze the combined covariance matrix in order to determine whether the at least one image patch is similar to the object.

24. The apparatus of claim 23, wherein the covariance matrix is defined based on a Bag of Features (BoF) description;

wherein rows in a BoF matrix correspond to points in the respective image patch; and

wherein columns of the BoF matrix comprise the x-location, y-location and the grayscale value of the points.