Pixel-level based micro-feature extraction

• US 10,049,293 B2
• Filed: 03/16/2017
• Issued: 08/14/2018
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni International, LLC (Blue Sail Medical Co., Ltd.)
• Inventors: Cobb, Wesley Kenneth, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath, Seow, Ming-Jung, Xu, Gang, Risinger, Lon W., Graham, Jeff
• Primary Examiner(s): Chu, Randolph I
• Application Number:
  US15/461,139
• Publication Number:
  US 20180032834A1
• Time in Patent Office: 516 Days
• Field of Search: None
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/473 using gradient analysis
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Techniques are disclosed for extracting micro-features at a pixel-level based on characteristics of one or more images. Importantly, the extraction is unsupervised, i.e., performed independent of any training data that defines particular objects, allowing a behavior-recognition system to forgo a training phase and for object classification to proceed without being constrained by specific object definitions. A micro-feature extractor that does not require training data is adaptive and self-trains while performing the extraction. The extracted micro-features are represented as a micro-feature vector that may be input to a micro-classifier which groups objects into object type clusters based on the micro-feature vectors.

Claims

1. A processor-implemented method for tracking an object in a video scene, the method comprising:

determining, via at least one processor, at least one pixel-level characteristic for a foreground patch in a first video frame from a plurality of video frames of a video scene, the foreground patch including a set of pixels containing a portion of scene foreground in the first video frame of the video scene;

determining, via the at least one processor, kinematics of an object in the video scene based at least in part on the at least one pixel-level characteristic;

generating, via the at least one processor, a micro-feature vector based at least in part on the kinematics of the object, the micro-feature vector corresponding to at least one micro-feature value;

updating, via the at least one processor, a neural network based at least in part on the micro-feature vector, the neural network including a plurality of clusters, each cluster in the plurality of clusters representing an object type;

classifying, via the at least one processor, the object in the first video frame of the video scene as a first object type based on the updated neural network, the first object type being represented by a first cluster in the plurality of clusters; and

tracking, via the at least one processor, the object in the video scene based on the micro-feature vector and the first object type.

2. The processor-implemented method of claim 1, wherein updating the neural network includes at least one of creating the first cluster representing the first object type or updating the first cluster in the plurality of clusters representing the first object type based on the micro-feature vector.

3. The processor-implemented method of claim 1, wherein updating the neural network further includes:

evaluating a rank for each cluster in the plurality of clusters relative to the micro-feature vector; and

evaluating each cluster in the plurality of clusters to determine if the object in the first video frame maps to at least one cluster in the plurality of clusters based on the ranking for each cluster.

4. The processor-implemented method of claim 1, wherein each cluster in the plurality of clusters in the neural network are created based on a plurality of micro-feature vectors generated via the at least one processor.

5. The processor-implemented method of claim 1, wherein generating the micro-feature vector includes evaluating appearance values of the pixels in the set of pixels to compute the at least one micro-feature value.

6. The processor-implemented method of claim 1, wherein the at least one micro-feature value includes a verticality of the foreground patch and the at least one pixel-level characteristic includes at least one per-pixel gradient value for the foreground patch.

7. The processor-implemented method of claim 1, wherein generating the micro-feature vector includes evaluating values representing the kinematics of the object.

8. A system, comprising:

a video input receiver configured to receive a plurality of video frames of a video scene;

at least one processor; and

a memory having processor-executable instructions and in communication with the at least one processor, the instructions in the memory comprising instructions for the at least one processor to;

determine at least one pixel-level characteristic for a foreground patch in a first video frame from the plurality of video frames of a video scene received by the video input receiver, the foreground patch including a set of pixels containing a portion of scene foreground in the first video frame of the video scene,

determine kinematics of an object in the video scene based at least in part on the at least one pixel-level characteristic,

generate a micro-feature vector based at least in part on at least on the kinematics of the object, the micro-feature vector corresponding to at least one micro-feature value;

update a neural network based at least in part on the micro-feature vector, the neural network including a plurality of clusters, each cluster in the plurality of clusters representing an object type,

classify the object in the first video frame of the video scene as a first object type based on the updated neural network, the first object type being represented by a first cluster in the plurality of clusters, and

track the object in the video scene based on the micro-feature vector and the first object type.

9. The system of claim 8, wherein the instructions for the at least one processor to update the neural network include instructions to:

(1) create the first cluster representing the first object type, and/or (2) update the first cluster in the plurality of clusters representing the first object type based on the micro-feature vector.

10. The system of claim 8, wherein the instructions for the at least one processor to update the neural network include instructions to:

evaluate a rank for each cluster in the plurality of clusters relative to the micro-feature vector, and

evaluate each cluster in the plurality of clusters to determine if the object in the first video frame maps to at least one cluster in the plurality of clusters based on the ranking for each cluster.

11. The system of claim 8, wherein the system is configured such that each cluster in the plurality of clusters in the neural network are created based on a plurality of micro-feature vectors generated via the at least one processor.

12. The system of claim 8, wherein the at least one micro-feature value includes a verticality of the foreground patch and the at least one pixel-level characteristic includes at least one per-pixel gradient value of the foreground patch.

13. The system of claim 8, wherein the instructions for the at least one processor to generate the micro-feature vector include instructions to evaluate values representing the kinematics of the object.

14. A non-transitory computer-readable storage medium storing computer-executable instructions for tracking an object in a video scene, the computer-executable instructions comprising instructions to:

determine at least one pixel-level characteristic for a foreground patch in a first video frame from a plurality of video frames of a video scene, the foreground patch including a set of pixels containing a portion of scene foreground in the first video frame of the video scene;

determine kinematics of an object in the video scene based at least in part on that at least one pixel-level characteristic;

generate a micro-feature vector based at least in part on the kinematics of the object, the micro-feature vector corresponding to at least one micro-feature value;

update a neural network based at least in part on the micro-feature vector, the neural network including a plurality of clusters, each cluster in the plurality of clusters representative of an object type;

classify the object in the first video frame of the video scene as a first object type based on the updated neural network, the first object type being represented by a first cluster in the plurality of clusters; and

track the object in the video scene based on the micro-feature vector and the first object type.

15. The non-transitory computer-readable storage medium of claim 14, wherein the computer-executable instructions to update the neural network include instructions to at least one of:

create the first cluster representative of the first object type; and

update the first cluster in the plurality of clusters representative of the first object type based on the micro-feature vector.

16. The non-transitory computer-readable storage medium of claim 14, wherein the computer-executable instructions to update the neural network include instructions to:

evaluate a rank for each cluster in the plurality of clusters relative to the micro-feature vector; and

evaluate each cluster in the plurality of clusters to determine if the object in the first video frame maps to at least one cluster in the plurality of clusters based on the ranking for each cluster.

17. The non-transitory computer-readable storage medium of claim 14, wherein each cluster in the plurality of clusters in the neural network are created based on a plurality of micro-feature vectors.

18. The non-transitory computer-readable storage medium of claim 14, wherein the computer-executable instructions to generate the micro-feature vector include instructions to:

evaluate appearance values of the pixels in the set of pixels to compute the at least one micro-feature value.

19. The non-transitory computer-storage medium of claim 14, wherein the at least one micro-feature value includes a verticality of the foreground patch and the at least one pixel-level characteristic includes at least one pixel-level gradient value of the foreground patch.

20. The non-transitory computer-storage medium of claim 14, wherein the computer-executable instructions to generate the micro-feature vector include instructions to evaluate values representing the kinematics of the object.