Image-based feature detection using edge vectors

• US 10,229,342 B2
• Filed: 12/15/2017
• Issued: 03/12/2019
• Est. Priority Date: 05/06/2014
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Song, Bing, Witchey, Nicholas J.
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/844,258
• Publication Number:
  US 20180107894A1
• Time in Patent Office: 452 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/758 : Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/64 : Three-dimensional objects

Abstract

Techniques are provided in which a plurality of edges are detected within a digital image. An anchor point located along an edge of the plurality of edges is selected. An analysis grid associated with the anchor point is generated, the analysis grid including a plurality of cells. An anchor point normal vector comprising a normal vector of the edge at the anchor point is calculated. Edge pixel normal vectors comprising normal vectors of the edge at locations along the edge within the cells of the analysis grid are calculated. A histogram of similarity is generated for each of one or more cells of the analysis grid, each histogram of similarity being based on a similarity measure between each of the edge pixel normal vectors within a cell and the anchor point normal vector, and a descriptor is generated for the analysis grid based on the histograms of similarity.

Claims

1. An automated image-based feature detection apparatus, comprising:

a digital image capturing device;

a non-transitory computer-readable memory storing feature detection software instructions; and

a processor coupled with the digital image capturing device and the memory, and configured to operate as a feature detector upon execution of the software instructions, wherein the feature detector is configurable to;

obtain, via the digital image capturing device, digital image frames of a real-time video stream comprising digital images of a streetscape including a plurality of objects, wherein the plurality of objects includes at least one vehicle;

detect a plurality of edges associated with at least one object of the plurality of objects within a plurality of frames of the digital image frames;

track, in real-time across the plurality of frames, image features based on one or more anchor points associated with at least one edge of the plurality of edges;

assign a vector for each of at least some points of a plurality of points along the at least one edge such that a plurality of vectors is associated with the at least one edge, wherein each anchor point of the one or more anchor points corresponds to a vector of the plurality of vectors, thereby defining an anchor point vector, and wherein each anchor point is associated with an analysis grid located on the at least one edge;

generate a similarity measure as a function of relative measures between one of the anchor point vectors and at least some of the plurality of vectors within an analysis grid that corresponds to the one of the anchor point vectors; and

enable a networked device to take an action associated with the at least one object as a function of the similarity measure.

2. The apparatus of claim 1, wherein the at least one object comprises at least one of a vehicle, a person, a building, printed media, and a barcode, a logo, an animal, a plant, a toy, a low-texture object, and a high-texture object located within the streetscape.

3. The apparatus of claim 1, wherein the at least one object is a three-dimensional object.

4. The apparatus of claim 3, wherein the digital image frames comprise images of the three-dimensional object captured from different perspectives.

5. The apparatus of claim 3, further comprising determining a relationship between one or more of the detected edge features and a structure of the three-dimensional object.

6. The apparatus of claim 1, wherein the one or more anchor points are selected based on at least one of a detected corner, an inflection point, and a change in curvature above a threshold along an edge.

7. The apparatus of claim 6, wherein the feature detector is further configured to select at least some of the anchor points for real-time tracking.

8. The apparatus of claim 6, wherein the one or more anchor points are selected in accordance with an object model.

9. The apparatus of claim 8, wherein the object model is selected based on one or more of a cityscape image, a streetscape image, map features, a street grid pattern, and topographical features.

10. The apparatus of claim 1, wherein the feature detector is further configurable to access a library of edge-based feature descriptors to associate objects within the streetscape with an object model.

11. The apparatus of claim 1, wherein the analysis grid is scaled based on one or more characteristics of an associated anchor point.

12. The apparatus of claim 1, further comprising distributing a plurality of analysis grids per unit area within the digital image frames.

13. The apparatus of claim 12, wherein the plurality of analysis grids is distributed evenly per unit volume within a three-dimensional space based on an object model and field of depth information derived from image data.

14. The apparatus of claim 1, wherein the digital image capture device is one of a sensor or a digital video camera configured to capture images or frames of video data.

15. The apparatus of claim 14, wherein the digital image capturing device comprises a plurality of sensors including at least one of a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, and an ultrasound sensor.

16. The apparatus of claim 1, wherein the digital image capturing device comprises a graphical rendering or modeling engine.

17. The apparatus of claim 16, wherein the digital image frames comprise a generated graphical rendering or modeling of the streetscape.

18. The apparatus of claim 1, wherein the digital image frames comprise one or more of an image filtered based on electromagnetic radiation, a 360-degree view image, a synthetic image, a machine or robot view image, video data, a graphical rendering, and a three-dimensional perspective view.

19. The apparatus of claim 1, further comprising scaling the digital image frames based on a scaling factor.

20. The apparatus of claim 1, wherein the real-time video stream is from a broadcast.

21. The apparatus of claim 1, wherein the tracked image features comprise vectors that correspond to one or more distinguishable features of the at least one object.

22. The apparatus of claim 1, wherein the tracked image features include at least one of the following types of features:

scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG), and Transform Invariant Low-rank Textures (TILT) features.

23. The apparatus of claim 1, wherein the feature detector is further configured to compress an amount of image features tracked across the plurality of frames.

24. The apparatus of claim 1, wherein the feature detector is further configurable to generate an analysis grid, wherein the analysis grid includes a plurality of cells associated with an anchor point.

25. The apparatus of claim 1, wherein detecting the plurality of edges comprises locating one or more discontinuities within the digital image frames.

26. The apparatus of claim 1, wherein detecting the plurality of edges comprises determining that at least one of the one or more discontinuities is significant based on at least one edge detection technique.

27. The apparatus of claim 26, wherein the at least one edge detection technique comprises one or more of a Canny edge technique, a tensor voting technique, and an edgelet connection technique.

28. The apparatus of claim 1, wherein the feature detector is further configurable to generate edge-based feature descriptors in real-time with respect to obtaining the digital image frames.

29. The apparatus of claim 1, wherein the feature detector is further configurable to track the digital images frame-by-frame within the digital image frames.

30. The apparatus of claim 1, wherein the plurality of vectors includes at least one of the following:

an edge normal vector, an eigenvector, a Harris eigenvector, and a geometric normal vector.

31. The apparatus of claim 1, wherein the relative measures comprise at least one of the following:

a dot product among vectors, a relative magnitude among vectors, a Euclidean distance, a normalization, and a Gaussian weight.

32. The apparatus of claim 1, wherein enabling the action comprises configuring the networked device based on the similarity measure.

33. An automated system for image-based feature detection, comprising:

a digital image capturing device configured to obtain digital image frames of a real-time video stream comprising digital images of a streetscape including a plurality of vehicles; and

a feature detector in communication with the digital image capturing device and configured to;

detect a plurality of edges associated with at least one vehicle of the plurality of vehicles within a plurality of frames of the digital image frames;

track, in real-time across the plurality of frames, image features based on one or more anchor points associated with at least one edge of the plurality of edges;

assign a vector for each of at least some points of a plurality of points along the at least one edge such that a plurality of vectors is associated with the at least one edge, wherein each anchor point of the one or more anchor points corresponds to a vector of the plurality of vectors, thereby defining an anchor point vector, and wherein each anchor point is associated with an analysis grid located on the at least one edge;

generate a similarity measure as a function of relative measures between one of the anchor point vectors and at least some of the plurality of vectors within an analysis grid that corresponds to the one of the anchor point vectors; and

enable a networked device to take an action associated with the at least one vehicle based on the similarity measure.