Feature density object classification, systems and methods

• US 10,102,446 B2
• Filed: 07/21/2017
• Issued: 10/16/2018
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Jaber, Mustafa, Sudol, Jeremi M., Song, Bing
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US15/656,975
• Publication Number:
  US 20170323175A1
• Time in Patent Office: 452 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

A system capable of determining which recognition algorithms should be applied to regions of interest within digital representations is presented. A preprocessing module utilizes one or more feature identification algorithms to determine regions of interest based on feature density. The preprocessing modules leverages the feature density signature for each region to determine which of a plurality of diverse recognition modules should operate on the region of interest. A specific embodiment that focuses on structured documents is also presented. Further, the disclosed approach can be enhanced by addition of an object classifier that classifies types of objects found in the regions of interest.

Claims

1. An object data processing system comprising:

at least one processor configured to execute;

a plurality of diverse recognition modules stored on at least one non-transitory computer-readable storage medium, each recognition module comprising at least one recognition algorithm and having feature density selection criteria;

wherein the feature density selection criteria include rules that operate as a function of features per unit of a one or more dimensional space; and

a data preprocessing module executed by at least one processor, the data preprocessing module comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density;

classify, by a region classifier, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation; and

use a classification result corresponding to the at least one of the regions of interest to classify another of the regions of interest according to object type.

2. The system of claim 1, wherein feature density selection criteria include rules that operate as a function of features per unit time.

3. The system of claim 1, wherein the feature density selection criteria include rules that operate as a function of features per unit area.

4. The system of claim 3, wherein the unit area represents pixels squared.

5. The system of claim 2, wherein the unit area represents geometrical area.

6. The system of claim 1, wherein the feature density selection criteria further include rules that operate as a function of features per unit volume representing geometrical area times time.

7. The system of claim 1, wherein the feature density selection criteria further include rules that operate as a function of features per unit volume representing geometrical volume.

8. The system of claim 1, wherein the feature density selection criteria comprise a low density threshold.

9. The system of claim 8, wherein the lower density threshold represents a minimum density.

10. The system of claim 1, wherein the feature density selection criteria comprise a high density threshold.

11. The system of claim 10, wherein the high density threshold represents a maximum density.

12. The system of claim 1, wherein the feature density selection criteria comprise feature density range.

13. The system of claim 1, wherein the digital representation comprises at least one of the following types of digital data:

image data, video data, and audio data.

14. The system of claim 1, wherein invariant feature identification algorithm comprises at least one of the following feature identification algorithms:

FAST, SIFT, FREAK, BRISK, Harris, DAISY, and MSER.

15. The system of claim 1, wherein the invariant feature identification algorithm includes at least one of the following:

edge detection algorithm, corner detection algorithm, saliency map algorithm, curve detection algorithm, a texton identification algorithm, and wavelets algorithm.

16. The system of claim 1, wherein at least one region of interest represents at least one physical object in the scene.

17. The system of claim 1, wherein at least one region of interest represents at least one printed media in the scene.

18. The system of claim 17, wherein the region of interest represents a document as the printed media.

19. The system of claim 18, wherein the region of interest represents a financial document.

20. The system of claim 18, wherein the region of interest represents a structured document.

21. The system of claim 1, wherein at least one region of interest comprises a region feature density representative of a homogeneous set of invariant features.

22. The system of claim 1, wherein at least one region of interest comprises a region feature density representative of a heterogeneous set of invariant features.

23. The system of claim 1, wherein the plurality of diverse recognition algorithms include at least one of the following:

a template driven algorithm, a face recognition algorithm, an optical character recognition algorithm, a speech recognition algorithm, and an object recognition algorithm.

24. The system of claim 1, wherein at least one of the regions of interest comprises feature substructure.

25. The system of claim 24, wherein the feature substructure comprises at least one of the following:

a sub-cluster of invariant features, a periodicity of invariant features, a block structure of invariant features, a frequency of invariant features, and a low density region of invariant features.

26. The system of claim 24, wherein the data preprocessing module is further configured to assign each region of interest at least one recognition module as a function of the invariant feature substructure.

27. The system of claim 1, wherein the data preprocessing module is further configured to assign each region of interest at least one recognition module as a function of a scene context derived from the digital representation.

28. The system of claim 27, wherein the scene context includes at least one of the following types of data:

a location, a position, a time, a user identity, a news event, a medical event, and a promotion.

29. The system of claim 1, wherein the attributes comprise interrelationship metrics derived from the region feature densities of other region of interest.

30. The system of claim 29, wherein the interrelationship metrics comprise at least one of the following:

a geometric metric, a time-based metric, an orientation metric, and a distribution metric.

31. The system of claim 1, further comprising a mobile device comprising the plurality of diverse recognition modules and the data preprocessing module.

32. The system of claim 31, wherein the mobile device comprises at least one of the following:

a smart phone, a tablet, wearable glass, a toy, a vehicle, a computer, and a phablet.

33. The system of claim 1, further comprising a network-accessible server device comprising the plurality of diverse recognition modules and the data preprocessing module.

34. The system of claim 1, wherein the object type includes at least one of the following:

a face, an animal, a vehicle, a document, a plant, a building, an appliance, clothing, a body part, and a toy.

35. An object data processing system comprising:

at least one processor configured to execute;

a plurality of diverse recognition modules stored on at least one non-transitory computer-readable storage medium, each recognition module comprising at least one recognition algorithm and having feature density selection criteria;

wherein the feature density selection criteria include rules that operate as a function of features per unit of a one or more dimensional space; and

a data preprocessing module executed by at least one processor, the data preprocessing module comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density;

classify, by a region classifier, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation;

use a classification result corresponding to the at least one of the regions of interest to classify another of the regions of interest according to object type;

assign each region of interest at least one recognition module from the plurality of diverse recognition modules as a function of the region feature density of each region of interest and the feature density selection criteria of the plurality of diverse recognition modules; and

configure the assigned recognition modules to process their respective regions of interest.

36. A device comprising:

at least one processor configured to execute;

a plurality of diverse recognition modules stored on at least one non-transitory computer-readable storage medium, each recognition module comprising at least one recognition algorithm and having feature density selection criteria; and

a data preprocessing module executed by at least one processor, the data preprocessing module comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density; and

classify, by a region classifier, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation.