Feature density object classification, systems and methods

• US 10,671,879 B2
• Filed: 09/11/2018
• Issued: 06/02/2020
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Jaber, Mustafa, Sudol, Jeremi M., Song, Bing
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US16/128,178
• Publication Number:
  US 20190012562A1
• Time in Patent Office: 630 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

A system capable of determining which recognition algorithms should be applied to regions of interest within digital representations is presented. A preprocessing module utilizes one or more feature identification algorithms to determine regions of interest based on feature density. The preprocessing modules leverages the feature density signature for each region to determine which of a plurality of diverse recognition modules should operate on the region of interest. A specific embodiment that focuses on structured documents is also presented. Further, the disclosed approach can be enhanced by addition of an object classifier that classifies types of objects found in the regions of interest.

Claims

1. An object data processing system comprising:

at least one processor configured to execute;

at least one implementation of a plurality of recognition algorithms stored on at least one non-transitory computer-readable storage medium, each recognition algorithm having feature density selection criteria; and

data preprocessing code executed by at least one processor, the data preprocessing code comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene, the scene comprising one or more textual media;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density;

classify, by region classifier code, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation, wherein the at least one of the classified regions of interest corresponds to text; and

use a classification result corresponding to the at least one of the regions of interest to classify another of the regions of interest according to object type, wherein the another of the regions of interest corresponds to a region of interest for images.

2. The system of claim 1, wherein preprocessing code, based on the feature density selection criteria, determines that an OCR algorithm is applicable to the text, and that other recognition algorithms are applicable to aspects of the photographs and to logos.

3. The system of claim 1, wherein a user creates a user profile for a camera-equipped smartphone that includes the information that the user is visually impaired, which causes prioritized execution of the OCR algorithm such that a text reader program begins reading the text to the user as quickly as possible.

4. The system of claim 3, further comprising an audio or tactile feedback mechanism that helps the user to position the smart phone relative to the text.

5. The system of claim 4, further comprising a “

hold still”

audio feedback signal that is sent to the user when the text is at the center of the captured scene.

6. The system of claim 1, wherein the digital representation comprises at least one of the following types of digital data:

image data, video data, and audio data.

7. The system of claim 1, wherein invariant feature identification algorithm comprises at least one of the following feature identification algorithms:

FAST, SIFT, FREAK, BRISK, Harris, DAISY, and MSER.

8. The system of claim 1, wherein the invariant feature identification algorithm includes at least one of the following:

edge detection algorithm, corner detection algorithm, saliency map algorithm, curve detection algorithm, a texton identification algorithm, and wavelets algorithm.

9. The system of claim 1, wherein at least one region of interest represents at least one physical object in the scene.

10. The system of claim 1, wherein at least one region of interest represents at least one textual media in the scene.

11. The system of claim 10, wherein the region of interest represents a document as the textual media.

12. The system of claim 11, wherein the region of interest represents a financial document.

13. The system of claim 11, wherein the region of interest represents a structured document.

14. The system of claim 1, wherein at least one implementation of a plurality of recognition algorithms includes at least one of the following:

a template driven algorithm, a face recognition algorithm, an optical character recognition algorithm, a speech recognition algorithm, and an object recognition algorithm.

15. The system of claim 1, wherein data preprocessing code is further configured to assign each region of interest at least one recognition algorithm as a function of a scene context derived from the digital representation.

16. The system of claim 15, wherein the scene context includes at least one of the following types of data:

a location, a position, a time, a user identity, a news event, a medical event, and a promotion.

17. The system of claim 1, further comprising a mobile device comprising at least one implementation of a plurality of recognition algorithms and data preprocessing code.

18. The system of claim 17, wherein the mobile device comprises at least one of the following:

a smart phone, a tablet, wearable glass, a toy, a vehicle, a computer, and a phablet.

19. The system of claim 1, further comprising a network-accessible server device comprising at least one implementation of a plurality of recognition algorithms and data preprocessing code.

20. The system of claim 1, wherein the object type includes at least one of the following:

a face, an animal, a vehicle, a document, a plant, a building, an appliance, clothing, a body part, and a toy.

21. An object data processing system comprising:

at least one processor configured to execute;

at least one implementation of a plurality of recognition algorithms stored on at least one non-transitory computer-readable storage medium, each recognition algorithm having feature density selection criteria; and

data preprocessing code executed by at least one processor, the data preprocessing code comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene, the scene comprising one or more textual media;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density;

classify, by region classifier code, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation;

wherein the at least one of the classified regions of interest corresponds to text; and

use a classification result corresponding to the at least one of the regions of interest to classify another of the regions of interest according to object type, wherein the another of the regions of interest corresponds to a region of interest for images;

assign each region of interest at least one recognition algorithm from at least one implementation of a plurality of diverse recognition algorithms as a function of the region feature density of each region of interest and the feature density selection criteria of the at least one implementation of a plurality of diverse recognition algorithms; and

configure the assigned recognition algorithms to process their respective regions of interest, wherein preprocessing code, based on the feature density selection criteria, determines that an OCR algorithm is applicable to the text, and that other recognition algorithms are applicable to aspects of the photographs and to logos.

22. A device comprising:

at least one processor configured to execute;

at least one implementation of a plurality of recognition algorithms stored on at least one non-transitory computer-readable storage medium, each recognition algorithm having feature density selection criteria; and

data preprocessing code executed by at least one processor, the data preprocessing code comprising an invariant feature identification algorithm and configured to;

obtain a digital representation of a scene, the scene comprising one or more textual media;

generate a set of invariant features by applying the invariant feature identification algorithm to the digital representation;

cluster the set of invariant features into regions of interest in the digital representation of the scene, each region of interest having a region feature density; and

classify, by region classifier code, at least one of the regions of interest according to object type as a function of attributes derived from the region feature density and the digital representation, wherein the at least one of the classified regions of interest corresponds to text; and

use a classification result corresponding to the at least one of the regions of interest to classify another of the regions of interest according to object type, wherein the another of the regions of interest corresponds to a region of interest for images.