LOW- AND HIGH-FIDELITY CLASSIFIERS APPLIED TO ROAD-SCENE IMAGES

• US 20170206434A1
• Filed: 01/14/2016
• Published: 07/20/2017
• Est. Priority Date: 01/14/2016
• Status: Abandoned Application

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Nariyambut Murali, Vidya, Schrier, Madeline Jane
• Application Number:
  US14/995,482
• Publication Number:
  US 20170206434A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

Disclosures herein teach applying a set of sections spanning a down-sampled version of an image of a road-scene to a low-fidelity classifier to determine a set of candidate sections for depicting one or more objects in a set of classes. The set of candidate sections of the down-sampled version may be mapped to a set of potential sectors in a high-fidelity version of the image. A high-fidelity classifier may be used to vet the set of potential sectors, determining the presence of one or more objects from the set of classes. The low-fidelity classifier may include a first Convolution Neural Network (CNN) trained on a first training set of down-sampled versions of cropped images of objects in the set of classes. Similarly, the high-fidelity classifier may include a second CNN trained on a second training set of high-fidelity versions of cropped images of objects in the set of classes.

Claims

1. A system, comprising:

a low-fidelity classifier, on a processor set, operable to select a candidate region, from a region set spanning a down-sampled version of an image from an automobile-affixed camera capturing road-scenes, upon determining the candidate depicts a classified object;

a high-fidelity classifier, on the processor set, operable to verify classified-object depiction in a patch, mapped from the candidate, of a high-fidelity version of the image, where the high-fidelity classifier indicates the depiction.

2. The system of claim 1, wherein:

the low-fidelity classifier, comprising a first Convolution Neural Network (CNN), is trained with a down-sampled training set comprising multiple, labeled, down-sampled versions of images of objects in a class characterizing the classified object, the labeled, down-sampled versions having dimension commensurate to dimensions of regions in the region set; and

the high-fidelity classifier, comprising a second CNN, is trained with a high-resolution training set comprising multiple, labeled, high-fidelity, versions of images of objects in the class.

3. The system of claim 2, further comprising a resolution module operable to generate the down-sampled versions in the down-sampled training set, at least some of which are down-sampled to a lowest resolution at which entropies in the down-sampled versions remain above a threshold defined relative to entropies in the images of objects in the class.

4. The system of claim 2, further comprising a down-sample module implemented on the processor set and operable to produce the down-sampled version of the image from the automobile-affixed camera at a down-sample factor determined to preserve, in the down-sampled version, a predetermined percent of entropy in the image from the camera.

5. The system of claim 4, wherein the predetermined percent of entropy comes from a range centered on eighty percent and extending above and below eighty percent by five percent.

6. The system of claim 2, further comprising:

a window module operable to;

abstract overlapping regions, from the of the down-sampled version, as can be framed by at least one window slid fully across the down-sampled version, for the region set; and

apply the overlapping regions to the low-fidelity classifier; and

a mapping module operable to map the candidate region from the down-sampled version of the image to the patch of the high-fidelity version of the image, such that the candidate region and the patch cover a common sector of the image in the down-sampled version and the high-fidelity version respectively.

7. The system of claim 6, wherein:

the at least one window comprises a first window with first dimensions differing from second dimensions for a second window, both the first dimensions and the second dimensions corresponding to different scales at which objects in the class can potentially be depicted and detected in the down-sampled version of the image;

the region set comprises a first region subset of first overlapping regions with dimensions commensurate to the first dimensions and a second region subset of second overlapping regions with dimensions commensurate to the second dimensions;

the down-sampled training set comprises a first down-sampled subset of first down-sampled versions having dimensions commensurate to the first dimensions and a second down-sampled subset with second down-sampled versions having dimensions commensurate to the second dimensions.

8. The system of claim 2, further comprising:

an imaging subsystem comprising at least one of a RAdio Detection And Ranging (RADAR) subsystem and a LIght Detection And Ranging (LIDAR) subsystem; and

a multi-stage-image-classification system comprising the camera and both the low-fidelity classifier and the high-fidelity classifier on the processor set; and

an aggregation module, implemented on the processor set, operable to apply the low-fidelity classifier with an exhaustive coverage of the down-sampled version of the image from the camera, as applied to the region set, to provide redundancy to and supply missing classification information absent from classification information provided by the imaging subsystem.

9. The system of claim 2, further comprising:

an image queue operable to sequentially queue a series of images of oncoming road-scenes captured by the camera;

at least one Graphical Processing Unit (GPU), within the processor set, implementing at least one of the low-fidelity classifier and the high-fidelity classifier; and

wherein

parameters of both the low-fidelity classifier and the high-fidelity classifier are set to limit computation requirements of the low-fidelity classifier and the high-fidelity classifier, relative to computing capabilities of the at least one GPU, enabling processing the series of images at a predetermined rate providing real-time access to classification information in the series of images.

10. A method for object classification and location, comprising:

down-sampling an image to a down-sampled version of the image;

extracting a set of overlapping zones covering the down-sampled version, as definable by a sliding window with dimensions equal to dimensions of the zones;

selecting a probable zone from the set of overlapping zones for which a low-fidelity classifier, comprising a first Convolution Neural Network (CNN), indicates a probability of a presence of an object pertaining to a class of objects classifiable by the low-fidelity classifier;

mapping the probable zone selected from the down-sampled version to a sector of a higher-resolution version of the image; and

confirming the presence of the object by applying the sector to a high-fidelity classifier, comprising a second CNN, where applying the sector indicates the presence.

11. The method of claim 10, further comprising:

cropping a set of images of objects at a set of image sizes, images in the set of images classified according to a set of detection classes by labels assigned to the images;

down-sampling the set of images to create a down-sampled set of labeled images;

training the low-fidelity classifier with the down-sampled set of labeled images; and

training the high-fidelity classifier with at least one of the set of images and comparable images selected for purposes of training.

12. The method of claim 10, further comprising:

collecting a training set of images depicting pedestrians in various positions and contexts for inclusion within the set of images;

and labeling the training set according to a common class in the set of detection classes.

13. The method of claim 10, further comprising calculating a maximum factor by which the image can be down-sampled to generate the down-sampled version while maintaining a ratio of entropy in the down-sampled version to entropy in the image above a predetermined threshold level.

14. The method of claim 10, further comprising searching zones in the set of overlapping zones to which the low-fidelity classifier has yet to be applied for at least one additional probable zone while simultaneously confirming the presence of the object by applying the sector to a high-fidelity classifier.

15. The method of claim 10, further comprising:

capturing, by a camera affixed to an automobile, a series of images of oncoming road-scenes at a frame-rate satisfying a predefined threshold; and

processing the series of images, by applying claim 10 on individual images in the series of images, at a processing-rate also satisfying the predefined threshold, the predefined threshold providing sufficient time for a pre-determined autonomous response by the automobile to classification information in the series of images.

16. The method of claim 10, further comprising:

abstracting a set of scaled zones from the down-sampled version, scaled zones in the set of scaled zones having differing dimensions from the dimensions of the sliding window and commensurate with scaled dimensions of a scaled sliding window;

selecting a scaled zone from the set of scaled zones for which the low-fidelity classifier indicates a probability of an existence of a scaled object classifiable by the low-fidelity classifier;

mapping the scaled zone to a scaled sector of the higher-resolution version; and

confirming the existence of the scaled object by applying the scaled sector to the high-fidelity classifier, where applying the scaled sector results in a probability of the existence.

17. An image-analysis system, comprising:

at least one database, on at least one storage medium, comprising;

a first dataset comprising cropped, down-sampled images with labels of a label set;

a second dataset comprising cropped, higher-resolution images with the labels from the label set; and

a processor set implementing;

a first Convolution Neural Network (CNN) operable to be trained on the first dataset to classify, relative to the label set, a section from a set of overlapping sections spanning a down-sampled version of a road-scene image, section dimensions being commensurate to dimensions of the down-sampled images; and

a second CNN, operable to be trained on the second dataset to re-classify, relative to the label set, an area of the road-scene image, at high fidelity, covering the section.

18. The system of claim 17, further comprising a resolution module operable to generate the down-sampled images in the first dataset comprising fully down-sampled images that are down-sampled to a limit resolution calculated as a lower limit on resolution capable of maintaining at least a predetermined percentage of entropy relative to an original, cropped image from which a corresponding down-sampled image is generated.

19. The system of claim 17, further comprising a set of processors implementing:

a down-sample module operable to down-sample a road-scene image to a low-resolution image;

an application module operable to;

canvass the full field of view captured by the low-resolution image by applying overlapping sections of the low-resolution image to the low-fidelity classifier;

note a set of potential sections in which the low-fidelity classifier identifies potential depictions of objects classifiable according to the label set; and

a determination module operable to;

project the set of potential sections on a high-fidelity version of the road-scene image to create a set of candidate areas; and

determine a confirmed set of areas by applying the high-fidelity classifier to the set of candidate areas.

20. The system of claim 19, further comprising:

a camera operable to be mounted on an automobile to capture a series road-scene images;

a Graphics Processing Unit (GPU) in the processors set implementing the first CNN to capitalize on parallel processing capabilities of the GPU, enabling the first CNN to process the series of road-scene images at a rate providing time for a predetermined, autonomous-vehicle response to classification information in the series of road-scene images as processed.