SYSTEM AND METHOD FOR IMAGE ANALYSIS

• US 20180012082A1
• Filed: 07/05/2017
• Published: 01/11/2018
• Est. Priority Date: 07/05/2016
• Status: Active Grant

• Current Assignee: Nauto, Inc.
• Original Assignee: Nauto, Inc.
• Inventors: Satazoda, Ravi Kumar, Sathyanarayana, Suchitra
• Granted Patent: US 10,037,471 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A method for image analysis, including recording an image sequence at a vehicle system mounted to a vehicle; automatically detecting an object within the image sequence with a detection module; automatically defining a bounding box about the detected object within each image of the image sequence; modifying the image sequence with the bounding boxes for the detected object to generate a modified image sequence; at a verification module associated with the detection module, labeling the modified image sequence as comprising one of a false positive, a false negative, a true positive, and a true negative detected object based on the bounding box within at least one image of the modified image sequence; training the detection module with the label for the modified image sequence; and automatically detecting objects within a second image sequence recorded with the vehicle system with the trained detection module.

Claims

1. A method for image analysis, comprising:

recording an image at a vehicle system mounted to a vehicle;

automatically detecting an object within the image with a first module;

automatically defining a first bounding box about the detected object;

modifying the image with the first bounding box for the detected object to generate a first modified image;

at a verification module associated with the first module, labeling the first bounding box within the modified image as one of a false positive, a false negative, a true positive, and a true negative detected object;

training the first module with the label for the first bounding box;

automatically classifying the detected object, wherein classifying the detected object comprises automatically assigning an object class to the detected object, wherein the object class is one of a plurality of object classes, to generate a classified object;

modifying the image with a second bounding box for the classified object to generate a second modified image;

at a second verification module associated with the object class, labeling the second bounding box within the second modified image as one of a false positive, a false negative, a true positive, and a true negative for the object class, wherein the second verification module is one of a plurality of verification modules, each associated with a different object class of the plurality of object classes;

training the second module with the label for the second bounding box within the second modified image; and

automatically detecting and classifying objects within a second image, recorded at the vehicle system, with the trained first and second modules, respectively.

2. The method of claim 1, wherein the first module is executed by the vehicle system, the method further comprising transmitting the first modified image from the vehicle system to a remote computing system, remote from the vehicle.

3. The method of claim 2, wherein the modified image is transmitted from the vehicle system to the remote computing system in response to detection of an object within the image.

4. The method of claim 1, wherein the first and second module are a first and second level of a cascaded classification system.

5. The method of claim 1, wherein the second modified image comprises a set of pixels, and wherein labeling the second bounding box comprises:

automatically determining a horizon in the second modified image based on image fiducials, and

automatically labeling the detected object as a false positive based on at least a portion of the second bounding box comprising pixels of the set of pixels located above the horizon.

6. The method of claim 1, further comprising:

automatically classifying the classified object by assigning a second object class to the classified object with a third module, wherein the second object class is one of the plurality of object classes, to generate a subclassified object;

modifying the image with a third bounding box for the subclassified object to generate a third modified image;

at a third verification module associated with the second object class, labeling the third bounding box within the third modified image as one of a false positive, a false negative, a true positive, and a true negative for the second object class;

training the third module with the label for the third bounding box within the third modified image; and

automatically classifying objects within the second image recorded with the vehicle system with the trained third module.

7. The method of claim 1, wherein the first and second verification modules each comprise a user interface, wherein labeling comprises:

receiving a user selection of the bounding box within the modified image, wherein receiving the user selection labels the bounding box as a false positive.

8. The method of claim 7, further comprising receiving a user boundary input about an image region, generating a user bounding box about the image region based on the user boundary input, and wherein receiving the user boundary input labels the user bounding box as a false negative.

9. The method of claim 1, further comprising:

recording a sequential image, wherein the image is a first frame of an image sequence recorded with the vehicle system, and the sequential image is a second frame of the image sequence;

automatically defining a tracked bounding box about the detected object within the second frame based upon a predicted trajectory of the detected object between the first and second frame;

wherein the predicted trajectory is determined by a tracking module;

modifying the second frame with the tracked bounding box to generated a modified second frame;

at the verification module, labeling the tracked bounding box within the modified second frame as one of a false positive, a false negative, a true positive, and a true negative based on a comparison between the predicted trajectory and an actual trajectory of the detected object between the first and second frame;

training the tracking module with the label for the tracked bounding box; and

automatically tracking detected objects within a second image sequence recorded with the vehicle system based on the trained tracking module.

10. A method for image analysis, comprising:

recording an image sequence at a vehicle system mounted to a vehicle;

automatically detecting an object within the image sequence with a detection module;

automatically defining a bounding box about the detected object within each image of the image sequence;

modifying the image sequence with the bounding boxes for the detected object to generate a modified image sequence;

at a verification module associated with the detection module, labeling the modified image sequence as comprising one of a false positive, a false negative, a true positive, and a true negative detected object based on the bounding box within at least one image of the modified image sequence;

training the detection module with the label for the modified image sequence; and

automatically detecting objects within a second image sequence recorded with the vehicle system with the trained detection module.

11. The method of claim 10, further comprising:

automatically classifying the detected object with a classification module, wherein classifying the detected object comprises automatically assigning an object class to the detected object, wherein the object class is one of a plurality of object classes, to generate a classified object;

modifying the image sequence with a second bounding box for the classified object within each image of the image sequence to generate a second modified image sequence;

at a second verification module associated with the object class, labeling the second modified image sequence as comprising one of a false positive, a false negative, a true positive, and a true negative for the object class based on the second bounding box within at least one image of the second modified image sequence, wherein the second verification module is one of a plurality of verification modules, each associated with a different object class of the plurality of object classes; and

training the classification module with the labels for the second bounding box within the second modified image sequence; and

automatically classifying objects within the second image sequence recorded with the vehicle system with the trained classification module.

15. The method of claim 10, wherein detecting the object comprises detecting the object at the vehicle system with a first copy of the detection module, wherein training the detection module comprises training a second copy of the detection module at the remote computing system, and further comprising updating the first copy of the detection module based on the trained second copy of the detection module.

16. The method of claim 10, wherein the vehicle system comprises an accelerometer that outputs an accelerometer signal, and further comprising recording the image sequence in response to an accelerometer signal amplitude exceeding a threshold amplitude.

17. The method of claim 10, wherein automatically defining the bounding box about the detected object comprises defining a bounding box within a first image of the image sequence, and further comprising defining bounding boxes around the object within subsequent images of the image sequence based on a predicted trajectory of the object.

18. The method of claim 17, further comprising receiving a user selection of a bounding box defined within the image sequence, wherein receiving the user selection labels the bounding box as a false positive.

19. The method of claim 10, wherein each vehicle system of a plurality of vehicle systems implements a version of the detection module, further comprising training a master version of the detection module based on a ranked voting among labeled boundary boxes corresponding to each version of the detection module, and modifying each version of the detection module based on the trained master version of the detection module.

20. The method of claim 19, wherein each version of the detection module comprises a substantially identical copy of the detection module.

12. The method of claim ii, wherein the object class is determined based on a vehicle context parameter, and wherein the vehicle context parameter is determined at the vehicle system.

13. The method of claim 12, wherein the vehicle context parameter comprises a roadway type on which the vehicle is driving.

14. The method of claim ii, further comprising:

automatically classifying the classified object by assigning second object class to the classified object with a second classification module, wherein the second object class is one of the plurality of object classes, to generate a subclassified object;

modifying the image sequence with a third bounding box for the subclassified object within each image of the image sequence to generate a third modified image sequence;

at a third verification module associated with the second object class, labeling the third modified image sequence as one of a false positive, a false negative, a true positive, and a true negative for the second object class based the third bounding box within at least one image of the third modified image sequence;

training the third module with the label for the third modified image sequence; and

automatically classifying objects within the second image sequence recorded with the vehicle system with the trained third module.