METHOD OF IDENTIFYING AN OBJECT IN A VISUAL SCENE

• US 20120045119A1
• Filed: 11/01/2011
• Published: 02/23/2012
• Est. Priority Date: 07/26/2004
• Status: Active Grant

• Current Assignee: Joyson Safety Systems Acquisition, LLC (Ningbo Junsheng Investment Group Co., Ltd.)
• Original Assignee: Automotive Systems Laboratory, Incorporated (Ningbo Junsheng Investment Group Co., Ltd.)
• Inventors: SCHAMP, Gregory G.
• Granted Patent: US 8,509,523 B2
• Time in Patent Office: Days
• US Class Current: 382/157
• CPC Class Codes: B60R 21/0134 responsive to imminent contact with an obstacle , e.g. using radar systems
• CPC Class Codes: B60R 21/34 Protecting non-occupants of a vehicle, e.g. pedestrians B60R19/02 takes precedence
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A plurality of features determined from at least a portion of an image containing information about an object are processed with an inclusive neural network, and with a plurality of exclusive neural networks, so as to provide a plurality of inclusive probability values representing probabilities that the portion of the image corresponds to at least one of at least two different classes of objects, and for each exclusive neural network, so as to provide first and second exclusive probability values representing probabilities that the portion of the image respectively corresponds. or not. to at least one class of objects. The plurality of inclusive probability values, and the first and second exclusive probability values from each of the exclusive neural networks, provide for identifying whether the portion of the image corresponds, or not, to any of the at least two different classes of objects.

Claims

1. A method of identifying an object in a visual scene, comprising:

a. extracting at least a portion of an image containing information about an object;

b. determining a first plurality of features of said at least said portion of said image;

c. processing said first plurality of features with an inclusive neural network, wherein said inclusive neural network is adapted to provide a second plurality of inclusive probability values responsive to said first plurality of features, and a third plurality of at least two of said second plurality of inclusive probability values represent a probability that said at least said portion of said image corresponds to a corresponding at least one of at least two different classes of objects;

d. processing said first plurality of features with a fourth plurality of exclusive neural networks, wherein said fourth plurality is equal in number to said third plurality, each exclusive neural network of said fourth plurality of exclusive neural networks provides a corresponding first exclusive probability value representing a probability that said at least said portion of said image corresponds to one of said at least two different classes of objects, each said exclusive neural network provides a corresponding second exclusive probability value representing a probability that said at least said portion of said image does not correspond to said one of said at least two different classes of objects, and different exclusive neural networks of said fourth plurality of exclusive neural networks provide said corresponding first and second probabilities for corresponding different classes of objects of said at least two different classes of objects; and

e. identifying whether said at least said portion of said image corresponds to any of said at least two different classes of objects, or whether said at least said portion of said image does not correspond to said any of said at least two different classes of objects, responsive to said second plurality of inclusive probability values from said inclusive neural network, and responsive to said corresponding first and second exclusive probability values from each of said fourth plurality of exclusive neural networks.

2. A method of identifying an object in a visual scene as recited in claim 1, wherein one of said second plurality of inclusive probability values represents a probability that said at least said portion of said image does not correspond to said any of said at least two different classes of objects.

3. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a combination of mathematical and geometric shape descriptors for either a binary segmentation image or a subsection thereof, wherein said combination of mathematical and geometric shape descriptors exhibit substantial intra-class clustering and substantial inter-class separation, and said combination of mathematical and geometric shape descriptors exhibit substantial invariance with respect to position, rotation and size within an image plane of said at least said portion of said image.

4. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a size of a segmented area of said at least said portion of said image.

5. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a location of a binary center-of-mass of a segmented area of said at least said portion of said image.

6. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a vertical extent of an object in said at least said portion of said image and a horizontal extent of said object in said at least said portion of said image.

7. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a best-fit rectangle aspect ratio.

8. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a best-fit rectangle fill factor.

9. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise an orientation angle of a best-fit ellipse and a ratio of lengths of major and minor axes of said best fit ellipse.

10. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a best correlation of a filtered harmonic profile of said object in said at least said portion of said image with one of a plurality of stored harmonic profiles for each type of filter used for said plurality of stored harmonic profiles, wherein said filter comprises a central moving average filter and said type of filter comprises a length of said central moving average filter.

11. A method of identifying an object in a visual scene as recited in claim 1, wherein said first plurality of features comprise a maximum horizontal extent of said object in a lower half of a best-fit rectangle of said object in said at least said portion of said image.

12. A method of identifying an object in a visual scene as recited in claim 1, wherein said inclusive neural network and said fourth plurality of exclusive neural networks are trained using at least thousands of images for each of said at least two different classes of objects.

13. A method of identifying an object in a visual scene as recited in claim 1, wherein said at least two different classes of objects comprises a first class of pedal cyclist objects, a second class of pedestrian objects that are either stationary or walking at mid stride, a third class of pedestrian objects that are either walking or running at full stride, and a fourth class of vehicle objects, said inclusive neural network provides a first inclusive probability value that said first plurality of features are associated with said object being in said first class of pedal cyclist objects, a second inclusive probability value that said first plurality of features are associated with said object being in said second class of pedestrian objects that are either stationary or walking at mid stride, a third inclusive probability value that said first plurality of features are associated with said object being in said third class of pedestrian objects that are either walking or running at full stride, a fourth inclusive probability value that said first plurality of features are associated with said object being in said fourth class of vehicle objects, and a fifth inclusive probability value that said first plurality of features are not associated with any of said first, second, third or fourth classes of objects, said fourth plurality of exclusive neural networks comprise first, second and third exclusive neural networks, said first exclusive neural network provides said corresponding first and second exclusive probability values indicative of whether or not said first plurality of features are associated with said object being in said first class of pedal cyclist objects, said second exclusive neural network provides said corresponding first and second exclusive probability values indicative of whether or not said first plurality of features are associated with said object being in said second class of pedestrian objects that are either stationary or walking at mid stride, and said third exclusive neural network provides said corresponding first and second exclusive probability values indicative of whether or not said first plurality of features are associated with said object being in said third class of pedestrian objects that are either walking or running at full stride.

14. A method of identifying an object in a visual scene as recited in claim 13, wherein the operation of identifying whether said portion of said image corresponds or not to said any of said at least two different classes of objects is further responsive to a best correlation of a filtered harmonic profile of said object in said portion of said image with one of a plurality of stored harmonic profiles for each type of filter used for said plurality of stored harmonic profiles.

15. A method of identifying an object in a visual scene as recited in claim 14, wherein if said fourth inclusive probability from said inclusive neural network is greater than or equal to a first threshold, then said portion of said image is identified as a vehicle;

otherwise if any of said first through third inclusive probabilities from said inclusive neural network is greater than or equal to a second threshold, then said portion of said image is identified as a vulnerable road user (VRU);

otherwise if any of said first through third inclusive probabilities from said inclusive neural network is greater than or equal to said first threshold, AND said first exclusive probability from any of said first through third exclusive neural networks is greater than said first threshold AND said best correlation of said filtered harmonic profile for a corresponding said stored harmonic profile is greater than or equal to said first threshold, then said portion of said image is identified as said vulnerable road user (VRU);

otherwise if said first exclusive probability from any of said first through third exclusive neural networks is greater than a third threshold OR said best correlation of said filtered harmonic profile is greater than or equal to said third threshold, then said portion of said image is identified as said vulnerable road user (VRU);

otherwise said portion of said image is not identified as a particular object, wherein said first threshold is less than said second threshold, and said second threshold is less than said third threshold.

16. A method of identifying an object in a visual scene as recited in claim 1, wherein the operation of identifying whether or not said portion of said image corresponds to said any of said at least two different classes of objects is further responsive to a rule-based decision process responsive to a rule-base that applies to all possible members of at least one of said at least two different classes of objects.

17. A method of identifying an object in a visual scene as recited in claim 16, wherein said rule-base is adapted to provide for at least partially preventing a false detection that said object corresponds to said any of said at least two different classes of objects, when said object is not actually a member of said any of said at least two different classes of objects.

18. A method of identifying an object in a visual scene as recited in claim 17, wherein said rule-base comprises comparing a measure of said object in said portion of said image threshold with a range-responsive threshold, if said measure of said object either exceeds a range-responsive first threshold OR is less than a range-responsive second threshold then said object is identified as not being a vulnerable road user (VRU), and said measure of said object is selected from a vertical extent of a corresponding best-fit rectangle, a horizontal extent of said best-fit rectangle, and a location of a lower boundary of said best-fit rectangle.

19. A method of identifying an object in a visual scene as recited in claim 1, wherein

a. at a first point in time, said at least said portion of said image containing information about said object comprises a cluster of pixels corresponding to a relatively closest said object;

b. if the operation of identifying whether said at least said portion of said image corresponds to said any of said at least two different classes of objects results in an identification that said at least said portion of said image corresponds to at least one of said at least two different classes of objects, then responding to said at least one of said at least two different classes of objects;

c. otherwise, if possible, then setting said at least said portion of said image containing information about said object to a different cluster of pixels corresponding to a next relatively closest said object and then continuing with step b.

20. A method of identifying an object in a visual scene as recited in claim 1, wherein if said portion of said image corresponds to any of said at least two different classes of objects, then tracking at least one object of said portion of said image corresponding to said at least one of said at least two different classes of objects.

21. A method of identifying an object in a visual scene as recited in claim 20, further comprising accumulating a confidence of recognition, a history of a location of a center of gravity of said at least one object in both down range and cross range relative to a vehicle, an estimate of a time to fire of an associated warning or protective device of said vehicle, and an estimate of a time to impact of said at least one object by said vehicle.

22. A method of identifying an object in a visual scene as recited in claim 21, wherein said at least one object comprises a plurality of objects, further comprising sorting said plurality of objects in ascending order responsive to an estimate of a distance to impact.

23. A method of identifying an object in a visual scene as recited in claim 22, further comprising determining if said vehicle is following at least one said at least one object responsive to a persistence of said at least one said at least one object and responsive to a history of an estimate of a distance from said vehicle to said at least one said at least one object.

24. A method of identifying an object in a visual scene as recited in claim 20, further comprising maintaining information about at least one said at least one object classified as a potential threat prior to said at least one said at least one object entering a collision possible space of a vehicle.

25. A method of identifying an object in a visual scene as recited in claim 20, wherein the operation of tracking said at least one object is responsive to an autoregressive model.

26. A method of identifying an object in a visual scene as recited in claim 25, further comprising tracking said at least one object during transient maneuvers of said at least one object.

27. A method of identifying an object in a visual scene as recited in claim 20, further comprising correlating at least one newly classified object with at least one said at least one object already being tracked.

28. A method of identifying an object in a visual scene as recited in claim 20, wherein the operation of identifying whether said portion of said image corresponds or not to said any of said at least two different classes of objects is further responsive to at least one element selected from the group consisting of a best correlation of a filtered harmonic profile of said at least one object in said portion of said image with one of a plurality of stored harmonic profiles for each type of filter used for said plurality of stored harmonic profiles, and a rule-based decision process responsive to a rule-base that applies to all possible members of at least one of said at least two different classes of objects.

29. A method of identifying an object in a visual scene as recited in claim 1, wherein the operation of identifying whether said portion of said image corresponds or not to said any of said at least two different classes of objects is further responsive to at least one correlation of a corresponding at least one filtered harmonic profile of said object in said at least said portion of said image with a corresponding stored harmonic profile, further comprising:

a. associating one or more outputs of each of said inclusive neural network, said fourth plurality of exclusive neural networks, and said at least one correlation of said corresponding at least one filtered harmonic profile with said corresponding stored harmonic profile for each of said at least two different classes of objects;

b. using a third neural network to combine said one or more outputs of each of said inclusive neural network, said fourth plurality of exclusive neural networks, and said at least one correlation of said corresponding at least one filtered harmonic profile so as to generate a corresponding output for each of said at least two different classes of objects.

30. A method of identifying an object in a visual scene as recited in claim 29, further comprising training said third neural network with a training set incorporating various levels of support providing for a clustering of related classes and a separation of dissimilar classes.

31. A method of identifying an object in a visual scene as recited in claim 29, further comprising:

a. associating one or more of said first plurality of features with each of said at least two different classes of objects; and

b. using said third neural network to combine said one or more and said one or more of said plurality of features so as to generate a corresponding output for each of said at least two different classes of objects.

32. A method of identifying an object in a visual scene as recited in claim 31, further comprising generating one or more groups of classes with said third neural network, wherein each group of said one or more groups of classes comprises a subset of one or more classes of said at least two different classes of objects.

33. A method of identifying an object in a visual scene as recited in claim 32, further comprising consolidating said subset of said one or more classes of at least one group of classes of said one or more groups of classes using said third neural network so as to generate an output for at least one corresponding consolidated class.

34. A method of identifying an object in a visual scene as recited in claim 33, wherein said at least two different classes of objects comprises a compact vehicle class, a bicyclist class, a stationary pedestrian class, and a walking pedestrian class, said one or more groups of classes comprises first and second groups of classes, said first group of classes comprises said compact vehicle class and said bicyclist class, said second group of classes comprises said stationary pedestrian class and said walking pedestrian class, and said at least one corresponding consolidated class comprises a pedestrian class that is consolidated from said second group of classes.

35. A method of identifying an object in a visual scene as recited in claim 34, wherein said at least one corresponding consolidated class further comprises a vulnerable road user (VRU) class that is consolidated from said pedestrian class and said bicyclist class, so as to provide for a first final output responsive to said compact vehicle class and a second final output responsive to said vulnerable road user (VRU) class.