Range-cued object segmentation system and method

• US 8,824,733 B2
• Filed: 05/07/2012
• Issued: 09/02/2014
• Est. Priority Date: 03/26/2012
• Status: Active Grant

• Current Assignee: Joyson Safety Systems Acquisition, LLC
• Original Assignee: TK Holdings Incorporated (Ningbo Junsheng Investment Group Co., Ltd.)
• Inventors: Schamp, Gregory Gerhard
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): CHOI, TIMOTHY WING HO
• Application Number:
  US13/465,059
• Publication Number:
  US 20130251194A1
• Time in Patent Office: 848 Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation

Abstract

Objects in a range map image are clustered into regions of interest responsive to range as determined from either a separate ranging system or from a top-down transformation of a range map image from a stereo vision system. A relatively-central location of each region of interest is transformed to mono-image geometry, and the corresponding portion of an associated mono-image is searched radially outwards from the relatively-central location along a plurality of radial search paths, along which the associated image pixels are filtered using an Edge-Preserving Smoothing filter in order to find an edge of the associated object along the radial search path. Edge locations for each of the radial search paths are combined in an edge profile vector that provides for discriminating the object.

Claims

1. A method of processing an image of a visual scene, comprising:

a. receiving or determining a range-map image of the visual scene, wherein said range-map image comprises a first plurality of points, each point of said first plurality of points comprises a first coordinate and a second coordinate of a corresponding portion of a corresponding physical element within said visual scene, said first coordinate either corresponds or is related to a down-range coordinate of said visual scene, and said second coordinate either corresponds or is related to a cross-range coordinate of said visual scene;

b. determining at least one region-of-interest within said visual scene, wherein said at least one region-of-interest comprises a second plurality of said points that are clustered with respect to distance in a space of first and second coordinates responsive to a corresponding distance threshold, said second plurality is less than or equal in number to said first plurality, and said space of first and second coordinates either corresponds or is related to a space of down-range and cross-range coordinates;

c. determining a corresponding relatively-central location of said second plurality of said points of said at least one region-of-interest, wherein said relatively-central location comprises a corresponding first coordinate and a corresponding second coordinate, said corresponding first coordinate either corresponds or is related to a corresponding down-range coordinate of said visual scene, and said corresponding second coordinate either corresponds or is related to a corresponding cross-range coordinate of said visual scene;

d. receiving the image of said visual scene, wherein said image comprises a first plurality of pixels organized in an image space as an array of columns and rows, each column of said array of columns and rows being identified by a corresponding X image coordinate, and each row of said array of columns and rows being identified by a corresponding Y image coordinate;

e. determining a location of said relatively-central location with respect to corresponding said X and Y image coordinates; and

f. determining locations of a plurality of edge points of a portion of said image corresponding to an object associated with a corresponding said at least one region-of-interest, wherein each edge point of said plurality of edge points corresponds to an outer boundary of said object along a radial search path extending from said relatively-central location along a corresponding outwardly-oriented direction, said outer boundary is distal relative to said relatively-central location, each said edge point of said plurality of edge points comprises corresponding coordinates in said image space, and said plurality of edge points provide for discriminating said object.

2. A method of processing an image of a visual scene as recited in claim 1, wherein the operation of receiving or determining said range-map image comprises:

a. receiving first and second stereo image components of said visual scene; and

b. determining said range-map image from said first and second stereo image components.

3. A method of processing an image of a visual scene as recited in claim 2, further comprising transforming said range-map image to a top-down coordinate system comprising corresponding first and second coordinates, wherein said corresponding first coordinate either corresponds or is related to a corresponding down-range coordinate of said visual scene, and said corresponding second coordinate either corresponds or is related to a corresponding cross-range coordinate of said visual scene.

4. A method of processing an image of a visual scene as recited in claim 1, wherein the operation of receiving or determining said range-map image comprises receiving a plurality of range measurements as a function of either azimuth angle or cross-range distance.

5. A method of processing an image of a visual scene as recited in claim 4, wherein said plurality of range measurements are generated responsive to backscattered electromagnetic radiation and are generated for a plurality of different azimuth angles.

6. A method of processing an image of a visual scene as recited in claim 5, wherein said plurality of range measurements are generated responsive to scanning a beam of electromagnetic radiation responsive to which said backscattered electromagnetic radiation is generated.

7. A method of processing an image of a visual scene as recited in claim 1, further comprising generating binned range-map image data by binning said points of said first plurality of points of said range-map image with respect to a plurality of range bins, wherein each range bin of said plurality of range bins is binned with respect to both down-range distance and cross-range distance of said visual scene.

8. A method of processing an image of a visual scene as recited in claim 7, wherein the operation of determining said at least one region-of-interest within said visual scene comprises clustering said binned range-map image data with respect to distance with respect to coordinates of said down-range distance and said cross-range distance.

9. A method of processing an image of a visual scene as recited in claim 7, further comprising generating a first histogram comprising a first count of said points of said first plurality of points with respect to a plurality of first histogram bins for a corresponding plurality of ranges of said cross-range distance, generating a second histogram comprising a second count of said points of said first plurality of points with respect to a plurality of second histogram bins for a corresponding plurality of ranges of said down-range distance, wherein said first plurality of said points are clustered responsive to said first and second histograms.

10. A method of processing an image of a visual scene as recited in claim 9, wherein said first and second histograms are generated from a plurality of range bins, and each range bin of said plurality of range bins comprises a count of said points of said first plurality of points associated with said range bin.

11. A method of processing an image of a visual scene as recited in claim 10, wherein said second plurality of said points are clustered with respect to at least one two-dimensional clustering bin.

12. A method of processing an image of a visual scene as recited in claim 11, wherein said at least one two-dimensional clustering bin comprises at least one predetermined two-dimensional nominal clustering bin, and said at least one predetermined two-dimensional nominal clustering bin is located responsive to at least one of a one-dimensional nominal cross-range clustering bin or a one-dimensional nominal down-range clustering bin.

13. A method of processing an image of a visual scene as recited in claim 11, wherein said at least one two-dimensional clustering bin is located responsive to at least one of a localized peak of said first histogram or a localized peak of said second histogram.

14. A method of processing an image of a visual scene as recited in claim 13, wherein said at least one two-dimensional clustering bin is substantially centered with respect to at least one of said localized peak of said first histogram or said localized peak of said second histogram.

15. A method of processing an image of a visual scene as recited in claim 11, wherein said at least one two-dimensional clustering bin is located responsive at least one of a spatial first derivative of a spatial filtering of said first histogram or a spatial first derivative of a spatial filtering of said second histogram.

16. A method of processing an image of a visual scene as recited in claim 11, wherein at least one said point of said second plurality of said points is clustered with respect to a distance to said relatively-central location of previously clustered said second plurality of said points of said at least one two-dimensional clustering bin.

17. A method of processing an image of a visual scene as recited in claim 9, wherein an order in which said second plurality of said points are clustered is responsive to at least one of said down-range distance or said cross-range distance.

18. A method of processing an image of a visual scene as recited in claim 9, wherein an order in which said second plurality of said points are clustered responsive to a feasibility of collision with said object associated with said second plurality of said points.

19. A method of processing an image of a visual scene as recited in claim 9, wherein said first and second histograms are modified to remove at least one said point that has been clustered before selecting one or more additional said points to be clustered so as to provide for determining said at least one region-of-interest.

20. A method of processing an image of a visual scene as recited in claim 1, further comprising limiting said points that are clustered with respect to distance in said space of said first and second coordinates to a region in said space of said first and second coordinates.

21. A method of processing an image of a visual scene as recited in claim 20, wherein at least one of a cross-range size of said region or a down-range size of said region is predetermined.

22. A method of processing an image of a visual scene as recited in claim 20, wherein at least one of a cross-range size of said region or a down-range size of said region is responsive to an associated down-range distance.

23. A method of processing an image of a visual scene as recited in claim 20, wherein at least one of a cross-range size of said region or a down-range size of said region is responsive to an associated situational awareness at either a current time or a previous time.

24. A method of processing an image of a visual scene as recited in claim 1, wherein, for at least one said region-of-interest, further comprising finding an elliptical profile corresponding to a corresponding at least one group of said second plurality of said points clustered with respect to distance in said space of first and second coordinates.

25. A method of processing an image of a visual scene as recited in claim 24, wherein, for said at least one said region-of-interest, further comprising determining an orientation of said object associated with said at least one region-of-interest responsive to an orientation of either a major axis or a minor axis of said elliptical profile.

26. A method of processing an image of a visual scene as recited in claim 25, further comprising classifying an object associated with said at least one said region-of-interest responsive to at least one comparison of either a length of said major axis or a length of said minor axis with a corresponding threshold.

27. A method of processing an image of a visual scene as recited in claim 1, wherein said coordinates in said image space comprise either row and column pixel coordinates or a corresponding polar angle and radius relative to said relatively-central location.

28. A method of processing an image of a visual scene as recited in claim 1, wherein the operation of determining said locations of said plurality of edge points comprises determining a radial distance from said relatively-central location for each of said plurality of edge points so as to determine a corresponding plurality of radial distances from said relatively-central location.

29. A method of processing an image of a visual scene as recited in claim 28, further comprising transforming said plurality of radial distances from said relatively-central location to a corresponding plurality of sets of coordinates in said image space.

30. A method of processing an image of a visual scene as recited in claim 1, wherein the operation of determining said locations of said plurality of edge points is limited to a region within said image space.

31. A method of processing an image of a visual scene as recited in claim 30, wherein at least one of a width of said region within said image space, a height of said region within said image space, or a maximum radial distance from said relatively-central location of said region within said image space is predetermined.

32. A method of processing an image of a visual scene as recited in claim 30, wherein at least one of a width of said region within said image space, a height of said region within said image space, or a maximum radial distance from said relatively-central location of said region within said image space is responsive to a down-range distance of said relatively-central location.

33. A method of processing an image of a visual scene as recited in claim 30, wherein a maximum radial distance from said relatively-central location of said region within said image space is responsive to said outwardly-oriented direction.

34. A method of processing an image of a visual scene as recited in claim 30, wherein at least one of a width of said region within said image space, a height of said region within said image space, or a maximum radial distance from said relatively-central location of said region within said image space is responsive to an associated situational awareness at either a current time or a previous time.

35. A method of processing an image of a visual scene as recited in claim 1, wherein the operation of determining said location of said plurality of edge points comprises:

a. selecting or determining a polar search direction of a plurality of polar search directions or a corresponding said radial search path of a corresponding plurality of radial search paths;

b. selecting an image pixel along said corresponding said radial search path, wherein the operation of selecting said image pixel provides a selected image pixel to be processed;

c. determining whether said selected image pixel constitutes an edge point along said radial search path;

d. if said selected image pixel does not constitute said edge point along said radial search path, then selecting a next image pixel along said radial search path, wherein said next image pixel is radially outwards of said selected image pixel along or proximate to said radial search path; and

e. repeating steps c and d for said selected image pixel replaced by said next image pixel.

36. A method of processing an image of a visual scene as recited in claim 35, wherein said selected image pixel is one of a plurality of selected image pixels, the operation of determining said locations of said plurality of edge points provides for processing a plurality of said selected image pixels, and a first said selected image pixel comprises an image pixel at or most closely proximate to said relatively-central location.

37. A method of processing an image of a visual scene as recited in claim 35, wherein said next image pixel is selected in one of a row or column that is incrementally further distant from said relatively-central location relative to said selected image pixel, and the other of said row or column of said next image pixel is selected so as most closely follow said radial search path in accordance with a corresponding said polar search direction.

38. A method of processing an image of a visual scene as recited in claim 35, further comprising filtering said selected image pixel with an edge preserving smoothing filter prior to the operation of determining whether said selected image pixel constitutes an edge point along said radial search path.

39. A method of processing an image of a visual scene as recited in claim 38, wherein said edge preserving smoothing filter comprises a maximum homogeneity neighbor filter using an associated plurality of homogeneity regions, and each different homogeneity region of said associated plurality of homogeneity regions incorporates a corresponding different set of associated active pixels centered about a corresponding different corresponding said polar search direction.

40. A method of processing an image of a visual scene as recited in claim 35, wherein said image pixel along said corresponding said radial search path is one of a plurality of image pixels, further comprising:

a. differentiating said plurality of image pixels with respect to either index or distance along said radial search path so as to generate a corresponding plurality of spatial derivative values, and

b. determining a location of said edge point corresponding to one image pixel of said plurality of image pixels responsive to a magnitude of an absolute value of at least one spatial derivative value of said plurality of spatial derivative values.

41. A method of processing an image of a visual scene as recited in claim 40, wherein the operation of differentiating said plurality of image pixels comprises determining a substantially zero-phase-shift spatial derivative.

42. A method of processing an image of a visual scene as recited in claim 40, further comprising spatially filtering said plurality of spatial derivative values prior to the operation of determining said location of said edge point so as to generate a corresponding plurality of spatially-filtered spatial derivative values, wherein the operation of determining said location of said edge point corresponding to said one image pixel of said plurality of image pixels is responsive to a magnitude of an absolute value of at least one spatially filtered spatial derivative value of said plurality of spatially-filtered spatial derivative values.

43. A method of processing an image of a visual scene as recited in claim 42, wherein the operation of spatially filtering said plurality of spatial derivative values is in accordance with a Savitzky-Golay Filter method.

44. A method of processing an image of a visual scene as recited in claim 43, wherein the operation of differentiating said plurality of image pixels is inherently accomplished by said Savitzky-Golay Filter method.

45. A method of processing an image of a visual scene as recited in claim 42, further comprising:

a. determining a relatively distal location of a zero-crossing of said plurality of said spatially-filtered spatial derivative values, wherein said relatively distal location is relatively distal with respect to said edge point;

b. determining whether each of said plurality of said spatially-filtered spatial derivative values that are relatively distal with respect to said relatively distal location of said zero-crossing exhibits a measure of variance that is less than a threshold; and

c. if at least one of said plurality of said spatially-filtered spatial derivative values that are relatively distal with respect to said relatively distal location of said zero-crossing exhibits a measure of variance is greater than or equal to said threshold, then indicating that said edge point is indeterminate.

46. A method of processing an image of a visual scene as recited in claim 45, wherein said measure of variance is responsive to weighted sum of a subset of said plurality of said spatially-filtered spatial derivative values.

47. A method of processing an image of a visual scene as recited in claim 1, wherein said relatively-central location comprises a centroid.

48. An object detection system, comprising:

a. a ranging system, wherein said ranging system provides for generating a range-map image of a visual scene, wherein said range-map image comprises a first plurality of points, each point of said first plurality of points comprises a first coordinate and a second coordinate of a corresponding portion of a corresponding physical element within said visual scene, said first coordinate either corresponds or is related to a down-range coordinate of said visual scene, and said second coordinate either corresponds or is related to a cross-range coordinate of said visual scene;

b. an imaging system, wherein said imaging system provides for generating an image of said visual scene; and

c. at least one processor, wherein said at least one processor provides for;

i. determining at least one region-of-interest within said range-map image with respect to a top-down coordinate system;

ii. determining a corresponding relatively-central location of said at least one region-of-interest with respect to said top-down coordinate system;

iii. determining a corresponding location of said relatively-central location within said image; and

iv. radially searching a portion of said image surrounding said relatively-central location along a plurality of radial search paths extending from said relatively-central location so as to locate an edge boundary of an object within said portion of said image, wherein each different radial search path of said plurality of radial search paths is at a corresponding different outwardly-oriented direction.

49. An object detection system as recited in claim 48, wherein said ranging system comprises:

a. first and second cameras of a stereo-vision system; and

b. a processor that provides for generating said range-map image from associated first and second image signals respectively generated by said first and second cameras of said stereo-vision system.

50. An object detection system as recited in claim 48, wherein said ranging system comprises a system that measures range directly responsive to a generated beam of electromagnetic energy.

51. An object detection system as recited in claim 48, wherein said ranging system comprises either a radar system or a lidar system.

52. An object detection system as recited in claim 48, wherein imaging system comprises one of said first or second cameras of said stereo-vision system.

53. An object detection system as recited in claim 48, wherein said relatively-central location comprises a centroid.