Industrial vehicles with overhead light based localization

• US 9,606,540 B2
• Filed: 09/22/2015
• Issued: 03/28/2017
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Bell, Mark, Chandrasekar, Kashyap, Waltz, Lucas B., Thomson, Jacob
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Pipala, Edward J
• Application Number:
  US14/861,626
• Publication Number:
  US 20160011595A1
• Time in Patent Office: 553 Days
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30172 Centreline of tubular or elongated structure
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/64 of convexity or concavity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

According to the embodiments described herein, a method for environmental based localization may include capturing an input image of a ceiling comprising a plurality of skylights. Features can be extracted from the input image. The features can be grouped into a plurality of feature groups such that each of the feature groups is associated with one of the skylights. Line segments can be extracted from the features of each feature group, automatically, with one or more processors executing a feature extraction algorithm on each feature group separately. At least two selected lines of the line segments of each feature groups can be selected. A centerline for each of the feature groups can be determined based at least in part upon the two selected lines. The center line of each of the feature groups can be associated with one of the skylights.

Claims

1. An industrial vehicle comprising a camera, a steering apparatus, a throttle, wheels, and one or more processors, wherein

the steering apparatus controls the orientation of at least one of the wheels;

the throttle controls a traveling speed of the industrial vehicle;

the camera is communicatively coupled to the one or more processors;

the camera captures an input image of ceiling lights of the ceiling of the warehouse; and

the one or more processors execute machine readable instructions to

associate raw features of the ceiling lights of the input image with one or more feature groups,

execute a Hough transform to transform the raw features of the one or more feature groups into line segments associated with the one or more feature groups,

determine a convex hull of the raw features of the one or more feature groups,

compare the line segments of the one or more feature groups and the convex hull in Hough space,

discard the line segments of the one or more feature groups that are outside of a threshold of similarity to the convex hull of the raw features of the one or more feature groups, whereby a preferred set of lines is selected for the one or more feature groups from the line segments of the one or more feature groups,

determine a centerline of the one or more feature groups from the preferred set of lines,

associate the centerline of the one or more feature groups with one of the ceiling lights of the input image, and

navigate the industrial vehicle through the warehouse utilizing the steering apparatus, the throttle, and the centerline of the one or more feature groups.

2. The industrial vehicle of claim 1, wherein the convex hull comprises hull line segments, and the line segments of the one or more feature groups are compared to the hull line segments.

3. The industrial vehicle of claim 2, wherein the one or more processors execute the machine readable instructions to:

convert the hull line segments into Hough space coordinates, wherein the hull line segments are infinite lines represented by coordinates ρ and

θ

.

4. The industrial vehicle of claim 1, wherein the one or more processors execute the machine readable instructions to:

rank the line segments of the one or more feature groups in order of strength; and

select a first edge line from the line segments of the preferred set of lines, wherein the first edge line is a highest ranked line of the line segments of the preferred set of lines.

5. The industrial vehicle of claim 4, wherein the first edge line is represented by coordinates ρ

and θ

.

6. The industrial vehicle of claim 5, wherein the one or more processors execute the machine readable instructions to:

select a second edge line from the line segments of the preferred set of lines, wherein the second edge line is selected based upon similarity to the θ

of the first edge line.

7. The industrial vehicle of claim 6, wherein the second edge line and the first edge line are separated by a distance threshold.

8. The industrial vehicle of claim 6, wherein the one or more processors execute the machine readable instructions to:

search the line segments of the preferred set of lines from a high rank to a low rank to select the second edge line.

9. The industrial vehicle of claim 6, wherein the one or more processors execute the machine readable instructions to:

find a vanishing point where the second edge line and the first edge line converge; and

calculate a line of bisection of the second edge line and the first edge line, wherein the centerline is calculated based upon the line of bisection.

10. The industrial vehicle of claim 1, wherein each of the one or more feature groups of the raw features are transformed separately into the line segments.

11. The industrial vehicle of claim 1, wherein the input image is underexposed to highlight the ceiling lights.

12. The industrial vehicle of claim 1, wherein the ceiling lights comprise skylights, and the one or more processors execute the machine readable instructions to:

extract raw feature contours from the skylights, wherein the raw features comprise the raw feature contours;

classify the raw feature contours as belonging to a skylights class; and

group the raw feature contours into the one or more feature groups, wherein the one or more feature groups comprises one group per unique skylight of the skylights, and wherein each of the one group comprises the raw feature contours of the unique skylight.

13. The industrial vehicle of claim 12, wherein the ceiling lights comprise round lights and merged lights, and the one or more processors execute the machine readable instructions to:

extract features from the round lights and the merged lights, wherein the raw features comprise the features from the round lights and the merged lights; and

classify the features from the round lights and the merged lights into a standard lights class and a merged lights class.

14. The industrial vehicle of claim 13, wherein the feature extracted from each round light is a centroid of the round light.

15. The industrial vehicle of claim 13, wherein the feature extracted from each merged light is two centroids of the merged light.

16. The industrial vehicle of claim 12, wherein the raw features comprise unwanted features, and the one or more processors execute the machine readable instructions to:

classify the unwanted features as noise.

17. The industrial vehicle of claim 12, wherein the raw feature contours are grouped into the one or more feature groups based upon relative proximity, and the one or more processors execute the machine readable instructions to:

calculate a minimum bounding rectangle for each of the raw feature contours; and

calculate the relative proximity based upon inter-feature distances of the minimum bounding rectangle for each of the raw feature contours.

18. The industrial vehicle of claim 1, wherein the one or more processors execute machine readable instructions to remove lens distortion effects from the input image captured by the camera.

19. The industrial vehicle of claim 1, wherein the one or more processors execute machine readable instructions to remove features outside of a region of interest from the input image.

20. The industrial vehicle of claim 1, wherein the one or more processors execute machine readable instructions to convert frame coordinates of the input image to environment based localization frame coordinates.

21. An industrial vehicle comprising a camera, a steering apparatus, a throttle, wheels, and one or more processors wherein:

the steering apparatus controls the orientation of at least one of the wheels;

the throttle controls a traveling speed of the industrial vehicle;

the camera is communicatively coupled to the one or more processors;

the camera is mounted to the industrial vehicle and focused to a ceiling of a warehouse;

the camera captures an input image of a ceiling light of the ceiling of the warehouse; and

the one or more processors execute machine readable instructions to

extract raw feature contours from the ceiling light of the input image of the ceiling,

group the raw feature contours into a feature group,

execute a Hough transform to transform the raw feature contours of the feature group into line segments associated with the feature group,

determine a convex hull of the raw feature contours of the feature group,

compare the line segments of the feature group and the convex hull in Hough space,

discard the line segments of the feature group that are outside of a threshold of similarity to the convex hull of the raw feature contours of the feature group, whereby a preferred set of lines is selected for the feature group from the line segments of the feature group,

determine a centerline of the feature group from the preferred set of lines of the feature group,

determine a pose of the industrial vehicle, a position of the industrial vehicle, or both based upon the centerline, and

navigate the industrial vehicle through the warehouse utilizing the steering apparatus, the throttle, and the pose of the industrial vehicle, the position of the industrial vehicle, or both.

22. The industrial vehicle of claim 21, wherein the input image comprises round lights and merged lights, and the one or more processors execute the machine readable instructions to:

extract features from the round lights and the merged lights of the input image;

extract unwanted features from the input image;

classify the features from the round lights and the merged lights into a standard lights class and a merged lights class; and

classify the unwanted features as noise.

23. The industrial vehicle of claim 21, wherein a distance threshold is utilized to group the raw feature contours into the feature group based upon relative proximity.

24. The industrial vehicle of claim 23, wherein the one or more processors execute the machine readable instructions to:

calculate a minimum bounding rectangle for each of the raw feature contours; and

calculate the relative proximity based upon inter-feature distances of the minimum bounding rectangle for each of the raw feature contours.

25. An industrial vehicle comprising a camera, one or more processors, a steering apparatus, wheels, and a throttle, wherein:

the steering apparatus controls the orientation of at least one of the wheels;

the throttle controls a traveling speed of the industrial vehicle;

the camera is communicatively coupled to the one or more processors;

the camera is mounted to the industrial vehicle and focused to a ceiling of a building structure;

the camera captures an input image of a skylight and a substantially circular light of a ceiling of a building structure; and

the one or more processors execute machine readable instructions to;

extract raw features from the skylight of the input image,

determine a convex hull of the raw features,

select a preferred set of lines from the raw features utilizing the convex hull of the raw features,

transform the substantially circular light of the input image into a point feature,

determine a centerline of the skylight from the preferred set of lines,

determine a pose of the industrial vehicle, a position of the industrial vehicle, or both, based upon the centerline of the skylight and the point feature transformed from the substantially circular light of the input image, and

navigate the industrial vehicle through the building structure utilizing the steering apparatus, throttle, and the pose of the industrial vehicle, the position of the industrial vehicle, or both.