Industrial vehicles with point fix based localization

• US 9,340,399 B2
• Filed: 10/01/2015
• Issued: 05/17/2016
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Bell, Mark
• Primary Examiner(s): Butler, Rodney
• Assistant Examiner(s): Mazzara, Anne
• Application Number:
  US14/872,830
• Publication Number:
  US 20160090281A1
• Time in Patent Office: 229 Days
• Field of Search: 701/28
• US Class Current: 1/1
• CPC Class Codes: B66F 9/063 Automatically guided
• CPC Class Codes: B66F 9/0755 Position control; Position detectors
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30172 Centreline of tubular or elongated structure
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection

Abstract

According to the embodiments described herein, an industrial vehicle can include a camera and one or more processors. The camera can capture an input image of ceiling lights of the ceiling of the warehouse. The one or more processors can execute machine readable instructions to detect transversal edges from the ceiling lights of the input image. The machine readable instructions executed by the one or more processors can determine points of convergence between the transversal edges and a first edge line and a second edge line. The machine readable instructions executed by the one or more processors can select a set of point fixes from the points of convergence. The machine readable instructions executed by the one or more processors can navigate the industrial vehicle through the warehouse utilizing the centerline and the set of point fixes.

Claims

1. An industrial vehicle comprising a camera and one or more processors, wherein:

the camera is communicatively coupled to the one or more processors;

the camera is mounted to the industrial vehicle and focused to a ceiling of a warehouse;

the camera captures an input image of ceiling lights of the ceiling of the warehouse; and

the one or more processors execute machine readable instructions to

provide a centerline of a row of ceiling light features in the captured input image by extracting raw feature components belonging to a class of ceiling lights from the captured input image as classified ceiling light features, associating the classified ceiling light features with separate ceiling lights, extracting ceiling light features from the raw feature components, filtering out ceiling light features not usable for determining the centerline, and determining the centerline from the remaining ceiling light features,

determine ceiling light corners by extracting raw feature components belonging to a class of ceiling lights from the captured input image as classified ceiling light features, selecting a usable portion of the ceiling lights for determining ceiling light corners, extracting ceiling light features from the raw feature components, filtering ceiling light features as usable or unusable for determining the ceiling light corners, determining the ceiling light corners from the remaining ceiling light features,

select a set of point fixes from the ceiling light corners, and

navigate the industrial vehicle through the warehouse utilizing the centerline and the set of point fixes.

2. The industrial vehicle of claim 1, wherein the one or more processors execute the machine readable instructions to:

set a region of interest that is coincident with a usable portion of the ceiling lights, wherein the extracted ceiling light features are detected from the usable portion of the ceiling lights.

3. The industrial vehicle of claim 2, wherein the region of interest is a bounding box that is axis aligned with the centerline.

4. The industrial vehicle of claim 3, wherein:

the extracted ceiling light features are line segments demarcated by end points; and

the region of interest is demarcated by one or more of the end points of the extracted ceiling light features.

5. The industrial vehicle of claim 3, wherein:

the centerline is demarcated by end points; and

the one or more processors execute the machine readable instructions to project the end points of the centerline upon the extracted ceiling light features to create intersection points, and wherein the region of interest is demarcated by one or more of the intersection points.

6. The industrial vehicle of claim 2, wherein:

the extracted ceiling light features overlap to define an overlapping portion; and

the region of interest is demarcated by the overlapping portion of the extracted ceiling light features.

7. The industrial vehicle of claim 2, wherein:

the extracted ceiling light features converge at a vanishing point;

the extracted ceiling light features are line segments demarcated by end points; and

the region of interest is determined utilizing the vanishing point.

8. The industrial vehicle of claim 1, wherein the extracted ceiling light features are detected using a Hough transform.

9. The industrial vehicle of claim 1, wherein the one or more processors execute the machine readable instructions to:

determine an average angle of the extracted ceiling light features; and

remove unusable edges from the extracted ceiling light features that are outside of a restricted range of angles on either side of the average angle of the extracted ceiling light features, whereby usable edges of the extracted ceiling light features remain, and wherein the set of point fixes are selected from the usable edges of the extracted ceiling light features.

10. The industrial vehicle of claim 1, wherein the ceiling light corners are determined by extrapolating the extracted ceiling light features.

11. The industrial vehicle of claim 10, wherein:

the set of point fixes are selected from the extracted ceiling light features having a length ratio above threshold level; and

the length ratio is given by a ratio of a length of each of the extracted ceiling light features to a length of a span between corresponding ceiling light corners.

12. The industrial vehicle of claim 1, wherein the one or more processors execute the machine readable instructions to:

label a point fix of the set of point fixes as corresponding to a transition of dark-to-bright or bright-to-dark in the input image.

13. A method for navigating an industrial vehicle comprising a camera and one or more processors, the method comprising:

capturing an input image of ceiling lights of a ceiling of a warehouse, wherein the input image is captured with the camera coupled to an industrial vehicle;

focusing the camera mounted to the industrial vehicle to a ceiling of the warehouse;

capturing an input image of the ceiling lights of the ceiling of the warehouse;

executing machine readable instructions with the one or more processors;

determining a centerline of a row of ceiling light features in of captured input image by extracting raw feature components belonging to a class of ceiling lights from the captured input image as classified ceiling light features, associating the classified ceiling light features with separate ceiling lights, extracting ceiling light features from the raw feature components, filtering out ceiling light features not usable for determining the centerline, and determining the centerline from the remaining ceiling light features;

determining ceiling light corners by extracting raw feature components belonging to a class of ceiling lights from the captured input image as classified ceiling light features, selecting a usable portion of the ceiling lights for determining ceiling light corners, extracting ceiling light features from the raw feature components, filtering ceiling light features as usable or unusable for determining the ceiling light corners, determining the ceiling light corners from the remaining ceiling light features;

selecting a set of point fixes from the ceiling light corners; and

navigating the industrial vehicle through the warehouse utilizing the pose, the position, or both.

14. The method of claim 13, further comprising:

extracting raw feature contours from the ceiling lights of the input image;

determining a convex hull from the ceiling lights of the input image, wherein the convex hull comprises hull line segments;

generating a mask from the hull line segments of the convex hull; and

selecting a selected portion from the raw feature contours, wherein the selected portion of the raw feature contours is coincident with the mask, and wherein the ceiling light features are extracted from the selected portion of the raw feature contours.

15. The method of claim 13, further comprising:

setting a region of interest that is coincident with a usable portion of the ceiling lights, wherein the extracted ceiling light features are detected from the usable portion of the ceiling lights.

16. The method of claim 15, further comprising using the centerline as a reference to set the region of interest.

17. The method of claim 13, further comprising:

determining an average angle of the extracted ceiling light features; and

removing unusable edges from the extracted ceiling light features that are outside of a restricted range of angles on either side of the average angle of the extracted ceiling light features, whereby usable edges of the extracted ceiling light features remain, and wherein the set of point fixes are selected from the usable edges of the extracted ceiling light features.

18. The method of claim 13, further comprising:

labeling a point fix of the set of point fixes as corresponding to a transition of dark-to-bright or bright-to-dark in the input image.

19. An industrial vehicle comprising a camera and one or more processors, wherein:

the camera is communicatively coupled to the one or more processors;

the camera is mounted to the industrial vehicle and focused to a ceiling of a warehouse;

the camera captures an input image of ceiling lights of the ceiling of the warehouse; and

the one or more processors execute machine readable instructions to determine a centerline of a row of ceiling lights from a row of ceiling light features in the captured input image by extracting raw feature components from the captured input image, separating the extracted raw feature components into different classes of ceiling lights, extracting ceiling light features from the extracted raw feature components based on the ceiling light class into which the extracted raw feature components are separated, filtering out selected ones of the extracted ceiling light features not usable for determining the centerline, and determining the centerline from the filtered ceiling light features,

determine ceiling light corners by extracting raw feature components from the captured input image, selecting a usable portion of the ceiling lights for determining ceiling light corners, extracting ceiling light features from the extracted raw feature components in the usable portion of the ceiling lights, filtering out selected ones of the extracted ceiling light features not usable for determining the ceiling light corners, determining the ceiling light corners from the filtered ceiling light features,

select a set of point fixes from the ceiling light corners, and

navigate the industrial vehicle through the warehouse utilizing the centerline and the set of point fixes.

20. A method for navigating an industrial vehicle comprising a camera and one or more processors, wherein the camera is communicatively coupled to the one or more processors, is mounted to the industrial vehicle, and is focused to a ceiling of a warehouse, and the method comprises:

using the camera to capture an input image of ceiling lights of the ceiling of a warehouse; and

executing machine readable instructions with the one or more processors to

determine a centerline of a row of ceiling lights from a row of ceiling light features in the captured input image by extracting raw feature components from the captured input image, separating the extracted raw feature components into different classes of ceiling lights, extracting ceiling light features from the extracted raw feature components based on the ceiling light class into which the extracted raw feature components are separated, filtering out selected ones of the extracted ceiling light features not usable for determining the centerline, and determining the centerline from the filtered ceiling light features,

determine ceiling light corners by extracting raw feature components from the captured input image, selecting a usable portion of the ceiling lights for determining ceiling light corners, extracting ceiling light features from the extracted raw feature components in the usable portion of the ceiling lights, filtering out selected ones of the extracted ceiling light features not usable for determining the ceiling light corners, determining the ceiling light corners from the filtered ceiling light features,

select a set of point fixes from the ceiling light corners, and

navigate the industrial vehicle through the warehouse utilizing the centerline and the set of point fixes.