Lost vehicle recovery utilizing associated feature pairs

• US 9,349,181 B2
• Filed: 06/26/2015
• Issued: 05/24/2016
• Est. Priority Date: 06/27/2014
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Chandrasekar, Kashyap, Estep, Ryan, Thomson, Jacob, Waltz, Lucas
• Primary Examiner(s): CHEUNG, CALVIN K
• Application Number:
  US14/751,383
• Publication Number:
  US 20150379704A1
• Time in Patent Office: 333 Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

According to the embodiments described herein, a method for lost vehicle recovery utilizing associated feature pairs may include, for a materials handling vehicle determined to be lost, creating a plurality of associated feature pairs by retrieving a subsequent set of camera data from a camera comprising two-dimensional UV space information, forming pairs from the UV space information, and associating each formed pair with pairs from each of a plurality of three-dimensional global feature points of an industrial facility map. The method may further include calculating a best estimate pose from calculated vehicle poses of the associated feature pairs based on a highest-rated pair of the plurality of associated feature pairs, using an accumulated odometry to update the best estimate pose to a current localized position, updating the seed position as the current localized position, and operating the materials handling vehicle utilizing the updated seed position.

Claims

1. A materials handling vehicle comprising a camera, one or more vehicular processors, a drive mechanism configured to move the materials handling vehicle along an inventory transit surface, a materials handling mechanism configured to store and retrieve goods in a storage bay of an industrial facility, and vehicle control architecture in communication with the drive and materials handling mechanisms, wherein:

the camera is communicatively coupled to the vehicular processor(s) and captures an input image of overhead features;

an industrial facility map comprises a plurality of three-dimensional global feature points associated with a mapping of the overhead features; and

the vehicular processor(s) of the materials handling vehicle executes vehicle functions to;

(i) determine a localized position of the materials handling vehicle with respect to an inventory transit surface of the industrial facility based on a seed position of the materials handling vehicle in the industrial facility and a dead reckoning associated with the materials handling vehicle;

(ii) use an initial set of camera data from the camera comprising two-dimensional UV space information to provide corrective feedback for the localized position;

(iii) use the localized position to track navigation of the materials handling vehicle along the inventory transit surface, navigate the materials handling vehicle along the inventory transit surface in at least a partially automated manner, or both;

(iv) determine if the materials handling vehicle is lost based on whether the industrial facility seed position comprises incorrect data, whether the camera data is insufficient to correct for an error associated with the dead reckoning, or both;

(v) if lost, create a plurality of associated feature pairs by retrieving a subsequent set of camera data from the camera comprising two-dimensional UV space information, forming pairs from the UV space information, and associating pairs from the UV space information with pairs from the plurality of three-dimensional global feature points of the industrial facility map;

(vi) calculate respective vehicle poses for associated feature pairs within a visual range of the camera;

(vii) calculate a best estimate pose of the materials handling vehicle from the calculated vehicle poses based on a highest-rated pair of the plurality of associated feature pairs;

(viii) use accumulated odometry to update the best estimate pose to a current localized position;

(ix) update the seed position as the current localized position; and

(x) operate the materials handling vehicle utilizing the updated seed position.

2. The materials handling vehicle of claim 1, wherein the vehicular processor(s) further executes vehicle functions to validate the current localized position prior to updating the seed position as the current localized position, such validation comprising functions to:

record the number of unique global feature points after each data association step until the number is above a threshold such that the seed position as the current localized position is validated.

3. The materials handling vehicle of claim 1, wherein the vehicular processor(s) further executes vehicle functions to indicate that the materials handling vehicle is found by publishing the seed position as the current localized position on a display after updating the seed position as the current localized position.

4. The materials handling vehicle of claim 1, wherein the vehicular processor(s) further executes vehicle functions to:

(i) transmit vehicle data indicative of the localized position to a management server; and

(ii) publish the localized position on a display associated with and communicatively coupled to the management server.

5. The materials handling vehicle of claim 1, wherein the plurality of associated feature pairs are created by data association steps comprising:

(i) creating UV space information pairs;

(ii) associating each global feature point with a first UV space feature in each UV space information pair; and

(iii) associating each global feature point within a maximum visual range of the camera with a second UV space feature in each UV space information pair.

6. The materials handling vehicle of claim 1, wherein the calculation of a vehicle pose for each associated feature pair within a visual range of the camera is based at least in part on use of a global localization algorithm, wherein the global localization algorithm comprises a formula to calculate a yaw of the vehicle about a first UV space feature of an associated feature pair that comprises an anchored feature, wherein the formula comprises:

(i) N is a cross product of two 3D vectors;

(ii) T is a target 3D point for yaw rotation;

7. The materials handling vehicle of claim 6, wherein the calculation of a best estimate pose of the materials handling vehicle based on a highest-rated pair of the plurality of associated feature pairs is based at least in part on a calculation of a pose sum error for each calculated vehicle pose wherein the highest-rated pair has the lowest pose sum error.

8. A system comprising a materials handling vehicle and a management server, wherein:

the materials handling vehicle comprises a camera, one or more vehicular processors, a drive mechanism configured to move the materials handling vehicle along an inventory transit surface, a materials handling mechanism configured to store and retrieve goods in a storage bay of an industrial facility, and vehicle control architecture in communication with the drive and materials handling mechanisms;

the camera is communicatively coupled to the vehicular processor(s) and the management server and captures an input image of overhead features;

the vehicular processor(s) is communicatively coupled to the management server;

an industrial facility map stored in the management server is communicatively coupled to the vehicular processor(s), wherein the industrial facility map comprises a plurality of three-dimensional global feature points associated with a mapping of the overhead features; and

the vehicular processor(s) of the materials handling vehicle executes vehicle functions to;

(i) determine a localized position of the materials handling vehicle with respect to an inventory transit surface of the industrial facility based on a seed position of the materials handling vehicle in the industrial facility and a dead reckoning associated with the materials handling vehicle;

(ii) use an initial set of camera data from the camera comprising two-dimensional UV space information to provide corrective feedback for the localized position;

(iii) use the localized position to track the navigation of the materials handling vehicle along the inventory transit surface, navigate the materials handling vehicle along the inventory transit surface in at least a partially automated manner, or both;

(iv) determine if the materials handling vehicle is lost;

(v) if lost, create a plurality of associated feature pairs by retrieving a subsequent set of camera data from the camera comprising two-dimensional UV space information, form pairs from the UV space information, and associate pairs from the UV space information with pairs from the plurality of three-dimensional global feature points of the industrial facility map;

(vi) calculate a vehicle pose for associated feature pairs within a visual range of the camera;

(vii) calculate a best estimate pose of the materials handling vehicle based on a highest-rated pair of the plurality of associated feature pairs;

(viii) use an accumulated odometry to update the best estimate pose to a current localized position;

(ix) update the seed position as the current localized position; and

(x) operate the materials handling vehicle utilizing the updated seed position.

9. The system of claim 8, wherein the vehicular processor(s) further executes vehicle functions to validate the current localized position prior to updating the seed position as the current localized position, such validation comprising functions to:

(i) record the number of unique global feature points after each data association step until the number is above a threshold such that the seed position as the current localized position is validated.

10. The system of claim 8, wherein the vehicular processor(s) further executes vehicle functions to indicate that the materials handling vehicle is found by publishing the seed position as the current localized position on a display after updating the seed position as the current localized position.

11. The system of claim 8, wherein the vehicular processor(s) further executes vehicle functions to:

(i) transmit vehicle data indicative of the localized position to the management server; and

(ii) publish the localized position on a display.

12. The system of claim 8, wherein the determination of whether the materials handling vehicle is lost is based at least in part on whether the industrial facility seed position comprises incorrect data.

13. The system of claim 8, wherein the determination of whether the materials handling vehicle is lost is based at least in part on whether the camera data is insufficient to correct for an error associated with the dead reckoning.

14. The system of claim 8, wherein the plurality of associated feature pairs are created by data association steps comprising:

(i) creating UV space information pairs;

(ii) associating each global feature point with a first UV space feature in each UV space information pair; and

(iii) associating each global feature point within a maximum visual range of the camera with a second UV space feature in each UV space information pair.

15. The system of claim 8, wherein the calculation of a vehicle pose for each associated feature pair within a visual range of the camera is based at least in part on use of a global localization algorithm, wherein the global localization algorithm comprises a formula to calculate a yaw of the vehicle about a first UV space feature of an associated feature pair that comprises an anchored feature, wherein the formula comprises:

(i) N is a cross product of two 3D vectors;

(ii) T is a target 3D point for yaw rotation;

16. The system of claim 15, wherein the calculation of a best estimate pose of the materials handling vehicle based on a highest-rated pair of the plurality of associated feature pairs is based at least in part on a calculation of a pose sum error for each calculated vehicle pose wherein the highest-rated pair has the lowest pose sum error.

17. A method of navigating or tracking the navigation of a materials handling vehicle along an inventory transit surface, the method comprising:

disposing a materials handling vehicle on an inventory transit surface of an industrial facility, wherein the materials handling vehicle is in communication with a management server and comprises a camera, one or more vehicular processors, a drive mechanism configured to move the materials handling vehicle along the inventory transit surface, a materials handling mechanism configured to store and retrieve goods in a storage bay of the industrial facility, and vehicle control architecture in communication with the drive and materials handling mechanisms;

utilizing the drive mechanism to move the materials handling vehicle along the inventory transit surface;

capturing an input image of overhead features of the industrial facility via the camera as the materials handling vehicle moves along the inventory transit surface;

providing an industrial facility map comprising a plurality of three-dimensional global feature points associated with a mapping of the ceiling lights;

determining via the vehicular processor(s) a localized position of the materials handling vehicle with respect to an inventory transit surface of the industrial facility based on a seed position of the materials handling vehicle in the industrial facility and a dead reckoning associated with the materials handling vehicle;

using an initial set of camera data from the camera comprising two-dimensional UV space information to provide corrective feedback for the localized position;

via the vehicular processor(s) and using the localized position, tracking the navigation of the materials handling vehicle along the inventory transit surface, navigating the materials handling vehicle along the inventory transit surface in at least a partially automated manner, or both;

determining if the materials handling vehicle is lost;

if lost, creating a plurality of associated feature pairs by retrieving a subsequent set of camera data from the camera comprising two-dimensional UV space information, forming pairs from the UV space information, and associating pairs from the UV space information with pairs from the plurality of three-dimensional global feature points of the industrial facility map;

via the vehicular processor(s), calculating a vehicle pose for associated feature pairs within a visual range of the camera;

via the vehicular processor(s), calculating a best estimate pose of the materials handling vehicle based on a highest-rated pair of the plurality of associated feature pairs;

using an accumulated odometry to update the best estimate pose to a current localized position via the vehicular processor(s);

updating the seed position as the current localized position; and

operating the materials handling vehicle utilizing the updated seed position.

18. The method of claim 17, further comprising:

validating the current localized position prior to updating the seed position as the current localized position, such validation comprising;

recording the number of unique global feature points after each data association step until the number is above a threshold such that the seed position as the current localized position is validated.

19. The method of claim 17, further comprising:

indicating that the materials handling vehicle is found by publishing the seed position as the current localized position on a display after updating the seed position as the current localized position.

20. The method of claim 17 wherein the management server is in communication with a display, a wireless communication circuit, and the vehicular processor(s), and the method comprises:

transmitting, via the wireless communication circuit, vehicle data indicative of the localized position to the management server; and

publishing the localized position on a display.

21. The method of claim 17, wherein the determination of whether the materials handling vehicle is lost is based at least in part on whether the industrial facility seed position comprises incorrect data.

22. The method of claim 17, wherein the determination of whether the materials handling vehicle is lost is based at least in part on whether the camera data is insufficient to correct for an error associated with the dead reckoning.

23. The method of claim 17, wherein the plurality of associated feature pairs are created by data association steps comprising:

creating UV space information pairs;

associating each global feature point with a first UV space feature in each UV space information pair; and

associating each global feature point within a maximum visual range of the camera with a second UV space feature in each UV space information pair.

24. The method of claim 17, wherein the calculation of a vehicle pose for each associated feature pair within a visual range of the camera is based at least in part on using a global localization algorithm, wherein the global localization algorithm comprises a formula to calculate a yaw of the vehicle about a first UV space feature of an associated feature pair that comprises an anchored feature, wherein the formula comprises:

(i) N is a cross product of two 3D vectors;

(ii) T is a target 3D point for yaw rotation;

25. The method of claim 24, wherein the calculation of a best estimate pose of the materials handling vehicle based on a highest-rated pair of the plurality of associated feature pairs is based at least in part on calculating a pose sum error for each calculated vehicle pose wherein the highest-rated pair has the lowest pose sum error.