Tracking vehicles in a warehouse environment

• US 10,528,828 B2
• Filed: 02/15/2019
• Issued: 01/07/2020
• Est. Priority Date: 05/30/2018
• Status: Active Grant

• Current Assignee: Lineage Logistics, LLC (Lineage Logistics Holdings, LLC)
• Original Assignee: Lineage Logistics, LLC (Lineage Logistics Holdings, LLC)
• Inventors: Eckman, Christopher Frank
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US16/277,338
• Publication Number:
  US 20190370567A1
• Time in Patent Office: 326 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/0833 Tracking
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

This specification generally discloses technology for tracking vehicle positions in a warehouse environment. A system receives stereoscopic image data from a camera on a forklift, in some implementations. The system recognizes an object that is represented in the stereoscopic image data, identifies a representation of the recognized object in a spatial model that identifies, for each of a plurality of objects in an environment, a corresponding location of the object in the environment, determines the location of the recognized object in the environment, determines a relative position between the forklift and the recognized object, based on a portion of the received stereoscopic image data that represents the recognized object, and determines a location of the forklift in the environment, based on the determined location of the recognized object in the environment, and the determined relative position between the forklift and the recognized object.

Claims

1. A computer-implemented method, comprising:

receiving first image data that includes one or more images received from a first camera, the first camera having been affixed to a warehouse vehicle for transporting pallets and having been positioned such that the first camera provides a first field of view that is to the left of the warehouse vehicle, the one or more images of the first image data depicting a scene from the first field of view;

receiving second image data that includes one or more images received from a second camera, the second camera having been affixed to the warehouse vehicle for transporting pallets and having been positioned such that the second camera provides a second field of view that is to the right of the warehouse vehicle, the one or more images of the second image data depicting a scene from the second field of view;

recognizing an object that is represented in (i) the first image data to the exclusion of the second image data or (ii) the second image data to the exclusion of the first image data;

identifying a representation of the recognized object in a spatial model that identifies, for each of a plurality of objects in an environment, a corresponding location of the object in the environment;

determining the location of the recognized object in the environment, as indicated by the spatial model;

determining a position of the warehouse vehicle relative to the recognized object, the position determined based on at least one of the first image data or the second image data that represents the recognized object; and

determining a location of the warehouse vehicle in the environment, the location of the warehouse vehicle determined based on (i) the determined location of the recognized object in the environment, and (ii) the determined position of the warehouse vehicle relative to the recognized object.

2. The computer-implemented method of claim 1, wherein the first image data and the second image data include respective series of stereoscopic images received in real time as the images are captured by the respective cameras, each stereoscopic image being captured at a fixed time interval.

3. The computer-implemented method of claim 1, further comprising determining that the recognized object is associated with a fixed location in the environment, including determining that the object'"'"'s location has not changed for a predetermined length of time.

4. The computer-implemented method of claim 1, further comprising determining that the recognized object is associated with a fixed location in the environment, including determining that the recognized object has been designated as a fixed location object.

5. The computer-implemented method of claim 4, further comprising:

while the warehouse vehicle is within a predefined area associated with the recognized object, determining locations of further recognized objects in the environment includes processing only a portion of the spatial model that corresponds to the predefined area.

6. The computer-implemented method of claim 1, further comprising:

receiving, from at least one environmental sensor other than the first or second camera, an environmental signal, wherein determining the location of the warehouse vehicle in the environment is based at least in part on the environmental signal.

19. The computer-implemented method of claim 1, wherein the first field of view of the first camera does not overlap with the second field of view of the second camera.

20. The computer-implemented method of claim 1, wherein the first field of view of the first camera partially overlaps with the second field of view of the second camera.

7. A computer-implemented method, comprising:

receiving image data that represents at least one image that was captured by a camera affixed to a warehouse vehicle for transporting pallets in an environment;

recognizing a first object and a second object that are shown in the at least one image represented in the image data;

identifying respective representations of the first object and the second object in a spatial model of the environment, wherein the spatial model indicates, for each of a plurality of objects in the environment, a corresponding location of the object in the environment and a level of confidence that the corresponding location is the object'"'"'s actual location;

evaluating the spatial model to determine a first location of the first object in the environment and a first level of confidence that the first location is the first object'"'"'s actual location;

evaluating the spatial model to determine a second location of the second object in the environment and a second level of confidence that the second location is the second object'"'"'s actual location; and

in response to determining that the first level of confidence for the first object'"'"'s location is greater than the second level of confidence for the second object'"'"'s location;

selecting the first object, rather than the second object, as a reference for determining a location of the warehouse vehicle in the environment; and

determining the location of the warehouse vehicle in the environment, based on a position of the warehouse vehicle relative to the first location of the first object in the environment.

8. The computer-implemented method of claim 7, wherein a given level of confidence indicated in the spatial model that a given location is a given object'"'"'s actual location is based on a permanence value that is proportional to an amount of time that the given object has been at the given location in the environment according to the spatial model.

9. The computer-implemented method of claim 7, further comprising:

receiving, from at least one environmental sensor other than the camera, an environmental signal, wherein determining the location of the warehouse vehicle in the environment is based at least in part on the environmental signal.

10. The computer-implemented method of claim 7, wherein the image data is based on a series of stereoscopic images received in real time as the images are captured by the camera, each stereoscopic image being captured at a fixed time interval.

11. The computer-implemented method of claim 7, further comprising (i) determining that the first object is associated with a fixed location in the environment according to the spatial model, and (ii) determining that the second object is not associated with a fixed location in the environment according to the spatial model.

12. The computer-implemented method of claim 11, wherein determining that the first object is associated with a fixed location according to the spatial model includes determining that the first object'"'"'s location has not changed for a predetermined length of time.

13. The computer-implemented method of claim 11, wherein determining that the first object is associated with a fixed location according to the spatial model includes determining that the first object has been designated as a fixed location object.

14. The computer-implemented method of claim 13, further comprising:

while the warehouse vehicle is within a predefined area associated with the first object, determining locations of further recognized objects in the environment includes processing only a portion of the spatial model that corresponds to the predefined area.

15. A non-transitory computer-readable storage medium having instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving image data that represents at least one image that was captured by a camera affixed to a warehouse vehicle for transporting pallets in an environment;

recognizing a first object and a second object that are shown in the at least one image represented in the image data;

identifying respective representations of the first object and the second object in a spatial model of the environment, wherein the spatial model indicates, for each of a plurality of objects in the environment, a corresponding location of the object in the environment and a level of confidence that the corresponding location is the object'"'"'s actual location;

evaluating the spatial model to determine a first location of the first object in the environment and a first level of confidence that the first location is the first object'"'"'s actual location;

evaluating the spatial model to determine a second location of the second object in the environment and a second level of confidence that the second location is the second object'"'"'s actual location; and

in response to determining that the first level of confidence for the first object'"'"'s location is greater than the second level of confidence for the second object'"'"'s location;

selecting the first object, rather than the second object, as a reference for determining a location of the warehouse vehicle in the environment; and

determining the location of the warehouse vehicle in the environment, based on a position of the warehouse vehicle relative to the first location of the first object in the environment.

16. The non-transitory computer-readable storage medium of claim 15, wherein a given level of confidence indicated in the spatial model that a given location is a given object'"'"'s actual location is based on a permanence value that is proportional to an amount of time that the given object has been at the given location in the environment according to the spatial model.

17. The non-transitory computer-readable storage medium of claim 15, wherein the image data is based on a series of stereoscopic images received in real time as the images are captured by the camera, each stereoscopic image being captured at a fixed time interval.

18. The non-transitory computer-readable storage medium of claim 15, the operations further comprising (i) determining that the first object is associated with a fixed location in the environment according to the spatial model, and (ii) determining that the second object is not associated with a fixed location in the environment according to the spatial model.