Tracking vehicles in a warehouse environment

• US 10,242,273 B1
• Filed: 05/30/2018
• Issued: 03/26/2019
• Est. Priority Date: 05/30/2018
• Status: Active Grant

• Current Assignee: Lineage Logistics, LLC (Lineage, Inc.)
• Original Assignee: Lineage Logistics, LLC (Lineage, Inc.)
• Inventors: Eckman, Christopher Frank
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US15/993,343
• Time in Patent Office: 300 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/0833 Tracking
• CPC Class Codes: G06Q 50/28 Logistics, e.g. warehousing, loading, distribution or shipping
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

This specification generally discloses technology for tracking vehicle positions in a warehouse environment. A system receives stereoscopic image data from a camera on a forklift, in some implementations. The system recognizes an object that is represented in the stereoscopic image data, identifies a representation of the recognized object in a spatial model that identifies, for each of a plurality of objects in an environment, a corresponding location of the object in the environment, determines the location of the recognized object in the environment, determines a relative position between the forklift and the recognized object, based on a portion of the received stereoscopic image data that represents the recognized object, and determines a location of the forklift in the environment, based on the determined location of the recognized object in the environment, and the determined relative position between the forklift and the recognized object.

Claims

1. A computer-implemented method comprising:

recognizing a first object that is represented in first stereoscopic image data based on one or more stereoscopic images received from a first stereoscopic camera that has been affixed to a vehicle;

recognizing a second object that is represented in second stereoscopic image data based on one or more stereoscopic images received from a second stereoscopic camera that has been affixed to the vehicle;

identifying respective representations of the first object and the second object in a spatial model that tracks, for each of a plurality of objects, a corresponding location in an environment and a level of confidence that the corresponding location is the object'"'"'s actual location;

determining a first location of the first object in the environment and a first level of confidence that the first location is the first object'"'"'s actual location, according to the spatial model;

determining a second location of the second object in the environment and a second level of confidence that the second location is the second object'"'"'s actual location, according to the spatial model; and

in response to determining that the first level of confidence for the first object'"'"'s location is greater than the second level of confidence for the second object'"'"'s location;

determining a relative position between the vehicle and the first object, based on a portion of the first stereoscopic image data that represents the first object; and

determining a location of the vehicle in the environment, based on the determined first location of the first object in the environment, and the determined relative position between the vehicle and the first object.

2. The computer-implemented method of claim 1, wherein a level of confidence that a corresponding location is an object'"'"'s actual location is proportional to an amount of time that the object has been at the corresponding location, according to the spatial model.

3. The computer-implemented method of claim 1, wherein a level of confidence that a corresponding location is an object'"'"'s actual location is a highest level of confidence when the object has been designated as a fixed location object.

4. The computer-implemented method of claim 1, further comprising:

receiving, from at least one environmental sensor other than the stereoscopic camera, an environmental signal, wherein determining the location of the vehicle within the environment is based at least in part on the environmental signal.

5. The computer-implemented method of claim 3, wherein the object is designated as a fixed location object when the object'"'"'s location has not changed for a predetermined length of time.

6. The computer-implemented method of claim 3, further comprising:

while the vehicle is within a predefined area associated with the fixed location object, determining locations of further recognized objects in the environment, including processing only a portion of the spatial model that corresponds to the predefined area.

7. The computer-implemented method of claim 1, wherein the first and second stereoscopic image data are each based on a respective series of stereoscopic images received in real time as the images are captured by the first and second stereoscopic cameras, each stereoscopic image being captured at a fixed time interval.

8. A system, comprising:

a vehicle;

a first stereoscopic camera affixed to the vehicle;

a second stereoscopic camera affixed to the vehicle; and

a computing device communicatively coupled to each of the first and second stereoscopic cameras, the computing device configured to perform operations comprising;

recognizing a first object that is represented in first stereoscopic image data based on one or more stereoscopic images received from a first stereoscopic camera that has been affixed to a vehicle;

recognizing a second object that is represented in second stereoscopic image data based on one or more stereoscopic images received from a second stereoscopic camera that has been affixed to the vehicle;

identifying respective representations of the first object and the second object in a spatial model that tracks, for each of a plurality of objects, a corresponding location in an environment and a level of confidence that the corresponding location is the object'"'"'s actual location;

determining a first location of the first object in the environment and a first level of confidence that the first location is the first object'"'"'s actual location, according to the spatial model;

determining a second location of the second object in the environment and a second level of confidence that the second location is the second object'"'"'s actual location, according to the spatial model; and

in response to determining that the first level of confidence for the first object'"'"'s location is greater than the second level of confidence for the second object'"'"'s location;

determining a relative position between the vehicle and the first object, based on a portion of the first stereoscopic image data that represents the first object; and

determining a location of the vehicle in the environment, based on the determined first location of the first object in the environment, and the determined relative position between the vehicle and the first object.

9. The system of claim 8, wherein the vehicle is a forklift.

10. The system of claim 9, wherein the first and second stereoscopic cameras are each affixed to an overhead guard of the forklift such that the first stereoscopic camera points away from the forklift and to a side of the forklift and the second stereoscopic camera points away from the forklift and to an opposite side of the forklift.

11. The system of claim 10, wherein the first and second stereoscopic cameras each include a lens heater.

12. The system of claim 8, the operations further comprising:

receiving, from at least one environmental sensor other than the stereoscopic camera, an environmental signal, wherein determining the location of the vehicle within the environment is based at least in part on the environmental signal.

13. The system of claim 8, wherein the first and second stereoscopic image data are each based on a respective series of stereoscopic images received in real time as the images are captured by the first and second stereoscopic cameras, each stereoscopic image being captured at a fixed time interval.

14. A non-transitory computer-readable storage medium having instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

recognizing a first object that is represented in first stereoscopic image data based on one or more stereoscopic images received from a first stereoscopic camera that has been affixed to a vehicle;

recognizing a second object that is represented in second stereoscopic image data based on one or more stereoscopic images received from a second stereoscopic camera that has been affixed to the vehicle;

identifying respective representations of the first object and the second object in a spatial model that tracks, for each of a plurality of objects, a corresponding location in an environment and a level of confidence that the corresponding location is the object'"'"'s actual location;

determining a first location of the first object in the environment and a first level of confidence that the first location is the first object'"'"'s actual location, according to the spatial model;

determining a second location of the second object in the environment and a second level of confidence that the second location is the second object'"'"'s actual location, according to the spatial model; and

in response to determining that the first level of confidence for the first object'"'"'s location is greater than the second level of confidence for the second object'"'"'s location;

determining a relative position between the vehicle and the first object, based on a portion of the first stereoscopic image data that represents the first object; and

determining a location of the vehicle in the environment, based on the determined first location of the first object in the environment, and the determined relative position between the vehicle and the first object.

15. The non-transitory computer-readable storage medium of claim 14, wherein a level of confidence that a corresponding location is an object'"'"'s actual location is proportional to an amount of time that the object has been at the corresponding location, according to the spatial model.

16. The non-transitory computer-readable storage medium of claim 14, wherein a level of confidence that a corresponding location is an object'"'"'s actual location is a highest level of confidence when the object has been designated as a fixed location object.

17. The non-transitory computer-readable storage medium of claim 16, wherein the object is designated as a fixed location object when the object'"'"'s location has not changed for a predetermined length of time.

18. The non-transitory computer-readable storage medium of claim 16, the operations further comprising:

while the vehicle is within a predefined area associated with the fixed location object, determining locations of further recognized objects in the environment, including processing only a portion of the spatial model that corresponds to the predefined area.

19. The non-transitory computer-readable storage medium of claim 14, the operations further comprising:

receiving, from at least one environmental sensor other than the stereoscopic camera, an environmental signal, wherein determining the location of the vehicle within the environment is based at least in part on the environmental signal.

20. The non-transitory computer-readable storage medium of claim 14, wherein the first and second stereoscopic image data are each based on a respective series of stereoscopic images received in real time as the images are captured by the first and second stereoscopic cameras, each stereoscopic image being captured at a fixed time interval.