Product delivery unloading assistance systems and methods

• US 10,360,528 B2
• Filed: 11/03/2016
• Issued: 07/23/2019
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: High, Donald R., Winkle, David C., Atchley, Michael D.
• Primary Examiner(s): Brown, Jr., Howard D
• Application Number:
  US15/342,716
• Publication Number:
  US 20170132561A1
• Time in Patent Office: 992 Days
• CPC Class Codes: G06Q 10/0832 Special goods or special handling procedures, e.g. handling of hazardous or fragile goods
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

In some embodiments, apparatuses, systems and methods are provided herein useful to assist in unloading products. Some embodiments provide a product unloading assistance system, comprising: a control circuit; a graphical display; a camera; and a memory with the control circuit performing the steps of: receive an image from the camera comprising at least a portion of a product delivery vehicle and one or more reference points on the delivery vehicle; obtain an identifier of a load in the delivery vehicle to be unloaded; and cause the graphical display to display, in accordance with an orientation of the camera relative to the delivery vehicle based on the one or more reference points, 3D graphical representations of multiple different packaged products as loaded into the delivery vehicle and their orientation relative to each other as they are positioned within the delivery vehicle.

Claims

1. A product unloading assistance system, comprising:

a control circuit;

a graphical display communicationally coupled with the control circuit and configured to display three-dimensional (3D) graphical representations;

a camera communicationally coupled with the control circuit; and

a memory coupled to the control circuit and storing computer instructions that when executed by the control circuit cause the control circuit to perform the steps of;

receive an image from the camera comprising at least a portion of a product delivery vehicle to be unloaded and one or more reference points on the delivery vehicle, wherein the one or more reference points are readily identifiable from the image;

locate within the image the one or more reference points on the delivery vehicle;

obtain an identifier of a load in the delivery vehicle to be unloaded;

obtain an orientation of the camera capturing the image relative to the delivery vehicle;

generate, based on the identified load, one or more graphical representations illustrating 3D graphical representations of multiple different packaged products as loaded into the delivery vehicle and their orientation relative to the orientation of the camera relative to the delivery vehicle, and relative to each other as the packaged products are positioned within the delivery vehicle based on a tracked loading log of the delivery vehicle; and

cause the graphical display to display, in accordance with the orientation of the camera relative to the delivery vehicle based on the one or more reference points, the 3D graphical representations of the multiple different packaged products as loaded into the delivery vehicle and their orientation relative to each other as they are positioned within the delivery vehicle.

2. The system of claim 1, wherein the control circuit is further configured to cause the graphical display to display 3D outlined and transparent graphical representations of a first set of multiple of the packaged products and to display 3D graphical representations of one or more other of the packaged products as non-transparent such that the non-transparent 3D graphical representations of the one or more other packaged products are distinguished from the 3D outlined and transparent graphical representations of the first set of the packaged products.

3. The system of claim 2, wherein the control circuit is further configured to receive an identification of the one or more other packaged products requested to be distinguished from other packaged products in the delivery vehicle;

and

wherein the control circuit in causing the graphical display to display the 3D graphical representations further causes the displaying of the non-transparent 3D graphical representations of the identified one or more other packaged products graphically located consistent to three dimensional locations within the delivery vehicle, based on a tracked loading log, of the identified one or more packaged products.

4. The system of claim 1, wherein the control circuit is further configured to:

track movements of at least the camera relative to delivery vehicle; and

cause updates of the displaying of the 3D graphical representations of multiple different packaged products such that the updated displayed 3D graphical representations of the different packaged products continue to be consistent with the orientation of the camera relative to the delivery vehicle as the camera is moved.

5. The system of claim 1, wherein the control circuit is further configured to receive an identification of one or more packaged products that are indicated as removed from the delivery vehicle, wherein the control circuit in causing the display of the 3D graphical representations of the multiple different packaged products updates the graphical display such that graphical representations of the one or more package products indicated as having been removed are not displayed.

6. The system of claim 1, comprising:

a portable user interface unit comprising a portable user interface control circuit coupled with the camera, the graphical display and a wireless transceiver providing wireless communication between the portable user interface control circuit and the control circuit, wherein the transceiver wirelessly receives the 3D graphical representations of the multiple different packaged products as loaded into the delivery vehicle.

7. The system of claim 1, further comprising:

a graphics server comprising the control circuit configured to;

communicate, through the transceiver, the one or more graphical representations to a remote display device to be displayed on the graphical display of the display device.

8. The system of claim 1, wherein the control circuit is further configured to:

determine the orientation of the camera capturing the image relative to the orientation of the delivery vehicle based on sensor data;

obtain an identifier of the delivery vehicle; and

obtain the tracked loading log of the delivery vehicle.

18. The system of claim 1, comprising a portable user interface unit comprising the control circuit, the camera and the graphical display.

19. The system of claim 1, wherein the server control circuit is further configured to:

detect, from the image, the one or more reference points; and

obtain, based on the one or more reference points detected in the image, an identifier of the delivery vehicle;

wherein the obtaining the identifier of the load comprises identifying the identifier of the load as a function of the identifier of the delivery vehicle to be unloaded.

9. A method of assisting in unloading a delivery vehicle, comprising:

by a control circuit;

receiving an image from a camera, wherein the image comprises at least a portion of a product delivery vehicle to be unloaded and one or more reference points on the delivery vehicle that are readily identifiable from the image;

locating within the image the one or more reference points on the delivery vehicle;

obtaining an identifier of a load in the delivery vehicle to be unloaded;

obtaining an orientation of the camera capturing the image relative to the delivery vehicle;

generating, based on the identified load, one or more graphical representations illustrating 3D graphical representations of multiple different packaged products as loaded into the delivery vehicle and their orientation relative to the orientation of the camera relative to the delivery vehicle, and relative to each other as the packaged products are positioned within the delivery vehicle based on a tracked loading log of the delivery vehicle; and

causing a graphical display to display, in accordance with the orientation of the camera relative to the delivery vehicle based on the one or more reference points, the 3D graphical representations of the multiple different packaged products as loaded into the identified delivery vehicle and their orientation relative to each other as they are positioned within the delivery vehicle.

10. The method of claim 9, wherein the causing the graphical display to the 3D graphical representations comprises causing the display of 3D outlined and transparent graphical representations of a first set of multiple of the packaged products and to display 3D graphical representations of one or more other of the packaged products as non-transparent such that the non-transparent 3D graphical representations of the one or more other packaged products are distinguished from the 3D outlined and transparent graphical representations of the first set of the packaged products.

11. The method of claim 10, further comprising:

receiving an identification of the one or more other packaged products requested to be distinguished from other packaged products in the delivery vehicle; and

wherein the causing the display of the 3D graphical representations further comprises causing the displaying of the non-transparent 3D graphical representations of the identified one or more other packaged products graphically located consistent to three dimensional locations within the delivery vehicle, based on a tracked loading log, of the identified one or more other packaged products.

12. The method of claim 9, further comprising:

tracking movements of at least the camera relative to delivery vehicle; and

causing updates of the displaying of the 3D graphical representations of multiple different packaged products such that the updated displayed 3D graphical representations of the different packaged products continue to be consistent with the orientation of the camera relative to the delivery vehicle as the camera is moved.

13. The method of claim 9, further comprising:

receiving an identification of one or more packaged products that are indicated as removed from the delivery vehicle; and

causing the display of updated 3D graphical representations of the multiple different packaged products such that graphical representations of the one or more package products indicated as having been removed are not displayed.

14. The method of claim 9, further comprising:

wirelessly receiving, from a remote graphics server, the 3D graphical representations of the multiple different packaged products as loaded into the delivery vehicle at a portable user interface unit and displaying the 3D graphical representations of the multiple different packaged products on the graphical display of the portable user interface unit.

15. The method of claim 9, further comprising:

by a graphics server that comprises the control circuit and that is remote from a display device;

communicating, through a transceiver and to the display device, the one or more graphical representations to be displayed on the graphical display of the display device.

16. The method of claim 9, further comprising:

determining the orientation of the camera capturing the image relative to the orientation of the delivery vehicle;

obtaining an identifier of the delivery vehicle; and

orientation relative to each other as they are positioned within the delivery vehicle obtaining the tracked loading log of the delivery vehicle.

17. A product unloading assistance system, comprising:

a transceiver;

a control circuit;

a memory coupled to the control circuit and storing computer instructions that when executed by the control circuit cause the control circuit to perform the steps of;

receive one or more images of a delivery vehicle as captured by a camera of a remote graphical display device, wherein each of the one or more images includes at least a portion of the delivery vehicle and one or more reference points on the delivery vehicle, wherein the one or more reference points are readily identifiable from the image;

locate within at least one of the one or more images the one or more reference points on the delivery vehicle;

determine an orientation of the camera capturing the image relative to an orientation of the delivery vehicle;

obtain an identifier of a load in the delivery vehicle to be unloaded;

generate, based on the identified load, one or more graphical representations illustrating 3D graphical representations of multiple different packaged products as loaded into the delivery vehicle and their orientation relative to the orientation of the camera relative to the delivery vehicle and relative to the positioning of the other packaged products within the delivery vehicle; and

communicating, through the transceiver and to the display device, the one or more graphical representations to be displayed by the display device.