Pick to Augmented Reality

• US 20180005446A1
• Filed: 07/01/2016
• Published: 01/04/2018
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: InVia Robotics Inc.
• Original Assignee: InVia Robotics Inc.
• Inventors: Elazary, Lior, Voorhies, Randolph Charles, Parks, II, Daniel Frank
• Granted Patent: US 10,078,916 B2
• Time in Patent Office: Days
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

The embodiments provide an augmented reality device for increasing human efficiency in navigating a distribution site and retrieving different customer order items from distinct locations within the distribution site. The augmented reality device uses at least one camera to scan reference points distributed across the distribution site. From the scanning, the device determines its position relative to a destination within the distribution site. The device layers over a real-world view, navigational information directing movement of a device user along a most efficient path to the destination. The camera further scans items falling within its field of view. The device performs feature matching in order to identify a particular item of interest within the field of view. The device layers over the real-world view, visual makers distinguishing the particular item from other items as well instructions or other identifying information relating to the particular item.

Claims

1. A method comprising:

receiving a destination within a particular site at an augmented reality device;

scanning a plurality of reference points within a field of view of the augmented reality device;

determining a path to the destination based on said scanning of the plurality of reference points;

layering over a real-world view presented by the augmented reality device, navigational information directing movement along the path to the destination and positional information comprising a remaining distance to the destination; and

layering over a real-world view while at the destination, (i) an action for a user to execute on a particular item at the destination and (ii) a quantity of the particular item implicated by said action, wherein said action comprises one of restocking or retrieving said quantity of the particular item.

2. The method of claim 1 further comprising decoding location information from the plurality of reference points.

3. The method of claim 2 further comprising computing a current position of the augmented reality device based on said location information from the plurality of reference points.

4. The method of claim 3 wherein determining the path to the destination comprises computing a most efficient route from a plurality of available routes between the current position and the destination.

5. The method of claim 3 further comprising receiving a plurality of destinations at the augmented reality device.

6. The method of claim 5 further comprising determining a most efficient traversal of the plurality of destinations from the current position, reordering the plurality of destinations according to the most efficient traversal, and selecting the destination as a first destination from the plurality of destinations based on said reordering.

7. The method of claim 1 further comprising identifying arrival at the destination based on a particular reference point coming into the augmented reality device field of view, wherein identifying said arrival comprises layering a stopping position over the real-world view presented by the augmented reality device, and wherein the particular reference point encodes location information identifying the destination.

8. The method of claim 1 further comprising identifying arrival at the destination in response to identifying at least one of a plurality of items in the augmented reality device field of view.

9. The method of claim 8 further comprising layering over a real-world view presented by the augmented reality device, at least one marker identifying a particular item from the plurality of items.

10. The method of claim 1 further comprising detecting an orientation and position of the augmented reality device.

11. The method of claim 10 further comprising performing one of a transform computation or visual odometry on the plurality of reference points and determining the orientation and position of the augmented reality device based on said performing.

12. The method of claim 10 further comprising generating said navigational information relative to said orientation and position of the augmented reality device and updating the navigational information in response to the orientation and position changing based on device movements.

13. The method of claim 1 further comprising reducing inventory of an item at the destination in response to scanning one or more units of said item being removed from the destination and increasing inventory of the item at the destination in response to scanning one or more units of said item being added to the destination.

14. A method comprising

receiving visual identification features of a particular item from a first sensor of an augmented reality device, wherein the first sensor captures a real-world view before the augmented reality device and the particular item is within said real-world view;

detecting a current location of the augmented reality device in a distribution site using at least one of the first sensor or a different second sensor of the augmented reality device;

obtaining from a database storing visual identification features of a plurality of known items stored throughout the distribution site, features of a subset of the plurality of known items stored adjacent to said current location of the augmented reality device;

scanning the visual identification features of the particular item appearing within the real-world view before the augmented reality device against the visual identification features of the subset of known items stored adjacent to the current location of the augmented reality device;

matching the features of the particular item to features of a specific known item from the subset of known items based on said scanning; and

layering over the real-world view presented on a screen or semi-transparent display of the augmented reality device, at least one visual maker distinguishing the particular item from a plurality of other items appearing in the real-world view and identifying information automatically identified for the particular item based on said scanning and matching, wherein the identifying information comprises at least one of a name, image, weight, and inventory count of the specific known item.

15. The method of claim 14, wherein said matching comprises detecting the specific known item from the plurality of known items by identifying the specific known item as having a threshold number of matching visual identification features with visual identification features of the particular item.

17. The method of claim 14 further comprising layering over the real-world view, instructions a user is to perform with respect to the particular item.

18. The method of claim 14 further comprising receiving a destination where the particular item is expected and layering over the real-world view, navigational information directing movement along a path to the destination.

16. (canceled)

19. An augmented reality device comprising:

at least one camera continually imaging a plurality of navigational fiducials items within the camera field of view, wherein the plurality of navigational fiducials identify expected locations of the plurality of items;

a surface providing a real-world view before the augmented reality device camera; and

a processor processing said imaging from the camera and layering over the real-world view presented by the surface;

(i) navigational information derived from the plurality of navigational fiducials, said navigation information comprising visual directional indicators directing movement to a particular destination, (ii) a bounded region about a floor within the real-world view adjacent to a particular fiducial associated with the particular destination at which navigation stops and scanning for a particular item about the particular destination commences, and (iii) visual makers distinguishing a particular item from a subset of the plurality of items at the destination.

20. The augmented reality device of claim 19 further comprising at least one directional or movement sensor communicably coupled to said processor, said sensor tracking speed and rotational velocity of the augmented reality device, and wherein said processor adjusts the visual directional indicators relative to orientation and movement of the augmented reality device as computed from said rotational velocity and speed of movement tracked by the sensor.