Method for automatically generating planograms of shelving structures within a store

• US 10,625,426 B2
• Filed: 05/19/2017
• Issued: 04/21/2020
• Est. Priority Date: 05/19/2016
• Status: Active Grant

• Current Assignee: Simbe Robotics, Inc.
• Original Assignee: Simbe Robotics, Inc.
• Inventors: Bogolea, Bradley, Shah, Mirza Akbar, Safi, Jariullah, Fraser, Luke, Vandegrift, Lorin, Gee, Jeffrey
• Primary Examiner(s): Rudy, Andrew Joseph
• Application Number:
  US15/600,591
• Publication Number:
  US 20180005035A1
• Time in Patent Office: 1,068 Days
• Field of Search: 700216, 700217, 705 28
• CPC Class Codes: B25J 11/008 Manipulators for service tasks
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/00 Payment architectures, schemes or protocols apparatus for performing or posting payment transactions G07F7/08, G07F19/00; electronic cash registers G07G1/12
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

One variation of a method for automatically generating a planogram for a store includes: dispatching a robotic system to autonomously navigate within the store during a mapping routine; accessing a floor map of the floor space generated by the robotic system from map data collected during the mapping routine; identifying a shelving structure within the map of the floor space; defining a first set of waypoints along an aisle facing the shelving structure; dispatching the robotic system to navigate to and to capture optical data at the set of waypoints during an imaging routine; receiving a set of images generated from optical data recorded by the robotic system during the imaging routine; identifying products and positions of products in the set of images; and generating a planogram of the shelving segment based on products and positions of products identified in the set of images.

Claims

1. A method for automatically generating a new planogram assigning products to shelving structures within a store, the method comprising:

dispatching a robotic system to autonomously collect map data of a floor space within the store during a first mapping routine;

initializing the new planogram of the store, the new planogram representing locations of a set of shelving structures within the store based on map data recorded by the robotic system;

in response to receipt of confirmation of a preferred stocking condition in the store, dispatching the robotic system to record optical data at a first waypoint proximal a first shelving structure, in the set of shelving structures, during a first imaging routine;

accessing a first image comprising optical data recorded by the robotic system while occupying the first waypoint;

detecting a first shelf at a first vertical position in the first image;

detecting a first object in a first lateral position over the first shelf in the first image;

identifying the first object as a unit of a first product based on features extracted from a first region of the first image representing the first object;

projecting the first vertical position of the first shelf and the first lateral position of the first object onto a representation of the first shelving structure in the new planogram to define a first slot in the new planogram; and

based on confirmation of the preferred stocking condition in the store during the first imaging routine and in response to identifying the first object as the unit of the first product, writing an assignment for the first product to the first slot in the new planogram, the assignment defining a target stock condition of the first slot.

2. The method of claim 1:

further comprising;

receiving map data from the robotic system

transforming the map data into a floor map of the floor space within the store;

defining a coordinate system in the floor map;

identifying a first shelving structure represented in the floor map;

labeling the first shelving structure as a first shelving structure within the store; and

defining a first set of waypoints, relative to the coordinate system, along the first shelving structure, the first set of waypoints comprising the first waypoint and specifying orientations facing the first shelving structure; and

wherein dispatching the robotic system to record optical data at the first waypoint comprises dispatching the robotic system to navigate to each waypoint in the first set of waypoints and to record optical data of the first shelving structure while occupying each waypoint in the first set of waypoints.

3. The method of claim 2:

wherein initializing the new planogram comprises initializing a first elevation map of a first side of the first shelving structure in the new planogram based on a location and a length of the first shelving structure represented in the floor map; and

wherein projecting the first vertical position of the first shelf and the first lateral position of the first object onto the representation of the first shelving structure in the new planogram comprises defining the first slot in the first elevation map based on the first vertical position of the first shelf, the first lateral position of the first object, and a location and an orientation defined by the first waypoint.

4. The method of claim 1, further comprising:

detecting, in the first image, an open region laterally offset from the first object on the first shelf;

in response to detecting the open region on the first shelf;

detecting, in the first image, a second shelf tag on the first shelf adjacent the open region;

extracting an identifier of a second product from the second shelf tag represented in the first image;

projecting the first vertical position of the first shelf and a second lateral position of the open region onto the representation of the first shelving structure in the new planogram to define a second slot; and

writing an assignment for the second product to the second slot.

5. The method of claim 2, wherein accessing the first image comprises:

receiving a first set of digital photographic images recorded by the robotic system through a set of discrete color cameras integrated into the robotic system while navigating through the first set of waypoints; and

assembling the first set of digital photographic images into the first image of the first shelving structure based on known positions of the set of discrete color cameras with the robotic system and actual positions and orientations of the robotic system within the store while occupying each waypoint in the first set of waypoints.

6. The method of claim 1, wherein identifying the first object as the unit of the first product comprises:

extracting a first set of features from the first region of the first image; and

identifying the first object as the unit of the first product in response to correlation between features in the first set of features and features in a first template image, the first template image stored in a database of template images and representing the first product.

7. The method of claim 6:

further comprising;

detecting a second shelf at a second vertical position, offset from the first position, in the first image;

detecting a set of shelf tags arranged along the first shelf and the second shelf in the first image;

extracting a set of product identifiers from the set of shelf tags represented in the first image; and

retrieving a first set of template images from the database of template images, each template image in the first set of template images comprising visual features of a product specified in the set of product identifiers; and

wherein identifying the first object as the unit of the first product comprises;

comparing the first set of features extracted from the first region of the first image to template images in the set of template images; and

identifying the first object as the unit of the first product in response to a similarity score between features in the first set of features and features in the first template image, in the first set of template images, exceeding a threshold score.

8. The method of claim 6:

wherein extracting the first set of features from the first region of the first image comprises extracting a first color spectrum of relative color values from the first region of the first image;

further comprising retrieving a first set of template images from the database of template images, each template image in the first set of template images exhibiting a color spectrum of relative color values approximating the first color spectrum of relative color values; and

wherein identifying the first object as the unit of the first product comprises;

comparing the first set of features extracted from the first region of the first image to template images in the set of template images; and

identifying the first object as the unit of the first product in response to a similarity score between features in the first set of features and features in the first template image, in the first set of template images, exceeding a threshold score.

9. The method of claim 6:

wherein extracting the first set of features from the first region of the first image comprises;

extracting a first textual string from the first region of the first image; and

identifying a first brand associated with the first object based on the first textual string;

further comprising retrieving a first set of template images from the database of template images, each template image in the first set of template images representing a product associated with the first brand; and

wherein identifying the first object as the unit of the first product comprises;

comparing the first set of features extracted from the first region of the first image to template images in the set of template images; and

identifying the first object as the unit of the first product in response to a similarity score between features in the first set of features and features in the first template image, in the first set of template images, exceeding a threshold score.

10. The method of claim 6, further comprising:

detecting a second object in a second lateral position over the first shelf in the first image;

extracting a second set of features from a second region of the first image representing the second object;

in response to identifying the first object as the unit of the first product;

identifying a product category of the first product; and

retrieving a second set of template images from the database of template images, each template image in the second set of template images representing a product in the product category;

identifying the second object as a unit of a second product in response to a similarity score between features in the second set of features and features in a second template image, in the second set of template images, exceeding a threshold score, the second template image representing the second product;

projecting a second vertical position and a second lateral position of the second object onto the new planogram to define a second slot; and

writing an assignment for the second product to the second slot in the new planogram in response to identifying the second object as the unit of the second product.

11. The method of claim 6:

detecting a second object in a second lateral position over the first shelf in the first image;

extracting a second set of features from a second region of the first image representing the second object;

labeling the second object as unknown in response to lack of a template image, in the database of template images, defining features exhibiting at least a threshold similarity to features in the second set of features;

projecting a second vertical position and a second lateral position of the second object onto the new planogram to define a second slot;

generating a prompt to manually label the second object in the new planogram; and

serving the prompt and the second region of the first image representing the second object to an operator portal.

12. The method of claim 6:

wherein writing the assignment for the first product to the first slot in the new planogram comprises generating the new planogram of the store based on optical data collected by the robotic system during the first imaging routine; and

further comprising;

accessing a second image comprising optical data recorded by the robotic system while occupying the first waypoint during a second imaging routine succeeding the first imaging routine;

detecting a second object in a second region of the second image corresponding to the first slot in the new planogram;

extracting a second set of features from the second region of the second image;

determining that a unit of the first product is improperly stocked in the first slot in the first shelving structure in response to differences between features in the second set of features and features in template images, in the database of template images, of the first product assigned to the first slot by the new planogram; and

in response to determining that the unit of the first product is improperly stocked in the first slot in the first shelving structure, generating a first restocking prompt for the first product in the first slot.

13. The method of claim 1, wherein dispatching the robotic system to autonomously collect map data of the floor space within the store comprises dispatching the robotic system to execute the first mapping routine within the store following initial delivery of the robotic system to the store and in response to absence of a preexisting planogram of the store.

14. The method of claim 1, further comprising:

detecting, in the first image, a first shelf tag on the first shelf adjacent the first object;

extracting a first product identifier from the first shelf tag represented in the first image;

in response to the first product identifier indicating other than the first product, generating a prompt to correct the first shelf tag to indicate the first product; and

serving the prompt and the first region of the first image representing the first object to a manager portal.

15. The method of claim 1:

wherein detecting the first object comprises detecting a cluster of objects on the first shelf in the first image;

wherein identifying the first object as the unit of the first product comprises identifying each object in the cluster of objects as a unit of the first product based on features extracted from the first region of the first image representing the cluster of objects;

further comprising;

generating a count of units of the first product in the cluster of objects; and

extracting an arrangement of objects in the cluster of objects from the first region of the first image; and

wherein writing the assignment for the first product to the first slot in the new planogram comprises writing the count and the arrangement to the assignment for the first slot in the new planogram.

16. A method for automatically generating a planogram assigning products to shelving structures within a store, the method comprising:

in response to receipt of confirmation of a preferred stocking condition in the store, dispatching a robotic system to navigate along and to capture optical data of a set of shelving structures within the store during an imaging routine;

accessing a floor map of a floor space within the store generated by the robotic system from map data collected by the robotic system during the imaging routine;

receiving a set of images generated from optical data recorded by the robotic system during the imaging routine;

identifying a set of products depicted in the set of images;

identifying positions of the set of products in the store based on locations of products, in the set of products, depicted in images in the set of images and based on locations, in the floor map of the store, of the robotic system during capture of optical data associated with images in the set of images; and

based on confirmation of the preferred stocking condition in the store during the imaging routine;

generating the new planogram of the store; and

populating the new planogram with a set of assignments defining a target stock condition of the store, each assignment in the set of assignments specifying an identifier and a position of a product, in the set of products, in the store.

17. The method of claim 16:

wherein populating the new planogram with the set of assignments defining the target stock condition of the store comprises;

generating a first assignment specifying a first identifier of a first product and specifying a first position of a first slot in the store; and

writing the first assignment to the new planogram; and

further comprising;

dispatching the robotic system to execute a scan cycle succeeding the imaging cycle;

accessing a second image comprising optical data recorded by the robotic system during the scan cycle;

extracting a set of features from a region of the second image depicting the first slot specified in the new planogram;

identifying improper stocking of the first product in the first slot based on differences between features in the set of features and features in a set of template images of the first product; and

in response to identifying improper stocking of the first product in the first slot, generating a first restocking prompt for the first slot.

18. The method of claim 16:

wherein receiving the set of images from the robotic system comprises accessing a first image;

comprising optical data recorded by the robotic system while occupying a first location in the store; and

depicting a first shelving structure in the set of shelving structures;

wherein identifying a set of products depicted in the set of images and identifying positions of the set of products in the store comprises;

detecting a first shelf at a first vertical position in the first image;

detecting a first object in a first lateral position over the first shelf in the first image; and

identifying the first object as a unit of a first product based on features extracted from a first region of the first image representing the first object; and

wherein generating the new planogram and populating the new planogram with the set of assignments comprises;

projecting the first vertical position of the first shelf and the first lateral position of the first object onto a representation of the first shelving structure in the new planogram to define a first slot; and

writing an assignment for the first product to the first slot in the new planogram in response to identifying the first object as the unit of the first product.

19. The method of claim 18, wherein identifying the first object as the unit of the first product comprises:

extracting a first textual string from the first region of the first image;

identifying a first brand associated with the first object based on the first textual string;

retrieving a first set of template images from a database of template images, each template image in the first set of template images representing a product associated with the first brand;

comparing the first set of features extracted from the first region of the first image to template images in the set of template images; and

identifying the first object as the unit of the first product in response to a similarity score between features in the first set of features and features in the first template image, in the first set of template images, exceeding a threshold score.

20. The method of claim 16, wherein dispatching the robotic system to navigate through the floor space within the store comprises dispatching the robotic system to execute the imaging routine within the store following initial delivery of the robotic system to the store and in response to absence of a preexisting planogram of the store.