METHOD FOR MONITORING TEMPERATURE-CONTROLLED UNITS IN A STORE

• US 20190212760A1
• Filed: 01/10/2019
• Published: 07/11/2019
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Simbe Robotics, Inc.
• Original Assignee: Simbe Robotics, Inc.
• Inventors: Tiwari, Durgesh, Bogolea, Bradley
• Granted Patent: US 10,642,293 B2
• Time in Patent Office: Days
• CPC Class Codes: F25D 2700/16 Sensors measuring the temperature of products
• CPC Class Codes: F25D 29/00 Arrangement or mounting of control or safety devices
• CPC Class Codes: G01N 21/255 Details, e.g. use of specially adapted sources, lighting or optical systems
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 2201/0216 Vehicle for transporting goods in a warehouse, factory or similar
• CPC Class Codes: G05D 23/27543 using the controlled element as sensing element
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

One variation of a method for monitoring cooling units in a store includes: at a robotic system, during a first scan routine, autonomously navigating toward a cooling unit in the store, recording a color image of the cooling unit, and scanning a set of temperatures within the cooling unit; identifying a set of products stocked in the cooling unit based on features detected in the color image; mapping the set of temperatures to the set of products at a first time during the first scan routine based on positions of products in the set of products identified in the color image; and generating a record of temperatures of the set of products stocked in the cooling unit at the first time.

Claims

1. A method for monitoring cooling units in a store comprising:

at a robotic system, during a first scan routine;

autonomously navigating toward a cooling unit in the store;

recording a color image of the cooling unit; and

scanning a set of temperatures within the cooling unit;

identifying a set of products stocked in the cooling unit based on features detected in the color image;

mapping the set of temperatures to the set of products at a first time during the first scan routine based on positions of products in the set of products identified in the color image; and

generating a record of temperatures of the set of products stocked in the cooling unit at the first time.

2. The method of claim 1:

wherein autonomously navigating toward the cooling unit comprises autonomously navigating toward the cooling unit comprising an open cooling unit;

wherein scanning the set of temperatures within the cooling unit comprises, during the first scan routine, recording a thermal image of the open cooling unit; and

wherein mapping the set of temperatures to the set of products comprises, for a first product in the set of products identified in the color image;

retrieving a stored emissivity of a surface of the first product from a database based on an identity of the first product;

selecting a thermal pixel in the thermal image depicting the first product; and

calculating a temperature of the first product at the first time based on a thermal value stored in the thermal pixel and the emissivity of the surface of the first product.

3. The method of claim 2:

further comprising, during the first scan routine;

recording a depth image of the open cooling unit; and

recording an ambient temperature, an ambient humidity, and an ambient light level proximal the robotic system;

further comprising calculating a distance from the robotic system to the first product based on the depth image; and

wherein calculating the temperature of the first product at the first time comprises passing the emissivity of the surface of the first product, the thermal value stored in the thermal pixel, the distance, the ambient temperature, the ambient humidity, and the ambient light level into a physics model to calculate the temperature of the first product at the first time.

4. The method of claim 2:

wherein identifying the set of products stocked in the cooling unit comprises identifying a stock keeping unit of each product in the set of products based on features detected in the color image; and

wherein retrieving the stored emissivity of the surface of the first product comprises querying the database for the stored emissivity of packaging associated with a first stock keeping unit of the first product.

5. The method of claim 1:

wherein autonomously navigating toward the cooling unit comprises autonomously navigating toward the cooling unit comprising an enclosed cooling unit;

wherein scanning the set of temperatures within the cooling unit comprises, during the first scan routine;

broadcasting a query to a set of temperature-enabled transmitters arranged in the enclosed cooling unit; and

recording a set of temperature values received from the set of temperature-enabled transmitters; and

wherein mapping the set of temperatures to the set of products comprises;

accessing locations of the set of temperature-enabled transmitters in the enclosed cooling unit; and

mapping the set of temperature values received from the set of temperature-enabled transmitters to the set of products at the first time based on;

locations of the set of temperature-enabled transmitters in the enclosed cooling unit; and

locations of the set of products identified in the color image.

6. The method of claim 5:

wherein broadcasting the query for temperature values from the set of temperature-enabled transmitters comprises interrogating the set of temperature-enabled transmitters comprising a set of temperature-enabled radio frequency identification tags arranged in the enclosed cooling unit; and

wherein mapping the set of temperature values received from the set of temperature-enabled transmitters to the set of products at the first time comprises;

calculating a temperature gradient across the enclosed cooling unit by interpolating between the set of temperature values based on locations of corresponding temperature-enabled transmitters, in the set of temperature-enabled transmitters, in the enclosed cooling unit;

projecting boundaries of the set of products, identified in the color image, onto the temperature gradient; and

for each product in the set of products, extracting a temperature of the product from a region of the temperature gradient contained within the boundary of the product projected onto the temperature gradient.

7. The method of claim 5:

wherein recording the set of temperature values received from the set of temperature-enabled transmitters comprises recording the set of temperature values received from the set of temperature-enabled transmitters, each temperature value in the set of temperature values paired with an identifier; and

wherein accessing locations of the set of temperature-enabled transmitters in the enclosed cooling unit comprises;

detecting a set of optical fiducials representing the set of temperature-enabled transmitters in the color image; and

for each temperature value in the set of temperature values, linking the temperature value to a location of an optical fiducial, in the set of optical fiducials, associated with the identifier received with the temperature value.

8. The method of claim 5:

wherein autonomously navigating toward the cooling unit in the store comprises, during the scan routine, autonomously navigating through a sequence of waypoints defined within the store, the sequence of waypoints comprising a particular waypoint adjacent the cooling unit; and

wherein scanning the set of temperatures within the cooling unit comprises broadcasting the query for temperature values in response to occupying the particular waypoint that indicates proximity of temperature-enabled transmitters.

9. The method of claim 5, further comprising:

calculating a temperature gradient across the enclosed cooling unit by interpolating between the set of temperature values based on locations of corresponding temperature-enabled transmitters, in the set of temperature-enabled transmitters, in the enclosed cooling unit;

calculating a spatial temperature variance in the temperature gradient across the enclosed cooling unit at the first time; and

in response to the spatial temperature variance of the cooling unit exceeding a threshold variance, serving a maintenance prompt to inspect the cooling unit to a computing device affiliated with an associate of the store.

10. The method of claim 1:

wherein autonomously navigating toward the cooling unit comprises autonomously navigating toward the cooling unit comprising an enclosed cooling unit;

wherein scanning the set of temperatures within the cooling unit comprises, during the first scan routine;

broadcasting a query to a temperature-enabled transmitter coupled to the enclosed cooling unit; and

recording a temperature value received from the temperature-enabled transmitter; and

wherein mapping the set of temperatures to the set of products comprises mapping the temperature value received from the temperature-enabled transmitter to a subset of products, in the set of products, proximal a known location of the temperature-enabled transmitter.

11. The method of claim 1, further comprising:

accessing a temperature requirement of a first product in the set of products detected in the color image;

based on the record, verifying that a temperature of the first product at the first time falls within a temperature range specified in the temperature requirement; and

in response to the temperature of the first product at the first time falling outside of the temperature range;

initializing an electronic notification;

identifying the first product in the electronic notification;

identifying the enclosed cooling unit in the electronic notification;

writing a prompt to remove the first product from the enclosed cooling unit to the electronic notification; and

serving the electronic notification to a computing device affiliated with an associate of the store.

12. The method of claim 1, wherein identifying the set of products stocked in the cooling unit comprises:

detecting a first shelf of the cooling unit in a first region of the color image;

identifying an address of the first shelf;

based on the address of the first shelf, retrieving a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of a product in the first list of products assigned to the first shelf by a planogram of the store;

extracting a first set of features from the first region of the color image; and

identifying presence of a first subset of products, in the first list of products, on the first shelf based on alignment of the first set of features and template images in the first set of template images.

13. The method of claim 12, further comprising:

detecting absence of a first product, in the first list of products, on the first shelf in response to absence of features, in the first set of features, that align with template images, in the first set of template images, depicting the first product;

in response to detecting absence of the first product on the first shelf, generating a restocking prompt to restock the first shelf with units of the first product; and

serving the restocking prompt to a computing device affiliated with an associate of the store.

14. The method of claim 1, further comprising:

at the robotic system, during the first scan routine;

autonomously navigating toward a heating unit in the store;

recording a second color image of the heating unit; and

scanning a second set of temperatures within the heating unit;

identifying a second set of products stocked in the heating unit based on features detected in the second color image;

mapping the second set of temperatures to the second set of products at a second time during the first scan routine based on positions of products in the second set of products identified in the second color image; and

appending the record with temperatures of the second set of products stocked in the heating unit at the second time.

15. The method of claim 1, further comprising:

calculating a mean temperature of the cooling unit at the first time based on the set of temperatures within the cooling unit during the first scan routine;

accessing a current temperature setting of the cooling unit; and

in response to the mean temperature of the cooling unit deviating from the current temperature setting by more than a threshold difference, serving a maintenance prompt to inspect the cooling unit to a computing device affiliated with an associate of the store.

16. The method of claim 1, further comprising:

at the robotic system, during a second scan routine succeeding the first scan routine;

autonomously navigating toward the cooling unit in the store;

recording a second color image of the cooling unit; and

scanning a second set of temperatures within the cooling unit;

identifying the set of products stocked in the cooling unit at a second time during the second scan routine based on features detected in the second color image;

mapping the second set of temperatures to the set of products based on positions of products in the set of products identified in the second color image;

estimating temperature exposures of products in the set of products between the first time and the second time by integrating between the first set of temperatures and the second set of temperatures of the set of products based on a time duration between the first time and the second time.

17. The method of claim 16, further comprising:

accessing a shelf life model of a particular product in the set of products;

estimating a remaining shelf life of the particular product based on the shelf life model and a temperature exposure of the particular product; and

in response to the shelf life of the particular product falling below a threshold duration;

generating an electronic notification identifying the particular product and the remaining shelf life of the particular product; and

serving the electronic notification to a computing device affiliated with an associate of the store.

18. A method for monitoring temperature-controlled units in a store comprising:

at a robotic system, during a first scan routine;

autonomously navigating toward a temperature-controlled unit in the store;

recording a color image and a thermal image of the temperature-controlled unit; and

identifying a set of products stocked in the temperature-controlled unit based on features detected in the color image;

for each product in the set of products stocked in the temperature-controlled unit;

retrieving a stored emissivity of a surface of the product from a database based on an identity of the first product;

selecting a thermal pixel in the thermal image depicting the product; and

calculating a temperature of the product at the first time based on a thermal value stored in the thermal pixel and the emissivity of the surface of the product; and

generating a record of temperatures of products, in the set of products, at the first time.

19. The method of claim 18:

further comprising, during the first scan routine;

recording a depth image of the open cooling unit; and

recording an ambient temperature, an ambient humidity, and an ambient light level proximal the robotic system; and

wherein calculating the temperature of each product in the set of products at the first time comprises, for each product in the set of products;

calculating a distance from the robotic system to the product based on the depth image; and

passing the emissivity of the surface of the product, the thermal value stored in the thermal pixel, the distance, the ambient temperature, the ambient humidity, and the ambient light level into a physics model to calculate the temperature of the product at the first time.

20. A method for monitoring temperature-controlled units in a store comprising:

at a robotic system, during a first scan routine;

autonomously navigating toward a temperature-controlled unit in the store;

recording a color image of the temperature-controlled unit; and

broadcasting a query for temperature values from a set of temperature-enabled transmitters coupled to the temperature-controlled unit; and

recording a set of temperature values transmitted by the set of temperature-enabled transmitters;

identifying a set of products stocked in the temperature-controlled unit based on features detected in the color image;

accessing locations of the set of temperature-enabled transmitters coupled to the temperature-controlled unit;

mapping the set of temperature values received from the set of temperature-enabled transmitters to the set of products at the first time based on;

locations of the set of temperature-enabled transmitters in the temperature-controlled unit; and

locations of the set of products identified in the color image; and

generating a record of temperatures of the set of products stocked in the temperature-controlled unit at the first time.