Method for detecting and responding to spills and hazards

• US 10,684,623 B2
• Filed: 01/10/2019
• Issued: 06/16/2020
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Simbe Robotics, Inc.
• Original Assignee: Simbe Robotics, Inc.
• Inventors: Tiwari, Durgesh, Bogolea, Bradley
• Primary Examiner(s): Shudy, Angelina
• Assistant Examiner(s): Algehaim, Mohamed Abdo
• Application Number:
  US16/245,117
• Publication Number:
  US 20190235511A1
• Time in Patent Office: 523 Days
• Field of Search: 701 23
• CPC Class Codes: G01B 21/18 for measuring depth
• CPC Class Codes: G01B 21/28 for measuring areas
• CPC Class Codes: G01N 21/27 using photo-electric detection G01N21/31 takes precedence; circuits for computing concentration logarithmic circuits G06G7/24; photometric circuits in general G01J
• CPC Class Codes: G01N 25/00 Investigating or analyzing materials by the use of thermal means G01N3/00 - G01N23/00 take precedence
• CPC Class Codes: G01V 9/005 by thermal methods, e.g. after generation of heat by chemical reactions
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

One variation of a method for detecting and responding to hazards within a store includes: autonomously navigating toward an area of a floor of the store; recording a thermal image of the area; recording a depth map of the area of the floor; detecting a thermal gradient in the thermal image; scanning a region of the depth map, corresponding to the thermal gradient detected in the thermal image, for a height gradient; in response to detecting the thermal gradient in the thermal image and in response to detecting absence of a height gradient in the region of the depth map, predicting presence of a fluid within the area of the floor; and serving a prompt to remove the fluid from the area of the floor of the store to a computing device affiliated with the store.

Claims

1. A method for detecting and responding to hazards within a store comprising:

at a robotic system, during a scan cycle;

autonomously navigating toward an area of a floor of the store;

recording a thermal image of the area of the floor;

recording a depth map of the area of the floor;

recording a color image of the area of the floor;

detecting a thermal gradient in the thermal image;

scanning a region of the depth map, corresponding to the thermal gradient detected in the thermal image, for a height gradient; and

scanning a region of the color image, corresponding to the thermal gradient in the thermal image, for a color gradient;

in response to detecting the thermal gradient in the thermal image and in response to detecting absence of the height gradient in the region of the depth map, predicting presence of a fluid within the area of the floor;

identifying the fluid as clear in response to detecting absence of the color gradient in the region of the depth map; and

in response to identifying the fluid as clear, serving a prompt to remove the fluid from the area of the floor of the store and identifying the fluid as a clear liquid to a computing device affiliated with the store.

2. The method of claim 1:

further comprising;

identifying an aisle of the store proximal the fluid based on a location of the area of the floor of the store;

querying a planogram of the store for a list of products stocked in the aisle;

in response to the list of products comprising a packaged oil product, predicting that the fluid comprises an oil;

wherein serving the prompt to the computing device comprises;

populating an electronic notification with the prompt to remove the fluid from the area of the floor;

indicating the location of the area of the floor in the store in the electronic notification;

inserting a recommendation for dry absorbent to remove the fluid into the electronic notification; and

transmitting the electronic notification to the computing device affiliated with an associate of the store.

3. The method of claim 1:

further comprising, during the scan cycle;

autonomously navigating toward a second area of the floor of the store;

recording a second thermal image of the second area of the floor;

recording a second depth map of the second area of the floor;

recording a second color image of the second area of the floor;

detecting a second thermal gradient in the second thermal image;

scanning a second region of the second depth map, corresponding to the second thermal gradient detected in the second thermal image, for a second height gradient; and

scanning a second region of the second color image, corresponding to the second thermal gradient in the second thermal image, for a second color gradient;

further comprising;

in response to detecting the second thermal gradient in the second thermal image and in response to detecting absence of the second height gradient in the second region of the second depth map, predicting presence of a second fluid within the second area of the floor;

identifying the second fluid as colored in response to detecting the second color gradient in the second region of the second color image; and

wherein serving the prompt to the computing device comprises serving the prompt, to the computing device, further specifying removal of the second fluid from the second area of the floor of the store and identifying the second fluid as a colored liquid.

4. The method of claim 1:

further comprising, during the scan cycle;

estimating an opacity of the fluid, proportional to the color gradient;

calculating a priority for removal of the fluid from the area of the floor of the store inversely proportional to the opacity of the fluid;

wherein serving the prompt to the computing device comprises;

inserting the prompt to remove the fluid from the area of the floor into an electronic notification;

inserting the priority for removal of the fluid from the area of the floor into the electronic notification; and

transmitting the electronic notification to the computing device affiliated with an associate of the store.

5. The method of claim 1, further comprising:

detecting a perimeter of the fluid in the area of the floor of the store based on the thermal gradient in the thermal image;

autonomously navigating toward the perimeter of the fluid;

holding proximal the perimeter of the fluid; and

outputting an indicator of presence of the fluid.

6. The method of claim 5:

wherein holding proximal the perimeter of the fluid comprises holding proximal the perimeter of the fluid to physically block access to the area of the floor of the store; and

wherein outputting the indicator of presence of the fluid comprises, while holding proximal the perimeter of the fluid;

rendering a notification of presence of the fluid nearby on a display integrated into the robotic system; and

outputting an audible alert.

7. The method of claim 1, further comprising:

detecting a perimeter of the fluid in the area of the floor of the store based on the thermal gradient in the thermal image; and

autonomously navigating around the fluid at greater than a threshold distance from the perimeter of the fluid.

8. The method of claim 1:

further comprising, during the scan cycle;

autonomously navigating along a set of aisles within the store; and

while autonomously navigating along a particular aisle in the set of aisles, recording a set of color images of a set of shelving structures facing the particular aisle;

wherein recording the thermal image comprises recording the thermal image at a first time while autonomously navigating along the particular aisle in the set of aisles;

wherein recording the depth map comprises recording the depth map at approximately the first time while autonomously navigating along the particular aisle; and

wherein serving the prompt to the computing device comprises serving the prompt, identifying the particular aisle in the set of aisles in the store, to the computing device.

9. The method of claim 8, further comprising:

detecting a first shelf in a first shelving structure in a first region of a first color image, in the set of color images, recorded at approximately the first time;

identifying an address of the first shelf;

based on the address of the first shelf, retrieving a first list of products assigned to the first shelf by a planogram of the store;

retrieving a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of a product in the first list of products;

extracting a first set of features from the first region of the first color image;

determining that a unit of the first product is improperly stocked on the first shelf in response to deviation between features in the first set of features and features in the first template image; and

in response to determining that the unit of the first product is improperly stocked on the first shelf, generating a first restocking prompt for the first product on the first shelf.

10. The method of claim 1, further comprising:

identifying a first set of aisles, in a set of aisles within the store, assigned at least one product comprising a liquid by a planogram of the store;

identifying a second set of aisles, in the set of aisles within the store, assigned dry goods and excluding products containing fluids by the planogram of the store; and

at the robotic system;

autonomously navigating along the first set of aisles within the store at a first frequency during the scan cycle; and

autonomously navigating along the second set of aisles within the store at a second frequency during the scan cycle, the second frequency less than the first frequency.

11. The method of claim 1, further comprising:

accessing a history of locations of fluid spill events in the store;

based on the history;

identifying a first aisle, in a set of aisles in the store, containing locations of a first quantity of historical fluid spill events over a period of time; and

identifying a second aisle, in the set of aisles in the store, containing locations of a second quantity of historical fluid spill events, less than the first quantity of historical fluid spill events, over the period of time;

based on the first quantity of historical fluid spill events, scheduling the robotic system to traverse the first aisle at a first frequency during the scan cycle; and

based on the second quantity of historical fluid spill events, scheduling the robotic system to traverse the second aisle at a second frequency, less than the first frequency, during the scan cycle.

12. The method of claim 1, further comprising:

at the robotic system, during the scan cycle;

autonomously navigating toward a second area of the floor of the store;

recording a second thermal image of the second area of the floor of the store;

recording a second depth map of the second area of the floor;

detecting a second thermal gradient in the second thermal image; and

scanning a second region of the second depth map, corresponding to the second thermal gradient detected in the second thermal image, for a second height gradient greater than a minimum height threshold; and

in response to detecting the second thermal gradient in the second thermal image and in response to detecting the second height gradient greater than the minimum height threshold in the second region of the second depth map;

predicting presence of a hazardous object on the floor of the store and within the second area of the floor; and

serving a second prompt to remove the hazardous object from the second area of the floor of the store to the computing device.

13. The method of claim 12, wherein predicting presence of the hazardous object on the floor of the store comprises predicting presence of the hazardous object on the floor of the store and within the second area of the floor in response to:

detecting the second thermal gradient in the second thermal image; and

detecting the second height gradient less than a maximum height threshold in the second region of the second depth map aligned with the second thermal gradient detected in the second thermal image.

14. The method of claim 1:

wherein scanning the region of the depth map for the height gradient comprises;

projecting a floor plan of the store onto the depth map to isolate a segment of the depth map representing the floor of the store and excluding a fixed display near the area of the floor in the store;

projecting a ground plane onto the segment of the depth map; and

scanning the segment of the depth map for an object offset above the ground plane; and

wherein predicting presence of the fluid within the area of the floor comprises predicting presence of the fluid within the area of the floor in response to;

detecting the thermal gradient in the thermal image; and

detecting absence of the object offset above the ground plane in the segment of the depth map.

15. The method of claim 1:

further comprising;

after recording the thermal image of the area of the floor of the store, recording a sequence of thermal images depicting the area of the floor of the store;

scanning the sequence of thermal images for thermal gradients proximal the area of the floor of the store; and

characterizing a spatial rate of change of thermal gradients, proximal the area of the floor of the store, detected in the sequence of thermal images;

wherein predicting presence of the fluid within the area of the floor comprises predicting presence of the fluid within the area of the floor in response to;

detecting the thermal gradient in the thermal image;

the spatial rate of change of thermal gradients detected in the sequence of thermal images falling below a threshold rate of change; and

detecting absence of the height gradient in a segment of the depth map cospatial with the thermal gradient in the thermal image; and

further comprising identifying the thermal gradient in the thermal image as other than fluid in the area of the floor of the store in response to the spatial rate of change of thermal gradients detected in the sequence of thermal images exceeding the threshold rate of change.

16. The method of claim 15, wherein predicting presence of the fluid within the area of the floor comprises, in response to detecting the thermal gradient in the thermal image and in response to detecting absence of a height gradient in the region of the depth map:

extracting a profile of the thermal gradient from the thermal image;

passing the profile of the thermal gradient into a fluid spill classifier to calculate a confidence for presence of the fluid within the area of the floor; and

predicting presence of the fluid within the area of the floor in response to the confidence for presence exceeding a threshold value.

17. A method for detecting and responding to hazards within a store comprising:

at a robotic system, during a scan cycle;

autonomously navigating toward an area of a floor of the store;

recording a thermal image of the area of the floor;

recording a depth map of the area of the floor;

recording a color image of the area of the floor;

detecting a thermal gradient in the thermal image;

scanning a region of the depth map, corresponding to the thermal gradient detected in the thermal image, for a height gradient; and

scanning a region of the color image, corresponding to the thermal gradient in the thermal image, for a color gradient;

in response to detecting the thermal gradient in the thermal image and in response to detecting absence of the height gradient in the region of the depth map, predicting presence of a fluid within the area of the floor;

estimating an opacity of the fluid based on the color gradient;

calculating a priority for removal of the fluid from the area of the floor of the store inversely proportional to the opacity of the fluid;

inserting a prompt to remove the fluid from the area of the floor into an electronic notification;

inserting the priority for removal of the fluid from the area of the floor into the electronic notification; and

transmitting the electronic notification to the computing device affiliated with an associate of the store.

18. A method for detecting and responding to hazards within a store comprising:

accessing a history of locations of fluid spill events in the store;

based on the history;

based on a first quantity of historical fluid spill events occurring in a first aisle in a set of aisles in the store over a period of time, scheduling a robotic system to traverse the first aisle at a first frequency during a scan cycle; and

based on a second quantity of historical fluid spill events, less than the first quantity, occurring in a second aisle in the set of aisles in the store over the period of time, scheduling the robotic system to traverse the second aisle at a second frequency less than the first frequency during the scan cycle;

at the robotic system, during the scan cycle;

autonomously navigating toward an area of a floor of the store;

autonomously navigating along the second aisle at a second frequency, less than the first frequency, during the scan cycle,

recording a thermal image of an area of the floor in the store;

recording a depth map of the area of the floor;

detecting a thermal gradient in the thermal image;

scanning a region of the depth map, corresponding to the thermal gradient detected in the thermal image, for a height gradient; and

in response to detecting the thermal gradient in the thermal image and in response to detecting absence of the height gradient in the region of the depth map;

predicting presence of a fluid within the area of the floor; and

serving a prompt to remove the fluid from the area of the floor of the store to a computing device affiliated with the store.

19. The method of claim 18:

further comprising, during the scan cycle;

recording a color image of the area of the floor;

scanning a region of the color image, corresponding to the thermal gradient in the thermal image, for a color gradient;

further comprising identifying the fluid as clear in response to detecting absence of the color gradient in the region of the depth map; and

wherein serving the prompt to the computing device comprises serving the prompt, to the computing device, specifying removal of the fluid from the area of the floor of the store and identifying the fluid as a clear fluid.