Moving and searching method of mobile robot for following human

• US 10,534,366 B2
• Filed: 05/08/2017
• Issued: 01/14/2020
• Est. Priority Date: 12/23/2016
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Choi, Jongsuk, Yun, Sang-Seok, Do, Hoang Minh
• Primary Examiner(s): Jen, Ian
• Application Number:
  US15/589,023
• Publication Number:
  US 20180181137A1
• Time in Patent Office: 981 Days
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A human-following robot for searching a following target when failing to track the following target includes a location estimating module configured to estimate a location of the following target based on map information and trajectory information, a search range setting module configured to set a search range of the following target based on the estimated location information and human walking pattern information, and tracking module configured to track the following target after moving to the search range. Since the robot estimates a location of a missed following target and the moves to the estimated location by utilizing map information and human walking pattern information including walking pattern data of persons in a surrounding environment, which is accumulated during a predetermined period, the following target may be detected again.

Claims

1. A robot for following a target, the robot comprising:

a processor configured to;

estimate a location of the target based on map information and a trajectory of the target when the target is lost;

set a search range for the target based on the estimated location and human walking pattern information; and

track the target after the robot moves to the search range, wherein the human walking pattern information is based on observed walking paths of people in a same environment in which the target is lost, wherein the human walking pattern information includes data about paths along which people of a specific group move at a specific time band, and wherein the search range is set based on the path data of the people in the specific group to which the target belongs, and wherein the processor is further configured to change a weight of color related to the target.

2. The robot according to claim 1, wherein the map information includes any one or any combination of any two or more of building interior structure information, furniture arrangement information, and surrounding obstacle location information.

3. The robot for according to claim 1, wherein the trajectory information is based on any one or any combination of any two or more of a distance, a direction, and a speed with which the target moves before the robot fails to track the target.

4. The robot according to claim 1, wherein the human walking pattern information is accumulated during a period of time.

5. The robot according to claim 1, wherein the search range is limited by the map information, and the map information includes any one or any combination of any two or more of a width of a passage, a number of passages, and a location of an obstacle.

6. The robot according to claim 1,

wherein the processor is further configured to calculate a moving path of the robot based on the map information.

7. The robot according to claim 1, wherein the processor is further configured to:

receive a depth image and a color image from a camera, and

detect the target by comparing the depth image and the color image with previously input target-related information,

wherein the target-related information includes any one or any combination of any two or more of a face shape of the target, color information of clothes worn by the target, and height information of the target.

20. The robot according to claim 1, wherein weight of color related to the target, is changed when tracking the target is changed based on a period of time that has passed after the target is lost.

8. A method for a robot to following a target, the method comprising:

estimating a location of the target based on map information and a trajectory of the target when the target is lost;

setting a search range for the target based on the estimated location and human walking pattern information; and

tracking the target after the robot moves to the search range, and change a weight of color related to the target;

wherein the human walking pattern information is based on observed walking paths of people in a same environment in which the target is lost wherein the human walking pattern information includes data about paths along which people of a specific group move at a specific time band, and wherein the search range is set based on the path data of the people in the specific group to which the target belongs.

9. The method according to claim 8, wherein the map information includes any one or any combination of any two or more of building interior structure information, furniture arrangement information, and surrounding obstacle location information.

10. The method according to claim 8, wherein the trajectory information is based on any one or any combination of any two or more of a distance, a direction, and a speed with which the target moves before the robot fails to track the target.

11. The method according to claim 8, wherein the human walking pattern information is accumulated during a predetermined period.

12. The method according to claim 8, wherein the search range is limited by the map information, and the map information includes any one or any combination of any two or more of a width of a passage, a number of passages, and a location of an obstacle.

13. The method according to claim 8, wherein the moving to the estimated location includes calculating a moving path of the robot based on the map information.

14. The method according to claim 8, wherein

the tracking of the target includes receiving a depth image and a color image from a camera, and detecting the target by comparing the depth image and the color image with previously input target-related information, and

the target-related information includes any one or any combination of any two or more of a face shape of the target, color information of clothes worn by the target, and height information of the target.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 8.

16. The method of claim 8, wherein the target is lost when the robot ceases to recognize the target through sensor data.

17. The method of claim 8, wherein the search range is positioned on an extrapolation of the trajectory.

18. The method of claim 17, where the search range is positioned at a distance along the extrapolation of the trajectory, and the trajectory is based on a moving speed of the target when the target is lost.

19. The method of claim 8, wherein the setting of the search range comprises:

in response to detecting multiple passageways, predicting respective paths of the target leading into the passageways,

selecting a passageway, among the passageways, that the target is determined to be most likely to enter based on the trajectory, and

designating a search range positioned on a path, among the predicted respective paths, leading into the selected passageway.