Map building and positioning of robot

• US 10,611,028 B1
• Filed: 04/26/2019
• Issued: 04/07/2020
• Est. Priority Date: 11/30/2018
• Status: Active Grant

• Current Assignee: NextVPU (Shanghai) Co., Ltd.
• Original Assignee: NextVPU (Shanghai) Co., Ltd.
• Inventors: Zhou, Ji, Feng, Xinpeng
• Primary Examiner(s): Holwerda, Stephen
• Application Number:
  US16/396,616
• Time in Patent Office: 347 Days
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

A robot, a map building method and a storage medium are provided. The map building method includes building an initial map according to a scene image captured by a robot in real time when the robot traverses a work area. The initial map includes a first map including a mapping of the work area and a map coordinate system, and a second map including a scene feature extracted based on the scene image, a geometric quantity of the scene feature and the scene image, which are stored in association with the pose of the robot in a scene database when capturing the scene image, the pose including a coordinate and an orientation of the robot, and the scene feature including a feature of a feature object in the scene and/or a feature of the scene image.

Claims

1. A computer-implemented method, comprising:

building an initial map according to a scene image captured by a robot at a pose in a work area in response to the robot traversing the work area according to a predetermined rule, wherein the initial map comprises a first map including a mapping of the work area and a map coordinate system, and a second map including the captured scene image, a scene feature extracted from the captured scene image, a geometric quantity of the extracted scene feature and the pose of the robot, and wherein the captured scene image, the scene feature and the geometric quantity are stored in association with the pose of the robot in a scene database, the pose including a coordinate and an orientation of the robot in the work area, and the extracted scene feature including a feature of a feature object in the scene and/or a feature of the scene image;

determining an estimated pose of the robot from the scene database according to an image captured by the robot in real time;

determining an accurate pose of the robot according to the image captured by the robot in real time and the estimated pose;

determining an updated scene image, an updated scene feature and a geometric quantity corresponding to the updated scene feature according to the accurate pose, the scene feature matching the estimated pose, the geometric quantity corresponding to the scene feature and the image captured by the robot in real time; and

updating the scene database with the accurate pose, the updated scene image, the updated scene feature, and the geometric quantity corresponding to the updated scene feature.

2. The computer-implemented method according to claim 1, wherein building the initial map according to the scene image captured by the robot comprises:

building a contour map in response to the robot traveling along a boundary of the work area;

building an internal map in response to the robot traveling along a traversal path inside the work area; and

obtaining the initial map from the contour map and the internal map.

3. The computer-implemented method according to claim 2, wherein the robot traveling along the boundary of the work area comprises:

by taking a direction in which the robot captures scene images as a positive direction, the robot traveling along the positive direction;

the robot changing from traveling along the positive direction to traveling along the boundary of the obstacle object when identifying an obstacle object.

4. The compute-implemented method according to claim 3,

wherein a current position of the robot is determined as a first position when the robot identifies the obstacle object, and

wherein the robot travelling along the boundary of the obstacle object further comprises:

when the robot travels to the first position again, the robot determines whether the obstacle object is an obstacle within the work area or the boundary of the work area;

in case the obstacle object is an obstacle in the work area, the robot changes from traveling along the boundary of the obstacle object to traveling in the positive direction;

in case the obstacle object is the boundary of the work area, the robot determines that the boundary of the work area is completely travelled.

5. The computer-implemented method according to claim 1, wherein when the robot traverses the work area according to the predetermined rule and identifies a feature object within the work area according to a captured image, a distance between the robot and the feature object is determined based on a map coordinate system of the first map, and when the distance is consistent with a predetermined distance, the current image captured by the robot in real time is taken as a scene image, and the scene feature is extracted based on the scene image.

6. The computer-implemented method according to claim 1, further comprising:

determining a current pose of the robot based on the initial map or an updated map, and based on the image captured by the robot in real time.

7. The computer-implemented method according to claim 6, wherein determining a current pose of the robot comprises:

matching a scene feature of the image captured in real time by the robot with the scene feature stored in the scene database, and

determining a difference between a real-time pose of the robot and the pose of the robot when capturing the scene image according to the comparison of the matched scene feature and the scene feature of the image captured by the robot in real time, so as to determine the current pose of the robot based on the pose of the robot when capturing the scene image.

8. The computer-implemented method according to claim 1, wherein determining the estimated pose of the robot from the scene database according to the image captured by the robot in real time comprises:

extracting a geometric feature set from the image captured by the robot in real time;

searching for image indexes in a scene sub-database according to the extracted geometric feature set; and

determining the estimated pose of the robot through an interpolation function, according to the extracted geometric feature set, a geometric feature set corresponding to the image indexes in a geometric information sub-database and poses of the robot corresponding to the image indexes.

9. A robot, comprising:

a motor configured to drive the robot to move;

a processor configured to;

build an initial map according to a scene image captured by the robot at a pose in a work area in response to the robot traversing the work area according to a predetermined rule, wherein the initial map comprises a first map including a mapping of the work area and a map coordinate system, and a second map including the captured scene image, a scene feature extracted from the captured scene image, a geometric quantity of the extracted scene feature and the pose of the robot, and wherein the captured scene image, the scene feature and the geometric quantity are stored in association with the pose of the robot in a scene database, the pose including a coordinate and an orientation of the robot in the work area, and the extracted scene feature including a feature of a feature object in the scene and/or a feature of the scene image;

determine an estimated pose of the robot from the scene database according to an image captured by the robot in real time;

determine an accurate pose of the robot according to the image captured by the robot in real time and the estimated pose;

determine an updated scene image, an updated scene feature and a geometric quantity corresponding to the updated scene feature according to the accurate pose, the scene feature matching the estimated pose, the geometric quantity corresponding to the scene feature and the image captured by the robot in a real time; and

update the scene database with the accurate pose, the updated scene image, the updated scene feature, and the geometric quantity corresponding the updated scene feature.

10. The robot according to claim 9, wherein the processor is further configured to determine a current pose of the robot based on the initial map or an updated map, and based on the image captured by the robot in real time.

11. The robot according to claim 10, wherein determining a current pose of the robot comprises:

matching a scene feature of the image captured in real time by the robot with the scene feature stored in the scene database, and

determining a difference between a real-time pose of the robot and the pose of the robot when capturing the scene image according to the comparison of the matched scene feature and the scene feature of the image captured by the robot in real time, so as to determine the current pose of the robot based on the pose of the robot when capturing the scene image.

12. The robot according to claim 9, wherein building the initial map according to the scene image captured by the robot comprises:

building a contour map in response to the robot traveling along a boundary of the work area;

building an internal map in response to robot traveling along a traversal path inside the work area; and

obtain initial map from the contour map and the internal map.

13. The robot according to claim 12, wherein the robot traveling along the boundary of the work area comprises:

by taking a direction in which the robot captures scene images as a positive direction, the robot traveling along the positive direction;

the robot changing from traveling along the positive direction to traveling along the boundary of the obstacle object when identifying an obstacle object.

14. The robot according to claim 13,

wherein a current position of the robot is determined as a first position when the robot identifies the obstacle object, and

wherein the robot travelling along the boundary of the obstacle object further comprises:

when the robot travels to the first position again, the robot determines whether the obstacle object is an obstacle within the work area or the boundary of the work area;

in case the obstacle object is an obstacle in the work area, the robot changes from traveling along the boundary of the obstacle object to traveling in the positive direction;

in case the obstacle object is the boundary of the work area, the robot determines that the boundary of the work area is completely travelled.

15. The robot according to claim 9, wherein when the robot traverses the work area according to the predetermined rule and identifies a feature object within the work area according to a captured image, a distance between the robot and the feature object is determined based on a map coordinate system of the first map, and when the distance is consistent with a predetermined distance, the current image captured by the robot in real time is taken as a scene image, and the scene feature is extracted based on the scene image.

16. The robot according to claim 9, wherein determining the estimated pose of the robot from the scene database according to the image captured by the robot in real time comprises:

extracting a geometric feature set from the image captured by the robot in real time;

searching for image indexes in a scene sub-database according to the extracted geometric feature set; and

determining the estimated pose of the robot through an interpolation function, according to the extracted geometric feature set, a geometric feature set corresponding to the image indexes in a geometric information sub-database and poses of the robot corresponding to the image indexes.

17. A non-transitory computer readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, performs the following steps:

building an initial map according to a scene image captured by a robot at a pose in a work area in response to the robot traversing a work area according to a predetermined rule, wherein the initial map comprises a first map including a mapping of the work area and a map coordinate system, and a second map including the captured scene image, a scene feature extracted from the captured scene image, a geometric quantity of the extracted scene feature and the pose of the robot, and wherein the captured scene image, the scene feature and the geometric quantity are a stored in association with the pose of the robot in a scene database, the pose including a coordinate and an orientation of the robot in the work area, and the extracted scene feature including a feature of a feature object in the scene and/or a feature of the scene image;

determining an estimated pose of the robot from the scene database according to an image captured by the robot in real time;

determining an accurate pose of the robot according to the image captured by the robot in real time and the estimated pose;

determining an updated scene image, an updated scene feature and a geometric quantity corresponding to the updated scene feature according to the accurate pose, the scene feature matching the estimated pose, the geometric quantity corresponding to the scene feature and the image captured by the robot in real time; and

updating the scene database with the accurate pose, the updated scene image, the updated scene feature, and the geometric quantity corresponding to the updated scene feature.

18. The storage medium according to claim 17, wherein when the robot traverses the work area according to the predetermined rule and identifies a feature object within the work area according to a captured image, a distance between the robot and the feature object is determined based on a map coordinate system of the first map, and when the distance is consistent with a predetermined distance, the current image captured by the robot in real time is taken as a scene image, and the scene feature is extracted based on the scene image.

19. The storage medium according to claim 17, wherein determining the estimated pose of the robot from the scene database according to the image captured by the robot in real time comprises:

extracting a geometric feature set from the image captured by the robot in real time;

searching for image indexes in a scene sub-database according to the extracted geometric feature set; and

determining the estimated pose of the robot through an interpolation function, according to the extracted geometric feature set, a geometric feature set corresponding to the image indexes in a geometric information sub-database and poses of the robot corresponding to the image indexes.