Localising transportable apparatus

• US 10,070,101 B2
• Filed: 09/27/2012
• Issued: 09/04/2018
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Chancellor Masters & Scholars of The University of Cambridge (University of Cambridge)
• Original Assignee: Chancellor Masters & Scholars of The University of Cambridge (University of Cambridge)
• Inventors: Newman, Paul Michael, Stewart, Alexander Douglas
• Primary Examiner(s): Sosanya, Obafemi
• Application Number:
  US14/348,451
• Publication Number:
  US 20140240501A1
• Time in Patent Office: 2,168 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Method and apparatus for localizing transportable apparatus (100) within an environment (120) includes obtaining (202) point cloud data (114) representing a 3D point cloud with appearance information of at least part of the environment. The method further obtains (204) first frame data (110) representing an image produced by a sensor (102) onboard transportable apparatus at a first time and location within the environment and second frame data (110) representing an image produced by the sensor at a second time and location within the environment. The method harmonizes (206) information about the first frame data, the second frame data and an overlapping subset of the point cloud data in order to determine a location within the point cloud data where at least one of the first frame and the second frame was produced, thereby localizing (208) the transportable apparatus within the environment.

Claims

1. A method of localising transportable apparatus within an environment, the method including:

obtaining point cloud data representing a 3D point cloud including a plurality of points in a reference coordinate system of at least part of the environment;

obtaining first frame data representing a first image produced by an imaging sensor on board transportable apparatus at a first location within the environment;

obtaining second frame data representing a second image produced by the imaging sensor at a second location within the environment; and

determining a first subset of points in the first frame data that overlap with at least some of the plurality of points of the 3D point cloud in the reference coordinate system;

determining a second subset of points in the second frame data that overlap with at least some of the plurality of points of the 3D point cloud in the reference coordinate system, the first frame data being different than the second frame data;

determining a location within the 3D point cloud where the overlapping first subset of points in the first frame data and the overlapping second subset of points in the second frame data were produced, thereby localising the transportable apparatus within the environment;

obtaining a velocity of the transportable apparatus at a specified time;

comparing the specified time and the velocity to a time that the first frame or the second frame was captured; and

when the specified time and the velocity match the time that the first frame or the second frame was captured, verifying the location of the transportable apparatus within the 3D point cloud.

2. The method according to claim 1, wherein determining the location within the 3D point cloud includes finding a minimum of a cost function that maximises similarity of information about appearance of parts of the environment in the first frame data, the second frame data and the overlapping first and second subsets of points in the first and second frame data.

3. The method according to claim 2, wherein the cost function comprises a Normalised Information Distance (NID) function.

4. The method according to claim 3, wherein determining the location within the 3D point cloud includes modelling the appearance of points in the first frame data, the second frame data and/or the overlapping first and second subsets of points in the first and second frame data as samples from random variables so that the NID cost function produces a measure of similarity between distributions of the variables.

5. The method according to claim 1, wherein the first frame data and the second frame data, respectively, comprise consecutive images taken over a period of time.

6. The method according to claim 4, including providing a plurality of imaging sensors on board the transportable apparatus and the method includes:

aligning image data produced by the plurality of imaging sensors;

computing the NID cost function between consecutive frames for each said imaging sensor independently; and

summing the NID cost functions over all the imaging sensors to produce a total cost function.

7. The method according to claim 6, including blurring the images prior to the aligning of the image data.

8. The method according to claim 1, wherein the 3D point cloud data is generated by a prior survey of the environment.

9. The method according to claim 1, wherein the imaging sensor comprises a monocular camera.

10. The method according to claim 1, wherein the appearance information comprises colour values for points in the 3D point cloud.

11. A transportable apparatus including:

a device configured to obtain point cloud data representing a 3D point cloud including a plurality of points in a reference coordinate system of at least part of an environment;

an imaging sensor configured to obtain first frame data representing a first image at a first location within the environment and second frame data representing a second image produced by the sensor at a second location within the environment; and

a device configured to determine a first subset of points in the first frame data that overlap with at least some of the plurality of points of the 3D point cloud in the reference coordinate system and determine a second subset of points in the second frame data that overlap with at least some of the plurality of points of the 3D point cloud in the reference coordinate system for determining a location within the 3D point cloud where the overlapping first subset of points in the first frame data and the overlapping second subset of points in the second frame data was produced, the first frame data being different than the second frame data, thereby localising the transportable apparatus within the environment, wherein the device is configured to obtain a velocity of the transportable apparatus at a specified time, compare the specified time and the velocity to a time that the first frame or the second frame was captured, and verify the location of the transportable apparatus within the 3D point cloud when the specified time and the velocity match the time that the first frame or the second frame was captured.

12. A vehicle including the transportable apparatus according to claim 11.

13. A non-transitory computer program product comprising instructions that when executed by one or more processors causes the one or more processors to perform a process for localising transportable apparatus within an environment, the process comprising:

obtaining point cloud data representing a 3D point cloud including a plurality of points in a reference coordinate system of at least part of the environment;

obtaining first frame data representing a first image produced by an imaging sensor on board transportable apparatus at a first location within the environment;

obtaining second frame data representing a second image produced by the imaging sensor at a second location within the environment;

determining a first subset of points in the first frame data that overlap with at least some of the plurality of points of the 3D point cloud in a reference coordinate system;

determining a second subset of points in the second frame data that overlap with at least some of the plurality of points of the 3D point cloud, the second frame data being different than the first frame data;

determining a location within the 3D point cloud where the overlapping first subset of points in the first frame data and the overlapping second subset of points in the second frame data was produced, thereby localising the transportable apparatus within the environment;

obtaining a velocity of the transportable apparatus at a specified time;

comparing the specified time and the velocity to a time that the first frame or the second frame was captured; and

when the specified time and the velocity match the time that the first frame or the second frame was captured, verifying the location of the transportable apparatus within the 3D point cloud.

14. The non-transitory computer program product according to claim 13, wherein determining the location within the 3D point cloud includes finding a minimum of a cost function that maximises similarity of information about appearance of parts of the environment in the first frame data, the second frame data and the overlapping first and second subsets of points in the first and second frame data.

15. The non-transitory computer program product according to claim 14, wherein the cost function comprises a Normalised Information Distance (NID) function.

16. The non-transitory computer program product according to claim 15, wherein determining the location within the 3D point cloud includes modelling the appearance of points in the first frame data, the second frame data and/or the overlapping first and second subsets of points in the first and second frame data as samples from random variables so that the NID cost function produces a measure of similarity between distributions of the variables.

17. The non-transitory computer program product according to claim 16, where a plurality of imaging sensors are provided on board the transportable apparatus and the process further includes:

aligning image data produced by the plurality of imaging sensors;

computing the NID cost function between consecutive frames for each said imaging sensor independently; and

summing the NID cost functions over all the imaging sensors to produce a total cost function.

18. The non-transitory computer program product according to claim 17, the process further including blurring the images prior to the aligning of the image data.

19. The non-transitory computer program product according to claim 13, wherein the first frame data and the second frame data, respectively, comprise consecutive images taken over a period of time.

20. The non-transitory computer program product according to claim 13, wherein at least one of:

the 3D point cloud data is generated by a prior survey of the environment;

the imaging sensor comprises a monocular camera; and

the appearance information comprises colour values for points in the 3D point cloud.