LOCALISING TRANSPORTABLE APPARATUS

• US 20140240501A1
• Filed: 09/27/2012
• Published: 08/28/2014
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Chancellor Masters & Scholars of The University of Cambridge (University of Cambridge)
• Original Assignee: Chancellor Masters & Scholars of The University of Cambridge (University of Cambridge)
• Inventors: Newman, Paul Michael, Stewart, Alexander Douglas
• Granted Patent: US 10,070,101 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Method and apparatus for localising transportable apparatus (100) within an environment (120) includes obtaining (202) point cloud data (114) representing a 3D point cloud with appearance information of at least part of the environment. The method further obtains (204) first frame data (110) representing an image produced by a sensor (102) onboard transportable apparatus at a first time and location within the environment and second frame data (110) representing an image produced by the sensor at a second time and location within the environment. The method harmonises (206) information about the first frame data, the second frame data and an overlapping subset of the point cloud data in order to determine a location within the point cloud data where at least one of the first frame and the second frame was produced, thereby localising (208) the transportable apparatus within the environment.

Claims

1. A method of localising transportable apparatus within an environment, the method including:

obtaining point cloud data representing a 3D point cloud with appearance information of at least part of the environment;

obtaining first frame data representing an image produced by a sensor on board transportable apparatus at a first time and location within the environment;

obtaining second frame data representing an image produced by the sensor at a second time and location within the environment; and

harmonising information about the first frame data, the second frame data and an overlapping subset of the point cloud data in order to determine a location within the point cloud data where at least one of the first frame and the second frame was produced, thereby localising the transportable apparatus within the environment.

2. A method according to claim 1, wherein harmonising the information is performed by finding a minimum of a cost function that maximises similarity of information about appearance of parts of the environment in the first frame data, the second frame data and the overlapping subset of the point cloud data.

3. A method according to claim 2, wherein the cost function comprises a Normalised Information Distance (NID) function.

4. A method according to claim 3, wherein harmonising the information includes modelling the appearance of points in the first frame data, the second frame data and/or the overlapping subset of the point cloud data as samples from random variables so that the NID cost function produces a measure of similarity between distributions of the variables.

5. A method according to claim 1, wherein the first frame data and the second frame data comprise consecutive images taken over a period of time.

6. A method according to claim 4, including providing a plurality of sensors on board the transportable apparatus and the method includes:

aligning image data produced by the plurality of sensors;

computing the NID cost function between consecutive frames for each said sensor independently; and

summing the NID cost functions over all the sensors to produce a total cost function.

7. A method according to claim 6, including blurring the images prior to the aligning of the image data.

8. A method according to claim 1, wherein the point cloud data is generated by a prior survey of the environment.

9. A method according to claim 1, wherein the sensor comprises a monocular camera.

10. A method according to claim 1, wherein the appearance information comprises colour values for points in the 3D point cloud.

21. A computer program product according to claim 1, wherein at least one of:

the point cloud data is generated by a prior survey of the environment;

the sensor comprises a monocular camera; and

the appearance information comprises colour values for points in the 3D point cloud.

11. (canceled)

12. Transportable apparatus including:

a device configured to obtain point cloud data representing a 3D point cloud with appearance information of at least part of an environment;

a sensor configured to obtain first frame data representing an image at a first time and location within the environment and second frame data representing an image produced by the sensor at a second time and location within the environment; and

a device configured to harmonise information about the first frame data, the second frame data and an overlapping subset of the point cloud data in order to determine a location within the point cloud data where at least one of the first frame and the second frame was produced, thereby localising the transportable apparatus within the environment.

13. A vehicle including transportable apparatus according to claim 12.

14. A non-transient computer program product comprising computer code that when executed by one or more processors causes a process for localising transportable apparatus within an environment, the process comprising:

obtaining point cloud data representing a 3D point cloud with appearance information of at least part of the environment;

obtaining first frame data representing an image produced by a sensor on board transportable apparatus at a first time and location within the environment;

obtaining second frame data representing an image produced by the sensor at a second time and location within the environment; and

harmonising information about the first frame data, the second frame data and an overlapping subset of the point cloud data in order to determine a location within the point cloud data where at least one of the first frame and the second frame was produced, thereby localising the transportable apparatus within the environment.

15. A computer program product according to claim 14, wherein harmonising the information is performed by finding a minimum of a cost function that maximises similarity of information about appearance of parts of the environment in the first frame data, the second frame data and the overlapping subset of the point cloud data.

16. A computer program product according to claim 15, wherein the cost function comprises a Normalised Information Distance (NID) function.

17. A computer program product according to claim 16, wherein harmonising the information includes modelling the appearance of points in the first frame data, the second frame data and/or the overlapping subset of the point cloud data as samples from random variables so that the NID cost function produces a measure of similarity between distributions of the variables.

18. A computer program product according to claim 17, where a plurality of sensors are provided on board the transportable apparatus and the process further includes:

aligning image data produced by the plurality of sensors;

computing the NID cost function between consecutive frames for each said sensor independently; and

summing the NID cost functions over all the sensors to produce a total cost function.

19. A computer program product according to claim 18, the process further including blurring the images prior to the aligning of the image data.

20. A computer program product according to claim 14, wherein the first frame data and the second frame data comprise consecutive images taken over a period of time.