Vehicle localization

• US 10,255,680 B2
• Filed: 05/29/2015
• Issued: 04/09/2019
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Oxford University Innovation Limited (University of Oxford)
• Original Assignee: Oxford University Innovation Limited (University of Oxford)
• Inventors: Newman, Paul, Maddern, William, Stewart, Alexander Douglas, Churchill, Winston, McManus, Colin
• Primary Examiner(s): Bekele, Mekonen T
• Application Number:
  US15/312,430
• Publication Number:
  US 20170076455A1
• Time in Patent Office: 1,411 Days
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method of localizing a transportable apparatus (102) within an environment around the apparatus is disclosed. The method comprises obtaining a sequence of images (200) of the environment and generating one or more sequences of transformed images (204) from the sequence of images wherein an image from the sequence of images has undergone a transformation to provide a transformed image within the sequence of transformed images. Processing circuitry (112) is then used to compare one or more images from the sequence of transformed images (204) and one or more images from at least one of the sequence of images (200) and a further sequence of transformed images against a representation of the environment. The comparison determines corresponding features within the images (200) and/or transformed images (204) and the representation. The transportable apparatus (102) is then localized according to a position of the one or more corresponding features.

Claims

1. A computerized method of localizing a transportable apparatus within an environment around the apparatus, comprising:

obtaining a sequence of images of the environment;

providing the sequence of images of the environment to a Visual Odometry (VO) system comprising processing circuitry comprising;

a first VO pipeline arranged for;

performing a first transformation on the sequence of images of the environment, to generate a first sequence of transformed images;

comparing one or more images from the first sequence of transformed images against a stored representation of the environment so as to determine first corresponding features within the images of the first sequence and the representation;

providing a first localization estimate for the transportable apparatus according to a position of the one or more first corresponding features; and

providing a first quality value for the first localization estimate; and

a second VO pipeline arranged for;

performing a second transformation on the sequence of images of the environment, to generate a second sequence of transformed images, wherein the second transformation is different from the first transformation;

comparing one or more images from the second sequence of transformed images against the stored representation of the environment so as to determine second corresponding features within the images of the second sequence and the representation;

providing a second localization estimate for the transportable apparatus according to a position of the one or more second corresponding features; and

providing a second quality value for the second localization estimate;

selecting either the first localization estimate or the second localization estimate based on a comparison of the first and the second quality values; and

localizing the transportable apparatus according to the selected first or second localization estimate.

2. A method according to claim 1 in which the representation of the environment is provided by one or more sequences of stored images.

3. A method according to claim 2 in which the sequence of stored images undergo transformations in order that at least one of the comparisons be performed.

4. A method according to claim 1 in which the representation of the environment is a 3D model of the environment.

5. A method according to claim 1 in which the sequence of images is obtained using any of the following:

an optical camera;

a stereoscopic optical camera; and

a thermal imaging camera.

6. A method according to claim 1 in which the images within the sequence of images are within an RGB color space.

7. A method according to claim 1 in which the transformation performed on an image transforms the image into one of the following:

an illumination invariant color space;

or greyscale.

10. A method according to claim 4 in which the 3D model of the environment comprises a 3D point cloud.

11. A method according to claim 1, in which the first transformation is a null transformation such that the first sequence of transformed images is the obtained sequence of images of the environment.

12. A method according to claim 1, in which the first and the second VO pipelines run in parallel.

13. A method according to claim 1, in which the first quality value or the second quality value is assessed:

based on a number of the first or second corresponding features matched, respectively, or

based on associated certainties of matches of the first or second corresponding features, respectively, that are determined to be correct.

8. An apparatus arranged to perform a localization of itself within an environment, the apparatus comprising:

a sensor configured to generate a sequence of images of an environment around the apparatus;

processing circuitry in communication with the sensor, the processing circuitry configured to execute a first Visual Odometry (VO) pipeline and a second VO pipeline;

the first VO pipeline arranged to;

perform a first transformation on the sequence of images of the environment, to generate a first sequence of transformed images;

compare, against a stored representation of the environment, one or more images from the first sequence of transformed images;

determine, during the comparison, first corresponding features within the images of the first sequence and the stored representation;

provide a first localization estimate for the apparatus according to a position of the one or more first corresponding features; and

provide a first quality value for the first localization estimate;

the second VO pipeline arranged to;

perform a second transformation on the sequence of images of the environment, to generate a second sequence of transformed images, wherein the second transformation is different from the first transformation;

compare, against the stored representation of the environment, one or more images from the second sequence of transformed images;

determine, during the comparison, second corresponding features within the images of the second sequence and the stored representation;

provide a second localization estimate for the apparatus according to a position of the one or more second corresponding features; and

provide a second quality value for the second localization estimate;

the processing circuitry further configured to;

select either the first localization estimate or the second localization estimate based on a comparison of the first and the second quality values; and

localize the apparatus according to the selected first or second localization estimate.

14. An apparatus according to claim 8, in which the first transformation is a null transformation such that the first sequence of transformed images is the obtained sequence of images of the environment.

15. An apparatus according to claim 8, in which the first and the second VO pipelines are executed in parallel.

16. An apparatus according to claim 8, in which the first quality value or the second quality value is assessed:

based on a number of the first or second corresponding features matched, respectively, or

based on associated certainties of matches of the first or second corresponding features, respectively, that are determined to be correct.

9. A non-transitory machine readable medium containing instructions, which when read by a computer, cause that computer to:

obtain a sequence of images of an environment around an apparatus;

execute a first Visual Odometry (VO) pipeline and a second VO pipeline;

the first VO pipeline arranged to;

perform a first transformation on the sequence of images of the environment, to generate a first sequence of transformed images;

compare one or more images from the first sequence of transformed images against a stored representation of the environment;

determine first corresponding features within the images of the first sequence and the stored representation;

provide a first localization estimate for the apparatus according to a position of the one or more first corresponding features; and

provide a first quality value for the first localization estimate;

the second VO pipeline arranged to;

perform a second transformation on the sequence of images of the environment, to generate a second sequence of transformed images, wherein the second transformation is different from the first transformation;

compare, against the stored representation of the environment, one or more images from the second sequence of transformed images;

determine second corresponding features within the images of the second sequence and the stored representation;

provide a second localization estimate for the apparatus according to a position of the one or more second corresponding features; and

provide a second quality value for the second localization estimate;

select either the first localization estimate or the second localization estimate based on a comparison of the first and the second quality values; and

localize the apparatus according to the selected first or second localization estimate.

17. A non-transitory machine readable medium according to claim 9, in which the first transformation is a null transformation such that the first sequence of transformed images is the obtained sequence of images of the environment.

18. A non-transitory machine readable medium according to claim 9, in which the first and the second VO pipelines are executed in parallel.

19. A non-transitory machine readable medium according to claim 9, in which the first quality value or the second quality value is assessed:

based on a number of the first or second corresponding features matched, respectively, or

based on associated certainties of matches of the first or second corresponding features, respectively, that are determined to be correct.