Vehicle localization in open-pit mining using GPS and monocular camera

• US 9,224,050 B2
• Filed: 03/16/2011
• Issued: 12/29/2015
• Est. Priority Date: 03/16/2010
• Status: Expired due to Fees

• Current Assignee: University of Sydney
• Original Assignee: University of Sydney
• Inventors: Elinas, Pantelis, Nettleton, Eric, Durrant-Whyte, Hugh, Perera, Linthotage Dushantha Lochana
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US13/049,822
• Publication Number:
  US 20110299730A1
• Time in Patent Office: 1,749 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

Described herein is a method and system for vehicle localization in an open pit mining environment having intermittent or incomplete GPS coverage. The system comprises GPS receivers associated with the vehicles and providing GPS measurements when available, as well as one or more cameras 50,55 overlooking the mine region 10. The cameras 50,55 are at a known location and are used for generating a sequence of images in a field of view with predetermined calibration in a fixed coordinate system. The system further comprises a vehicle recognition processor 120 for analyzing individual images from the camera to identify and locate within an image a vehicle in the mine region, as well as a vehicle tracking processor 130 for analyzing a sequence of images from the camera to track the identified vehicle location in the sequence of images. A data fusion processor 160 is coupled to receive GPS measurements, when available, from the vehicle GPS receivers, to fuse the received GPS measurement and corresponding vehicle image location, and to output a vehicle localization output 125.

Claims

1. A method for vehicle localization in an open pit mining environment, comprising:

receiving a sequence of picture frames from a camera calibrated in a fixed coordinate system with a field of view encompassing a region of vehicle operation in the mining environment;

processing pixels of at least one picture frame of the sequence of picture frames to identify interest points using a vehicle recognition algorithm;

analysing the interest points to determine the presence of a vehicle in the camera field of view;

tracking an image position of the vehicle within the region of vehicle operation by analysing the sequence of picture frames using a vehicle tracking algorithm;

ascertaining at least one independent measure of the vehicle location within the region of vehicle operation relative to the fixed coordinate system; and

fusing the image position of the vehicle and the at least one independent vehicle location measurement to provide an output of vehicle localization over time.

2. The method according to claim 1 wherein the independent measure of vehicle location is available intermittently and if the independent measure is unavailable the method comprises:

providing the output of vehicle localization over time dependent on the image position of the vehicle.

3. The method according to claim 1 wherein the independent measure of vehicle location is a measurement from a GPS receiver.

4. The method according to claim 1 wherein the sequence of picture frames is received from a plurality of cameras calibrated in the fixed coordinate system with respective fields of view encompassing the region of vehicle operation in the mining environment, the method further comprising:

determining respective image positions of the vehicle from picture frames of at least two cameras; and

providing the output of vehicle localization over time dependent on the respective image positions and, if available, the independent vehicle location measurement.

5. The method according to claim 1, comprising:

applying Bayesian filtering to provide the output of vehicle localization.

6. The method according to claim 5 wherein the Bayesian filtering uses a Kalman filter or a particle filter.

7. The method according to claim 1 comprising:

calibrating the camera field of view in the fixed coordinate system.

8. The method according to claim 1 wherein at least one camera is mounted on a camera support vehicle and the method comprises calibrating a field of view of the mounted camera in the fixed coordinate system.

9. A method for continuous vehicle localization in an open pit mining environment in which vehicles carrying GPS receivers operate in a region of intermittent or incomplete GPS coverage, comprising:

generating digital images with a field of view overlooking the mine region from a fixed location as vehicles move about the mine;

processing the digital images by analysing pixels of the image data and identifying interest points;

analysing the interest points to identify the presence and location of a vehicle in an image; and

fusing the identified image location of the vehicle with a measurement from the corresponding vehicle GPS receiver to provide continuous vehicle localization data.

10. A system for localization of vehicles in an open pit mining environment having intermittent or incomplete GPS coverage, comprising:

GPS receivers associated with the vehicles and providing GPS measurements when available;

a camera overlooking the mine region from a known location for generating a sequence of images in a field of view with predetermined calibration in a fixed coordinate system;

a vehicle recognition processor to analyse pixels of individual images from the camera, to identify interest points, and to analyse the interest points to identify and locate within an image a vehicle in the mine region;

a vehicle tracking processor to analyse a sequence of images from the camera to track the identified vehicle location in the sequence of images; and

a data fusion processor coupled to receive GPS measurements, when available, from the vehicle GPS receivers, to fuse the received GPS measurement and corresponding vehicle image location, and to output a vehicle localization output.

11. The system according to claim 10 comprising a plurality of cameras, wherein the data fusion processor fuses the received GPS measurement with corresponding vehicle image locations derived from images from the plurality of cameras.

12. The system according to claim 10 wherein the data fusion processor comprises a Bayesian filter selected from the group consisting of a Kalman filter and a particle filter.

13. The system according to claim 10 wherein at least one camera is mobile.