Visual positioning system that utilizes images of a working environment to determine position

• US 9,420,275 B2
• Filed: 11/01/2012
• Issued: 08/16/2016
• Est. Priority Date: 11/01/2012
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: Morin, Kristian, Kennedy, Sandy
• Primary Examiner(s): Prince, Jessica M
• Application Number:
  US13/666,358
• Publication Number:
  US 20140118536A1
• Time in Patent Office: 1,384 Days
• Field of Search: 348/143
• US Class Current: 1/1
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: H04N 1/32117 in a separate transmission or protocol signal prior to or subsequent to the image data transmission, e.g. in digital identification signal [DIS], in non standard setup [NSS] or in non standard field [NSF] for mode signalling H04N1/333
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3254 Orientation, e.g. landscape or portrait; Location or order of the image data, e.g. in memory
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A system determines an otherwise unknown position and orientation of a camera in a working environment, relative to an associated coordinate system, based on visually identifiable unique objects in images taken by the camera. The system utilizes a database that includes or may be updated to include position coordinates for unique objects of interest. The system identifies a plurality of objects within images taken by the camera at a given location, and enters the database either to determine position coordinates for the respective identified objects or to add position coordinates to the data base for the respective identified objects, or both. The system may also update the database to include newly identified unique objects and objects that are altered between images, to determine position and orientation of the camera in a changing environment. Sensors may be included to add additional information relative to the position and orientation of the camera.

Claims

1. A positioning system including

a database that associates, with visually identifiable unique objects in a working environment, position coordinates within an associated coordinate system;

one or more cameras of known internal characteristics and unknown positions, movements, or both within the environment, the one or more cameras taking one or more images of the environment from respective positions;

a processing subsystem for

identifying unique objects within one or more of the images taken by a given camera;

determining if at least three unique objects are identified within the one or more images taken by the given camera;

based on determining that the at least the three unique objects are identified within the one or more images taken by the given camera;

(1) entering the database to extract position coordinates associated with the at least three unique objects that match entries in the database, and

(2) determining position and orientation of the given camera in the environment based on the retrieved position coordinates for the at least three unique objects;

based on determining that the at least three unique objects are not identified within the one or more images taken by the given camera, processing one or more additional images to identify the at least three unique objects; and

updating the database to include one or both of information relating to newly identified unique objects in the one or more images taken by the respective cameras, the information including the position coordinates for the newly identified objects as determined from the images taken by the cameras and more precise position coordinates for previously identified objects.

2. The positioning system of claim 1 wherein the database further includes orientation coordinates for the identifiable unique objects that are composites of other unique objects, and the orientation coordinates are updated when the associated position coordinates are updated.

3. The positioning system of claim 2 wherein the database is empty when the one or more cameras take a first image and the processing subsystem populates the database to include the position coordinates, the orientation coordinates, or both, relating to identified unique objects in the first image and next images taken by the one or more cameras.

4. The positioning system of claim 3 wherein

the database further associates with the identified unique objects attributes of the identified objects that are determined from the images.

5. The positioning system of claim 1 wherein the position coordinates of the identified unique objects are calculated by 3-dimensional triangulation based on the images taken by the one or more cameras.

6. The positioning system of claim 1 wherein the processing subsystem processes altered unique objects as the newly identified unique objects.

7. The positioning system of claim 1 wherein the system utilizes information from maps, floor plans and combinations thereof and associates with the database entry of a unique object a description of where the unique object is visible from within the map, floor plans and combinations thereof.

8. The positioning system of claim 7, wherein the description of where the unique object is on a wall consists of a mathematical description of a plane and a side of the plane from which the object is visible.

9. The positioning system of claim 8, wherein the camera moves in directions of traffic flow in the environment to take the images.

10. The positioning system of claim 6 wherein the system includes in the database as observations known reference information about the environment.

11. The positioning system of claim 10 wherein the reference information is one or more of elevations of respective floors, elevations of defined areas within the environment, pointing directions of views from windows, pointing directions based on visibility of features located on walls, relative locations of walls, windows and other features of a floor plan.

12. The positioning system of claim 1 wherein a position and orientation of the camera for a first image are set to arbitrary values and positions and orientations of the unique objects are determined relative to the arbitrary values.

13. The positioning system of claim 12 further including

an uncalibrated inertial measurement unit (IMU), and

the one or more processors determine a gravity vector based on readings of the IMU and provide an orientation reference to the working environment.

14. The positioning system of claim 1 wherein the one or more processors identify within the images unique strategically placed objects that are associated in the database with known observations, the one or more processors use the known observations in calculations to determine position and orientation of the camera.

15. The positioning system of claim 14 wherein the unique strategically placed objects correspond to one or more of different elevations and different pointing directions.

16. The positioning system of claim 1 wherein the one or more processors determine the unknown movement of the camera to a next position and orientation based on the images taken before and after the movement.

17. The positioning system of claim 16 further including

an inertial measurement subsystem associated with a given camera,

the one or more processors calibrate and update the inertial measurement subsystem based on the determination of position and orientation of the given camera using the images, and

the one or more processors use information from the inertial measurement system to determine camera position and orientation in between times the one or more processors process the images to determine camera position and orientation.

18. The positioning system of claim 2 wherein a given camera is associated with a Global Navigation Satellite System (GNSS) receiver and GNSS position and orientation information is used by the processing subsystem to determine position and orientation information for identified unique objects, update position and orientation information for the identified unique objects, or both.

19. The positioning system of claim 18 wherein

the given camera is further associated with an inertial measurement system, and

the processing subsystem initializes and updates the inertial measurement subsystem using the position and orientation of the given camera as determined by the system and uses information from the inertial measurement subsystem to determine position and orientation information for the given camera based on sensed movement.