Tracking poses of 3D camera using points and planes

• US 9,420,265 B2
• Filed: 06/19/2013
• Issued: 08/16/2016
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Garaas, Tyler W, Taguchi, Yuichi, Ataer-Cansizoglu, Esra, Ramalingam, Srikumar
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US13/921,296
• Publication Number:
  US 20140002597A1
• Time in Patent Office: 1,154 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A method registers data using a set of primitives including points and planes. First, the method selects a first set of primitives from the data in a first coordinate system, wherein the first set of primitives includes at least three primitives and at least one plane. A transformation is predicted from the first coordinate system to a second coordinate system. The first set of primitives is transformed to the second coordinate system using the transformation. A second set of primitives is determined according to the first set of primitives transformed to the second coordinate system. Then, the second coordinate system is registered with the first coordinate system using the first set of primitives in the first coordinate system and the second set of primitives in the second coordinate system. The registration can he used to track a pose of a camera acquiring the data.

Claims

1. A method for tracking a pose of a camera by registering data using a set of primitives, wherein the data have three dimensions (3D) and the primitives include points and planes, comprising steps of:

acquiring the data as a sequence of frames by the camera, wherein the camera is a red, green, blue and depth (RGB-D) camera and the camera is movable, wherein for each acquired frame the method comprises;

predicting a current pose of the camera acquiring a current frame using a previous pose of the camera and a model of a motion of the camera;

determining a transformation between a first coordinate system and a second coordinate system of the camera in the current pose;

transforming landmarks in the first coordinate system into the second coordinate system using the transformation, wherein the landmarks include

at least three primitives and at least one plane;

determining landmarks in the current frame in the second coordinate system;

determining a correspondence between the landmarks determined in the second coordinate system and the landmarks transformed from the first coordinate system; and

registering the second coordinate system with the first coordinate system by updating the transformation according to the determined correspondence; and

tracking the pose of the camera according to the registering, wherein the steps are performed in a processor.

2. The method of claim 1, wherein the first set of primitives includes at least one point and at least one plane in the first coordinate system, and the second set of primitives includes at least one point and at least one plane in the second coordinate system.

3. The method of claim 1, wherein the data include textures and depths.

4. The method of claim 1, wherein the registering uses RANdom SAmple Consensus (RANSAC).

5. The method of claim 1, further comprising:

selecting a set of frames as keyframes from the sequence of frames; and

storing the keyframes in a map, wherein the keyframes include the points and the planes, and the points and the planes are stored in the map as landmarks.

6. The method of claim 5, further comprising:

predicting a pose of the camera for each frame; and

determining the pose of the camera for each frame according to the registering to track the camera.

7. The method of claim 6, wherein the pose of a k th frame is

8. The method of claim 6, wherein the predicting uses a constant velocity assumption.

9. The method of claim 5, further comprising:

applying bundle adjustment using the points and the planes to refine the landmarks in the map.

10. The method of claim 1, wherein the registering is in real time.

11. The method of claim 1, wherein an optical flow procedure is used to locate the points in the frames.

12. The method of claim 1, wherein correspondences of the planes are prioritizes over correspondences of the points.

13. The method of claim 1, wherein the registering is used for simultaneous localization and mapping (SLAM).

14. A system for tracking a pose of a camera by registering data using a set of primitives, wherein the data have three dimensions (3D) and the primitives include points and planes, comprising:

a camera for acquiring the data as a sequence of frames, wherein the camera is a red, green, blue and depth (RGB-D) camera; and

a processor operatively connected to the camera for tracking the pose of the camera, wherein the processor, for each acquired frame,

predicts a current pose of the camera acquiring a current frame using a previous pose of the camera and a model of a motion of the camera;

determines a transformation between a first coordinate system and a second coordinate system of the camera in the current pose;

transforms landmarks in the first coordinate system into the second coordinate system using the transformation, wherein the landmarks include at least three primitives and at least one plane;

determines landmarks in the current frame in the second coordinate system;

determining a correspondence between the landmarks determined in the second coordinate system and the landmarks transformed from the first coordinate system; and

registering the second coordinate system with the first coordinate system by updating the transformation according to the determined correspondence.