Method and apparatus for estimating a pose

• US 9,159,134 B2
• Filed: 12/16/2011
• Issued: 10/13/2015
• Est. Priority Date: 12/16/2011
• Status: Active Grant

• Current Assignee: Gestigon GmbH (Valeo SE)
• Original Assignee: Universitat Zu Lubeck
• Inventors: Martinetz, Thomas, Barth, Erhardt, Ehlers, Kristian, Timm, Fabian, Klement, Sascha
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US14/365,222
• Publication Number:
  US 20140328519A1
• Time in Patent Office: 1,397 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20044 Skeletonization; Medial axis transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models

Abstract

The invention relates to a real time-capable analysis of a sequence of electronic images for estimating the pose of a movable object captured by means of the images. The invention further relates to implementing the invention in software and, in this connection, to a computer-readable medium that stores commands, the execution of which causes the method according to the invention to be carried out. The invention proceeds from a skeleton model, which is described by a small number of nodes in 3D space and permits a good data compression of the image information when the co-ordinates of the nodes describe at any time the position of predetermined parts of the moving object. The skeleton model simultaneously represents previous knowledge of the object, by defining e.g. node pairs and optionally also node triplets in the skeleton model that describe cohesive object parts or optionally object surfaces, which are contained in the measured 2½-D image information, i.e. are visible to the camera. The skeleton model is to be fitted quickly and accurately into the image information. The fitting is effected between two images of an image sequence by the continuous displacement of the nodes and the continuous updating of the skeleton model.

Claims

1. A method for pose estimation of a moving object by computer calculation of displacements of the 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds, the node coordinates being present in table form in an electronic memory and the 3D point clouds being determined from electronically recorded images of a depth sensor camera that represent the moving object,

the skeleton model being a topology that exhibits as topology elements N₁ nodes, N₂ edges, and N₃ triangles having N₁, N₂>

0 and N₃≧

0, and each topology element being described by nodes, node pairs, or node triplets and being firmly assigned to a part of the moving object, and that an arithmetic unit carries out the following steps;

a. randomly selecting a data point X of the 3D point cloud;

b. calculating the crossing point P relative to X with reference to each topology element and identifying whether P lies in each case inside the topology element;

c. calculating the distance from X to each topology element as the norm of the differential vector X-P;

d. determining that topology element that exhibits the minimum distance from X among all topology elements whose crossing point P lies inside the topology element;

e. displacing the topology element determined in step d by displacing all nodes establishing the topology element in the direction of the vector X-P, the displacement vector for a node being multiplied by a learning rate and by a weight that results from the crossing point P relative to X with reference to the topology element determined in step d, and

f. repeating the steps a to e for a predetermined number of learning steps while gradually reducing the learning rate;

g. updating the node coordinates in the table of the electronic memory after K passes of the predetermined number of learning steps with K≧

1;

h. providing at least the node coordinates updated in the table for further processing.

2. The method according to claim 1, wherein in step b the crossing point P with reference to a topology element is represented as a linear combination of the node coordinate vectors establishing the topology element and it is determined from the representation coefficients whether P lies inside the topology element.

3. The method according to claim 1, wherein the weight in step e is calculated from the representation coefficients of P.

4. The method according to claim 1, the number of repetitions of the steps a to e being between 1,000 and 5,000, in particular between 2,000 and 3,000.

5. The method according to claim 1, to the learning rate being between the starting value 0.5 and the end value 0.01.

6. The method according to claim 1, further comprising generating control commands for an apparatus to be controlled on the basis of information from the table of the updated node coordinates and of controlling the apparatus to be controlled by means of the control commands.

7. An apparatus for pose estimation of a moving object, comprising:

a depth sensor camera configured to detect electronic images of the moving object,

an electronic memory configured to store the electronic images of the depth sensor camera, and

an arithmetic unit that is designed to determine a 3D point cloud representing the object from the electronic images temporally in step with the image recording by the camera, and

wherein the memory further stores a list of 3D coordinates for the nodes of a skeleton model and the arithmetic unit being able to read out and change these coordinates for individual nodes, predetermined node pairs, and predetermined node triplets as representation of the topology elements of the skeleton model,

wherein the arithmetic unit is further designed, after determining the 3D point cloud representing the object, to carry out the following steps;

a. randomly selecting a data point X of the 3D point cloud;

b. calculating the crossing point P relative to X with reference to each topology element and identifying whether P lies in each case inside the topology element;

c. calculating the distance from X to each topology element as the norm of the differential vector X-P;

d. determining that topology element that exhibits the minimum distance from X among all topology elements where the crossing point P lies within the topology element;

e. displacing the topology element determined in step d by displacing all nodes establishing the topology element in the direction of the vector X-P, the displacement vector for a node being multiplied by a learning rate and by a weight that results from the crossing point P relative to X with reference to the topology element determined in step d, and

f. repeating the steps a to e for a predetermined number of learning steps while gradually reducing the learning rate;

g. updating the node coordinates in the table of the electronic memory after K passes of the predetermined number of learning steps with K≧

1;

h. providing at least the node coordinates updated in the table for further processing.

8. A non-transitory computer-readable storage medium that stores commands that, once executed by an arithmetic unit, cause the arithmetic unit to carry out 6 pose estimation of a moving object by computer calculation of displacements of the 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds, the node coordinates being present in table form in an electronic memory and the 3D point clouds being determined from electronically recorded images of a depth sensor camera that represent the moving object,

the skeleton model being a topology that exhibits as topology elements N₁ nodes, N₂ edges, and N₃ triangles having N₁, N₂>

0 and N₃≧

0, and each topology element being described by nodes, node pairs, or node triplets and being firmly assigned to a part of the moving object,

wherein the execution of said instructions cause the arithmetic unit to perform the following;

a. randomly selecting a data point X of the 3D point cloud;

b. calculating the crossing point P relative to X with reference to each topology element and identifying whether P lies in each case inside the topology element;

c. calculating the distance from X to each topology element as the norm of the differential vector X-P;

d. determining that topology element that exhibits the minimum distance from X among all topology elements whose crossing point P lies inside the topology element;

e. displacing the topology element determined in step d by displacing all nodes establishing the topology element in the direction of the vector X-P, the displacement vector for a node being multiplied by a learning rate and by a weight that results from the crossing point P relative to X with reference to the topology element determined in step d, and

f. repeating the steps a to e for a predetermined number of learning steps while gradually reducing the learning rate;

g. updating the node coordinates in the table of the electronic memory after K passes of the predetermined number of learning steps with K≧

1;

h. providing at least the node coordinates updated in the table for further processing.