Method for the real-time-capable, computer-assisted analysis of an image sequence containing a variable pose

• US 9,058,661 B2
• Filed: 05/06/2010
• Issued: 06/16/2015
• Est. Priority Date: 05/11/2009
• Status: Active Grant

• Current Assignee: Gestigon GmbH (Valeo SE)
• Original Assignee: Universitat Zu Lubeck
• Inventors: Barth, Erhardt, Martinetz, Thomas, Haker, Martin
• Primary Examiner(s): Nguyen, Kimbinh T
• Assistant Examiner(s): Isanians, Raffi
• Application Number:
  US13/319,968
• Publication Number:
  US 20120120073A1
• Time in Patent Office: 1,867 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The invention relates to a method for the real-time-capable, computer-assisted analysis of an image sequence of an object consisting of elements that can be moved relative to each other and are interconnected, said sequence containing a variable pose, wherein the individual images of the image sequence are recorded by way of a time-of-flight (TOF) camera such that said images can be processed by a computer, and contain brightness and distance data as functions of the pixel coordinates of the camera for each image of the sequence, comprising the following steps: a. Capturing the pixels of an individual image forming the object, b. calculating a three-dimensional (3D) point cloud in a virtual space, said point cloud representing the surface of the object that is visible to the camera, by a computational projection of object-depicting pixels in such a space, while taking captured distance data to the object into consideration, c. fitting a model of the object consisting of nodes and edges into the computer-generated 3D point cloud for the individual images, wherein the nodes represent a selection of elements of the object and the edges represent the connections of said elements amount each other, d. iteratively updating all node positions by applying a learning rule for training a self-organizing map having a previously defined number of randomly selected dots of the point cloud, e. repeating steps a. to d. for each subsequent individual image of the sequence, wherein for the fitting in step c. the result of step e. of the preceding image is used in each case, and f. determining the varying pose from the positions of predetermined nodes of the model which have been captured in at least representative images of the image sequence.

Claims

1. A method for real-time-capable computer-assisted analysis of an image sequence containing a variable pose of an object composed of interconnected elements movable in relation to each other, wherein the frames of the image sequence have been recorded by a distance measuring camera and are processed by a computer and the frames have brightness and distance data as functions of pixel coordinates of the camera for each frame of the sequence;

the method comprising;

(a) the computer detecting the pixels of a frame which map the object;

(b) the computer calculating a three-dimensional (3D) aggregate of points within a virtual space, which represents that surface of the object which is visible for the camera, by calculated projection of object-mapping pixels into such a space, taking into account acquired data of a distance from the object;

(c) the computer fitting a model of the object, which consists of nodes and edges, into the computer-generated 3D aggregate of points for the frame, the nodes representing a selection of elements of the object and the edges representing interconnections of these elements;

(d) the computer iteratively updating all node positions by using a learning rule for training a self-organizing map with a predetermined number of randomly sampled points of the aggregate of points;

(e) repeating at least 25 times per second the steps (a) through (d) for each subsequent frame of the sequence, the result of step (d) of the preceding frame being used respectively for the fitting process in step (c);

(f) the computer determining the changing pose from positions of predetermined nodes of the model, which have been detected in at least representative frames of the image sequence;

(g) wherein the node positions of the model are updated in iteration steps, one point x being randomly sampled for the predetermined number of points of the 3D aggregate of points in each iteration step and all nodes being shifted towards x; and

(h) the degree of the shift being largest for the node which had the smallest distance from x prior to the iteration step wherein the number of the randomly sampled points x or that of the iteration steps is about 10% of the total number of the points within the aggregate of points.

2. The method according to claim 1, wherein the camera pixels onto which the object is mapped are determined by an image segmentation of the image sequence of the TOF camera.

3. The method according to claim 2, wherein two threshold values for brightness and depth data of the TOF camera are determined for each frame and a contiguous region of camera pixels is identified whose measurement data are larger than the brightness threshold value and are less than the depth threshold value.

4. The method according to claim 3, wherein the brightness data are plotted in a histogram and two accumulation points are identified and the brightness threshold value is selected in such a way that it is between the accumulation points.

5. The method according to claim 3, wherein the depth data are plotted in a histogram and at least two accumulation points are identified and the depth threshold value is selected in such a way that it is between the accumulation point of the least depth and the other ones.

6. The method according to claim 1, wherein the degree of the shift of all nodes is set in such a way that it decreases with the number of iteration steps.

7. The method according to claim 1, wherein prior to each iteration step, for the first node having the smallest distance from the sampled point x, a second node is selected which is a neighbor of the first node and whose distance from the first node does not exceed a predetermined value during the shift of the nodes.

8. The method according to claim 1, wherein the estimated pose serves as a user input for controlling a computer-assisted system.

9. The method according to claim 8, wherein the computer-assisted system represents a technical control system in the medical or industrial field.

10. The method according to claim 8, wherein the computer-assisted system represents an information terminal.

11. The method according to claim 8, wherein the computer-assisted system represents a multimedia system for home use.

12. The method according to claim 8, wherein the computer-assisted system represents a computer game.

13. A computer system for real-time-capable computer-assisted analysis of an image sequence containing a variable pose of an object composed of interconnected elements movable in relation to each other, wherein the frames of the image sequence have been recorded by a distance measuring camera so that they are processed by the computer system and the frames have brightness and

distance data as functions of pixel coordinates of the camera for each frame of the sequence, the computer system comprising:

a detecting unit adapted to perform the step of (a) detecting the pixels of a frame which map the object;

a calculating unit adapted to perform the step of (b) calculating a three-dimensional (3D) aggregate of points within a virtual space, which represents that surface of the object which is visible for the camera, by calculated projection of object-mapping pixels into such a space, taking into account acquired data of a distance from the object;

a fitting unit adapted to perform the step of (c) fitting a model of the object, which consists of nodes and edges, into the computer-generated 3D aggregate of points for the frame, the nodes representing a selection of elements of the object and the edges representing interconnections of these elements;

an updating unit adapted to perform the step of (d) iteratively updating all node positions by using a learning rule for training a self-organizing map with a predetermined number of randomly sampled points of the aggregate of points;

a determination unit adapted to perform the step of (f) determining the changing pose from positions of predetermined nodes of the model, which have been detected in at least representative frames of the image sequence,

wherein the computer system is adapted to cause its units to (e) repeat at least 25 times per second the steps (a) through (d) for each subsequent frame of the sequence, and to cause the fitting unit to use the result of the repetition of steps (a) through (d) of the preceding frame being used respectively for performing process in step (c);

wherein the node positions of the model are (q) updated in iteration steps, one point x being randomly sampled for the predetermined number of points of the 3D aggregate of points in each iteration step and all nodes being shifted towards x; and

the degree of the shift being largest for the node which (h) had the smallest distance from x prior to the iteration step wherein the number of the randomly sampled points x or that of the iteration steps is about 10% of the total number of the points within the aggregate of points.

14. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a computer system for real-time-capable computer-assisted analysis of an image sequence, cause the computer system to:

(a) detect the pixels of a frame which map an object, said object being comprised in said image sequence which contains a variable pose of the object composed of interconnected elements movable in relation to each other wherein the frames of the image sequence have been recorded by a distance measuring camera so that they are processed by a computer and the frames have brightness and distance data as functions of pixel coordinates of the camera for each frame of the sequence

(b) calculate a three-dimensional (3D) aggregate of points within a virtual space, which represents that surface of the object which is visible for the camera, by calculated projection of object-mapping pixels into such a space, taking into account acquired data of a distance from the object;

(c) fit a model of the object, which consists of nodes and edges, into the computer-generated 3D aggregate of points for the frame, the nodes representing a selection of elements of the object and the edges representing interconnections of these elements;

(d) iteratively update all node positions by using a learning rule for training a self-organizing map with a predetermined number of randomly sampled points of the aggregate of points;

(e) repeat at least 25 times per second the steps (a) through (d) for each subsequent frame of the sequence, the result of step (e) of the preceding frame being used respectively for the fitting process in step (c);

(f) determine the changing pose from positions of predetermined nodes of the model, which have been detected in at least representative frames of the image sequence

(g) wherein the node positions of the model are updated in iteration steps, one point x being randomly sampled for the predetermined number of points of the 3D aggregate of points in each iteration step and all nodes being shifted towards x; and

(h) the degree of the shift being largest for the node which had the smallest distance from x prior to the iteration step wherein the number of the randomly sampled points x or that of the iteration steps is about 10% of the total number of the points within the aggregate of points.