IMAGE PROCESSING APPARATUS AND METHOD

• US 20090141933A1
• Filed: 12/04/2008
• Published: 06/04/2009
• Est. Priority Date: 12/04/2007
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: WAGG, David
• Granted Patent: US 8,233,721 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition

Abstract

A method of matching a pose of a synthesised representation of a human or animal body to a captured image of that human or animal body is provided, which can be used to generate a graphical model of the body when disposed on a plane, such as a synthesised model of a football player on a field of play. The method includes receiving the captured image data, determining from the captured image data a plurality of limb position estimates, each position estimate corresponding to an amount by which limbs of the body are separated with respect to each other and deriving from the plurality of limb positions an estimated gait phase of the body. The estimated gait phase is then applied to a basis gait model in order to provide an estimated pose of the body, the basis gait model comprising data which defines a displacement of the limbs or parts thereof with respect to a gait cycle period. The estimated pose is then matched to that of the synthesised representation of the body.

Claims

1. A computer-readable medium including a program which, when executed by a processor, causes the processor to perform a method of matching a pose of a synthesised representation of a human or animal body to a captured image of the human or animal body, the method comprising:

receiving the captured image data;

determining from the captured image data a plurality of limb position estimates, each position estimate corresponding to an amount by which limbs of the body are separated with respect to each other;

deriving from the plurality of limb positions an estimated gait phase of the body;

applying the estimated gait phase to a basis gait model in order to provide an estimated pose of the body, the basis gait model comprising data which defines a displacement of the limbs or parts thereof from which the body is comprised with respect to a gait cycle period; and

matching the estimated pose of the body to that of the synthesised representation of the body.

2. The method according to claim 1, comprising:

determining from the captured image data an estimated velocity of the body, and

calculating the basis gait model by combining a plurality of gait models in accordance with a basis gait function, each gait model corresponding to a mode of movement of the body, the basis gait function having a plurality of blending coefficients in dependence upon the estimated body velocity, the blending coefficients determining a relative weighting with which each gait model is combined with the gait models.

3. The method according to claim 1, wherein the plurality of gait models include a walking mode gait model, running mode gait model and a sprinting mode gait model.

4. The method according to claim 2, wherein the plurality of blending coefficients comprise a first, second and third blending coefficients, the first blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to a predetermined running speed threshold value and a predetermined sprinting speed threshold value, the second blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to a predetermined walking speed threshold value, the predetermined running speed threshold value and the predetermined sprinting speed threshold value, and the third blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to the predetermined walking speed threshold value and the predetermined running speed threshold value.

5. The method according to claim 1, wherein

when the gait phase estimate cannot be determined from the image data, calculating the gait phase estimate in accordance with an update equation, the update equation providing a gait phase estimate by adding a phase change to a previously provided gait phase estimate, the phase change being determined from a relationship between the speed of movement of the body and gait frequency.

6. The method according to claim 2, further comprising:

determining from the captured image data an estimated orientation of the body;

determining from a comparison of the estimated player orientation and the estimated body velocity if the body has a forward gait or a backward gait with respect to the estimated body orientation;

when the body has a forward gait applying the estimated gait phase to the basis gait model as a positive phase value; and

when the body has a backward gait applying the estimated gait phase to the basis gait model as a negative phase value.

7. The method according to claim 1, further comprising:

receiving body tracking data, providing a position of the body on a plane in the captured images with respect to time;

generating a graphical model representing the synthesised body on the plane, and

combining the body tracking data with the estimation of the pose of the body to adjust the position and pose of the body on the plane in accordance with the body tracking data.

8. The method according to claim 7, wherein the body tracking data is generated by

processing the captured images so as to extract one or more image features from each body,

comparing the one or more image features with sample image features from a predetermined set of possible example bodies which the video images may contain,

identifying the body from the comparison of the image features with the stored image features of the possible example bodies,

generating body tracking data, which provides a position of the object on the plane in the video images with respect to time;

calculating a projection matrix for projecting the position of the body according to the body tracking data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the body according to the body tracking data into the plane of the three dimensional model of the scene using the projection matrix.

9. A computer-readable medium including a program which, when executed by a processor, causes the processor to perform a method of generating a three dimensional representation of a scene, which includes a plurality of bodies disposed on a plane, the three dimensional representation being generated from one or more video images of the scene, which video images include the bodies on the plane produced from a view of the scene by a video camera, the method comprising:

receiving body tracking data for each of the bodies, the body tracking data representing a position of the body on the plane in the video images with respect to time, within the video images,

receiving body orientation data representing a relative orientation of each of the bodies on the plane within the video images with respect to time,

receiving body pose data providing a pose of each of the bodies on the plane within the video images with respect to time, and

generating the three dimensional model of the bodies on the plane, each of the bodies being positioned on the plane according to the body tracking data and each of the bodies being orientated according to the orientation data.

10. The method according to claim 9, wherein the body pose data is generated by receiving the captured image data,

determining from the captured image data a plurality of limb position estimates, each position estimate corresponding to an amount by which limbs of the body are separated with respect to each other,

deriving from the plurality of limb positions an estimated gait phase of the body,

applying the estimated gait phase to a basis gait model in order to provide an estimated pose of the body, the basis gait model comprising data which defines a displacement of the limbs or parts thereof from which the body is comprised with respect to a gait cycle period, and

matching the estimated pose of the body to that of the synthesised representation of the body.

11. The method according to claim 9, wherein the body tracking data is generated by

processing the captured video images so as to extract one or more image features from each body,

comparing the one or more image features with sample image features from a predetermined set of possible example bodies which the video images may contain,

identifying the bodies from the comparison of the image features with the stored image features of the possible example bodies,

generating body tracking data, which includes body identification data for each body, which identifies the respective body;

and provides a position of the body on the plane in the video images with respect to time, and

calculating a projection matrix for projecting the position of each of the bodies according to the body path data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the bodies according to the body path data into the plane of the three dimensional model of the scene using the projection matrix.

12. An image processing apparatus for matching a pose of a synthesised representation of a human or animal body to a captured image of the human or animal body, the apparatus including a data processor operable

to receive the captured image data,

to determine from the captured image data a plurality of limb position estimates, each limb position estimate corresponding to an amount by which limbs of the body are separated with respect to gait,

to derive from the plurality of limb positions an estimated gait phase of the body,

to apply the estimated gait phase to a basis gait model in order to provide an estimated pose of the body, the basis gait model comprising data which defines the displacement of the limbs or parts thereof with respect to a gait cycle period, and

to match the estimated pose of the body to that of the synthesised representation of the body.

13. The image processing apparatus according to claim 12, wherein

the data processor is operable

to determine from the received captured image data an estimated velocity of the body, and

to calculate the basis gait model by combining a plurality of gait models in accordance with a basis gait function, each gait model corresponding to a mode of movement of the body, the basis gait function having a plurality of blending coefficients in dependence upon the estimated body velocity, the blending coefficients determining a relative weighting with which each gait model is combined with the gait models.

14. The image processing apparatus according to claim 13, wherein the plurality of gait models include a walking mode gait model, a running mode gait model and a sprinting mode gait model.

15. The image processing apparatus according to claim 14, wherein the plurality of blending coefficients comprise a first, second and third blending coefficients, the first blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to a predetermined running speed threshold value and a predetermined sprinting speed threshold value, the second blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to a predetermined walking speed threshold value, the predetermined running speed threshold value and the predetermined sprinting speed threshold value, and the third blending coefficient being set in dependence on the magnitude of the estimated body velocity in relation to the predetermined walking speed threshold value and the predetermined running speed threshold value.

16. The image processing apparatus according to claim 12, wherein

if the gait phase estimate cannot be determined from the image data, calculating the gait phase estimate in accordance with an update equation, the update equation providing a gait phase estimate by adding a phase change to a previously provided gait phase estimate, the phase change being determined from a relationship between the speed of movement of the body and gait frequency.

17. The image processing apparatus according to claim 12, wherein the data processor is operable to determine from the image data an estimated orientation of the body and from a comparison of the estimated player orientation and the estimated body velocity to determine if the body has a forward gait or a backward gait with respect to the estimated body orientation, and

if the body has a forward gait the apparatus is operable to apply the estimated gait phase to the basis gait model as a positive phase value, and if the body has a backward gait applying the estimated gait phase to the basis gait model as a negative phase value.

18. The image processing apparatus according to claim 17, further comprising

a graphical processor operable to receive body tracking data, providing a position of the body on a plane in the captured images with respect to time, to generate a graphical model representing the synthesised body on the plane, and to combine the body tracking data with the estimation of the pose of the body to adjust the position and the pose of the synthesised body on the plane in accordance with the body tracking data, and the estimation of the pose of the body on the plane.

19. The image processing apparatus according to claim 18, wherein the data processor is operable to generate the body tracking data by

processing the captured images so as to extract one or more image features from each body,

comparing the one or more image features with sample image features from a predetermined set of possible example bodies which the video images may contain,

identifying the body from the comparison of the image features with the stored image features of the possible example bodies,

generating body tracking data, which provides a position of the object on the plane in the video images with respect to time;

calculating a projection matrix for projecting the position of the body according to the body tracking data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the body according to the body tracking data into the plane of the three dimensional model of the scene using the projection matrix.

20. An apparatus for generating a three dimensional representation of a scene, which includes a plurality of bodies disposed on a plane, the three dimensional representation being generated from one or more video images of the scene, which video images include the bodies on the plane produced from a view of the scene by a video camera, the apparatus including a data processor which is operable

to receive body tracking data for each of the bodies, the object path representing a position of the body on the plane in the video images with respect to time, within the video images,

to receive body orientation data representing a relative orientation of each of the bodies on the plane within the video images with respect to time,

to receive body pose data providing a pose of each of the bodies on the plane within the video images with respect to time, and

to generate the three dimensional model of the bodies on the plane, each of the bodies being positioned on the plane according to the object path data and each of the bodies being orientated according to the orientation data.

21. The apparatus according to claim 20, wherein the body pose data is generated by receiving the captured image data,

determining from the captured image data a plurality of limb position estimates, each position estimate corresponding to an amount by which limbs of the body are separated with respect to each other,

deriving from the plurality of limb positions an estimated gait phase of the body,

applying the estimated gait phase to a basis gait model in order to provide an estimated pose of the body, the basis gait model comprising data which defines a displacement of the limbs or parts thereof from which the body is comprised with respect to a gait cycle period, and

matching the estimated pose of the body to that of the synthesised representation of the body.

22. The apparatus according to claim 20, wherein the body tracking data is generated by

processing the captured video images so as to extract one or more image features from each body,

comparing the one or more image features with sample image features from a predetermined set of possible example bodies which the video images may contain,

identifying the bodies from the comparison of the image features with the stored image features of the possible example bodies,

generating body tracking data, which includes body identification data for each body, which identifies the respective body;

and provides a position of the body on the plane in the video images with respect to time, and

calculating a projection matrix for projecting the position of each of the bodies according to the body path data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the bodies according to the body path data into the plane of the three dimensional model of the scene using the projection matrix.