Image processing apparatus and method

• US 8,284,238 B2
• Filed: 09/04/2008
• Issued: 10/09/2012
• Est. Priority Date: 09/05/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Stone, Jonathan James, Porter, Robert Mark Stefan, Gillard, Clive Henry
• Primary Examiner(s): NGUYEN, PHUOC H
• Application Number:
  US12/204,417
• Publication Number:
  US 20090066784A1
• Time in Patent Office: 1,496 Days
• Field of Search: 348/46, 348/700
• US Class Current: 348/46
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

An image processing apparatus and a method for generating a three dimensional representation of a scene, which includes a plurality of objects disposed on a plane, is disclosed. The three dimensional representation is generated from one or more video images of the scene, which include the objects on the plane produced from a view of the scene by a video camera. The method includes processing the captured video images to extract image features from each object, comparing image features with sample image features from a predetermined set of possible example objects, and identifying the objects from the comparison of the image features with the sample image features. The method also includes generating object path data, which includes object identification data for each object which identifies the respective object; and provides a position of the object on the plane in the video images with respect to time.

Claims

1. A method of generating a three dimensional representation of a scene, which includes a plurality of objects disposed on a plane, the three dimensional representation being generated from one or more video images of the scene, which video images include the objects on the plane produced from a view of the scene by a video camera, the method comprising:

processing the captured video images so as to extract one or more image features from each object;

comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

identifying the objects from the comparison of the image features with the stored image features of the possible example objects;

generating object path data, which includes object identification data that identifies each of the objects, and which provides a position of each of the objects on the plane in the video images with respect to time; and

calculating a projection matrix for projecting the position of each of the objects according to the object path data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix.

2. The method as claimed in claim 1, further comprising:

generating the three dimensional representation of the scene which includes a synthesised representation of each of the plurality of objects on the plane, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of the height of each of the objects or a simulated distance of a camera viewing the three dimensional representation.

3. The method as claimed in claim 1, further comprising

changing a relative view point of the generated three dimensional model in response to a user control, when displayed to the user on a display screen.

4. The method as claimed in claim 1, wherein

the generating object path data includes

generating for each object an indication of a relative orientation of the object in the plane, and

including the indication of the relative orientation in the object path data, and

the generating the three dimensional representation of the scene includes orientating the synthesised objects in the plane of the three dimensional model using the indication of the orientation provided with the object path data.

5. The method as claimed in claim 1, wherein

each of the objects is a human being,

the generating the object path data includes

generating for each object an indication of a relative attitude, posture or demeanour of the human beings in the plane, and

including the indication of the relative attitude, posture or demeanour of the human beings in the object path data, and

the generating the three dimensional representation of the scene includes

accessing a data store, which includes a set of representations of each of the human beings in the video images, each of the representations in the set corresponding to one of the possible indications of the relative attitude, posture or demeanour of the human beings in the object path data, and

selecting a representation of each of the human beings from the set of possible representations of each of the human beings in accordance with the indication of the relative attitude, posture or demeanour of the human beings in the object path data and generating the selected representation in the plane of the three dimensional model at the position and the orientation according to the object path data.

6. The method as claimed in claim 1, wherein the projection matrix is determined in accordance with

7. The method as claimed in claim 1, further comprising:

identifying each of the objects in the video images using the object path data with respect to the three dimensional model of the plane on which the objects are disposed;

generating a graphical representation of an identifier which identifies each of the objects in the video images; and

overlaying the identifier on the objects in the video images to the effect that one or more of the objects is provided with an identifier as the objects move on the plane in the video images.

8. The method as claimed in claim 1, further comprising:

generating the three dimensional model of the plane on which the objects are disposed; and

calibrating the plane in the generated model to the video image of the plane using the corners of the plane.

9. The method as claimed in claim 1, wherein the objects are players of a sports game, and the plane is a field of play.

10. A non-transitory data carrier comprising computer readable instructions that when executed by a computer, cause the computer to carry out the method in accordance with claim 1.

11. An image processing apparatus operable to generate a three dimensional representation of a scene which includes a plurality of objects disposed on a plane, the three dimensional representation being generated from one or more video images of the scene, which video images include the objects on the plane produced from a view of the scene by a video camera, the image processing apparatus comprising a data processor operable

to process the captured video images so as to extract one or more image features from each object,

to compare the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

to identify the objects from the comparison of the image features with the predetermined image features of the possible example objects,

to generate object path data, which includes object identification data for each object which identifies the respective object, and which provides a position of the object on the plane in the video images with respect to time, and

to calculate a projection matrix for projecting the position of each of the objects according to the object path data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix.

12. The image processing apparatus as claimed in claim 11, wherein the data processor is operable to generate the three dimensional representation of the scene which includes a synthesised representation of each of the plurality of objects on the plane, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of the height of each of the objects or a simulated distance of a camera viewing the three dimensional representation.

13. The image processing apparatus as claimed in claim 11, further comprising:

a user interface for providing user control signals to the data processor, representative of a change of the view of the three dimensional model,

wherein the data processor is operable to change a relative view point of the generated three dimensional model, in response to the user control signals, when the representation of the three dimensional model is displayed on a display screen.

14. The image processing apparatus as claimed in claim 11, wherein

the data processor is operable

to generate for each object an indication of a relative orientation of the object in the plane, and

to include the indication of the relative orientation in the object path data, and

the generating the three dimensional representation of the scene includes orientating the synthesised objects in the plane of the three dimensional model using the indication of the orientation provided with the object path data.

15. The image processing apparatus as claimed in claim 11, wherein

each of the objects is a human being,

the data processor is operable

to generate for each object an indication of a relative attitude, posture or demeanour of the human beings in the plane, and

to include the indication of the relative attitude, posture or demeanour of the human beings in the object path data, and

the generating the three dimensional representation of the scene includes

accessing the data store, which includes a set of representations of each of the human beings in the video images, each of the representations in the set corresponding to one of the possible indications of the relative attitude, posture or demeanour of the human beings in the object path data, and

selecting a representation of each of the human beings from the set of possible representations of each of the human beings in accordance with the indication of the relative attitude, posture or demeanour of the human beings in the object path data and generating the selected representation in the plane of the three dimensional model at the position and the orientation according to the object path data.

16. The image processing apparatus as claimed in claim 11, wherein the data processor is operable

to identify each of the objects in the video images using the object path data with respect to the three dimensional model of the plane on which the objects are disposed,

to generate a graphical representation of an identifier which identifies each of the objects in the video images, and

to overlay the identifier on the objects in the video images to the effect that one or more of the objects is provided with an identifier as the objects move on the plane in the video images.

17. A method of enhancing video images of a scene, which includes a plurality of objects disposed on a plane produced from a view of the scene by a video camera, the method comprising:

processing the captured video images so as to extract one or more image features from each object;

comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain;

identifying the objects from the comparison of the image features with the stored image features of the possible example objects;

generating object path data, which includes object identification data for each object which identifies the respective object, and which provides a position of the object on a three dimensional model of the plane in the video images with respect to time;

calculating a projection matrix for projecting the position of each of the objects according to the object path data from the plane into the three dimensional model of the plane;

identifying each of the objects in the video images using the object path data with respect to the three dimensional model of the plane on which the objects are disposed;

generating a graphical representation of an identifier which identifies each of the objects in the video images; and

overlaying the identifier on the objects in the video images to the effect that one or more of the objects is provided with an identifier as the objects move on the plane in the video images.

18. An image processing apparatus operable to enhance video images of a scene, which includes a plurality of objects disposed on a plane produced from a view of the scene by a video camera, the image processing apparatus comprising a data processor operable:

to process the captured video images so as to extract one or more image features from each object,

to compare the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

to identify the objects from the comparison of the image features with the stored image features of the possible example objects,

to generate object path data, which includes object identification data for each object which identifies the respective object; and

provides a position of the object on a three dimensional model of the plane in the video images with respect to time,

to calculate a projection matrix for projecting the position of each of the objects according to the object path data from the plane into the three dimensional model of the plane,

to identify each of the objects in the video images using the object path data with respect to the three dimensional model of the plane on which the objects are disposed,

to generate a graphical representation of an identifier which identifies each of the objects in the video images, and

to overlay the identifier on the objects in the video images to the effect that one or more of the objects is provided with an identifier as the objects move on the plane in the video images.

19. An apparatus for generating a three dimensional representation of a scene, which includes a plurality of objects disposed on a plane, the three dimensional representation being generated from one or more video images of the scene, which video images include the objects on the plane produced from a view of the scene by a video camera, the apparatus comprising:

means for processing the captured video images so as to extract one or more image features from each object;

means for comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain;

means for identifying the objects from the comparison of the image features with the stored image features of the possible example objects;

means for generating object path data, which includes object identification data for each object which identifies the respective object, and which provides a position of the object on the plane in the video images with respect to time; and

means for calculating a projection matrix for projecting the position of each of the objects according to the object path data from the plane into a three dimensional model of the plane, for generating the three dimensional representation of the scene, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix.