System for communicating and method

• US 8,355,532 B2
• Filed: 09/04/2008
• Issued: 01/15/2013
• Est. Priority Date: 09/05/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Gillard, Clive Henry, David, Morgan William Amos
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US12/204,447
• Publication Number:
  US 20090060321A1
• Time in Patent Office: 1,594 Days
• Field of Search: 382/103, 382/190, 382/278, 348/47, 348/94
• US Class Current: 382/103
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/44 Event detection

Abstract

A system communicates a representation of a scene, which includes a plurality of objects disposed on a plane, to one or more client devices. The representation is generated from one or more video images of the scene captured by a video camera. The system comprises an image processing apparatus operable to receive the video images of the scene which includes a view of the objects on the plane, to process the captured video images so as to extract one or more image features from each object, to compare the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain, to identify the objects from the comparison of the image features with the predetermined image features of the possible example objects, and to generate object path data for each object which identifies the respective object; and provides a position of the identified object on a three dimensional model of the plane in the video images with respect to time. The image processing apparatus is further operable to calculate a projection matrix for projecting the position of each of the objects according to the object path data from the plane in the video image into the three dimensional model of the plane. A distribution server is operable to receive the object path data and the projection matrix generated by the image processing apparatus for distribution of the object path data and the projection matrix to one or more client devices. The system is arranged to generate a representation of an event, such as a sporting event, which provides a substantial data in an amount of information which must be communicated to represent the event. As such, the system can be used to communicate the representation of the event, via a bandwidth limited communications network, such as the internet, from the server to one or more client devices in real time. Furthermore, the system can be used to view one or more of the objects within the video images by extracting the objects from the video images.

Claims

1. A system for communicating a representation of a scene which includes a plurality of objects disposed on a plane to one or more client devices, the representation being generated from one or more video images of a scene captured by a video camera, the system comprising

an image processing apparatus operable to receive the video images of the scene which includes a view of the objects on the plane, received from the video camera,

to process the captured video images so as to extract one or more image features from each object,

to compare the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

to identify the objects from the comparison of the image features with the predetermined image features of the possible example objects,

to generate object path data for each object which identifies the respective object;

and provides a position of the identified object on a three dimensional model of the plane in the video images with respect to time;

to calculate a projection matrix for projecting the position of each of the objects according to the object path data from the plane in the video images into the three dimensional model of the plane, and

a distribution server operable to receive the object path data and the projection matrix generated by the image processing apparatus for distribution of the object path data and the projection matrix to one or more client devices.

2. A system as claimed in claim 1, wherein the client device is operable

to generate a three dimensional representation of the scene, which includes a synthesised representation of each of the plurality of objects disposed on the plane, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of a height of each of the objects.

3. A system as claimed in claim 1, wherein the client device includes a user interface for receiving user control signals representative of a request for a change of the view of the three dimensional model, and in response to the user control signals the client device is operable to change a relative view point of the generated three dimensional model, when the representation of the three dimensional model is displayed on a display screen.

4. A system as claimed in claim 1, wherein the image processing apparatus is operable

to generate for each object an indication of a relative orientation of the object in the plane, and

to include the indication of the relative orientation in the object path data, and the client device is operable to orientate the synthesised objects in the plane of the three dimensional model using the indication of the orientation provided with the object path data.

5. A system as claimed in claim 1, wherein the client device is arranged to receive data representative of the synthesised representation of each of the objects for use in generating the three dimensional model.

6. A system as claimed in claim 1, wherein the client device is operable to communicate a request to the server, the request representing a user selection of one at least one of a representation of the objects, the plane within the scene of the model or the projection matrix, and the server is operable to adapt the representation of the objects, the plane of the scene of the model or the projection matrix in accordance with the request and to communicate data representative of the adapted representation of the objects, the plane within the scene of the model or the projection matrix for use in generating the three dimensional model.

7. A system as claimed in claim 1, wherein each of the objects are human beings and the image processing apparatus is operable

to generate for each object an indication of a relative attitude, posture or demeanour of the human beings in the plane from the video images, and

to include the indication of the relative attitude, posture or demeanour of the human beings in the object path data, and the client device is operable

to retrieve a set of representations of each of the human beings in the video images, each of the representations in the set corresponding to one of the possible indications of the relative attitude, posture or demeanour of the human beings in the object path data, and

to select a representation of each of the human beings from the set of possible representations of each of the objects in accordance with the indication of the relative attitude, posture or demeanour of the human beings in the object path data, and

to generate the selected representation in the plane of the three dimensional model at the position and the orientation according to the object path data.

8. A system as claimed in claim 1, wherein the object path data and the projection matrix are communicated to the client devices using an internet protocol via an internet network.

9. A system as claimed in claim 1, wherein the distribution server is arranged to receive the video images of the scene, and the distribution server is operable to communicate the video images with the object path data and the projection matrix to the one or more client devices, and the client devices are operable

to identify each of the objects, using the object path data and the projection matrix received from the distribution server, and

in response to a user selection, to display a part of the video images within which one of more of the identified objects are located.

10. A client device operable to generate a three dimensional representation of a scene which includes a plurality of objects disposed on a plane, the three dimensional representation being generated from one or more video images captured by a camera from a view of the scene which includes the plurality of objects, the client device including

a communications interface operable to receive data representative of object path data and a projection matrix generated by an image processing apparatus, the object path data providing for each object an identity of the respective object;

and a position of the identified object on a three dimensional model of the plane in the video images with respect to time, and

a data processor operable to generate the three dimensional representation of the scene which includes a synthesised representation of each of the plurality of objects on the plane from the object path data and the projection matrix received from the server.

11. A client device as claimed in claim 10, wherein the data processor is operable to generate the three dimensional representation of the scene which includes the synthesised representation of each of the plurality of objects on the plane, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of the height of each of the objects.

12. A client device as claimed in claim 10, comprising a user interface for receiving user control signals representative of a request for a change of the view of the three dimensional model, and in response to the user control signals the client device is operable to change a relative view point of the generated three dimensional model, when the representation of the three dimensional model is displayed on a display screen.

15. A client device as claimed in claim 12, comprising a user interface for receiving user control signals representative of a request for a part of the video images associated with one or more of the identified objects to be displayed, and in response to the user control signals the client device is operable to extracted the part of the video images in which the identified one or more objects are located, and to display the extracted part.

13. A client device operable to display a part of a one or more video image of a scene which includes a plurality of objects disposed on a plane, the video images being captured by a camera from a view of the scene which includes the plurality of objects, the client device including

a communications interface operable to receive data representative of object path data and a projection matrix generated by an image processing apparatus, the object path data providing for each object an identity of the respective object;

and a position of the identified object on a three dimensional model of the plane in the video images with respect to time, and to receive the one or more video images,

a data processor operable to identify each of the objects on the plane within the video images, using the object path data and the projection matrix, and

in response to a user selection, to display a part of the video images within which one of more of the identified objects are located.

14. A client device as claimed in claim 13, wherein the data processor is operable to identify the objects on the plane in the video images, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of the height of each of the objects, and matching the plane in the video images to the plane in the three dimensional model.

16. A server for distributing object path data and a projection matrix to one or more client devices produced by an image processing apparatus, the image processing apparatus producing the object path data and the projection matrix from one or more video images of a scene which includes a view of a plurality of objects on a plane, the video images being produced by a video camera, processing the captured video images so as to extract one or more image features from each object, comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain, identifying the objects from the comparison of the image features with the stored image features of the possible example objects, generating the object path data for each object which identifies the respective object;

and provides a position of the identified object on the plane of the three dimensional model with respect to time; and

calculating the projection matrix for projecting the position of each of the objects according to the object path data from the plane in the video image into a three dimensional model of the scene, the server being operable

to receive the object path data and the projection matrix, and

to communicate on request, the object path data and the projection matrix to the client devices.

17. A method of communicating a representation of a scene which includes a plurality of objects disposed on a plane to one or more client devices, the representation being generated from one or more video images of the scene, which include the objects on the plane produced by a video camera, the method comprising

processing the captured video images so as to extract one or more image features from each object,

comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

identifying the objects from the comparison of the image features with the predetermined image features of the possible example objects,

generating object path data for each object, which identifies the respective object;

and provides a position of the identified object on a three dimensional model of the plane in the video images with respect to time;

calculating a projection matrix for projecting the position of each of the objects according to the object path data from the plane in the video image into the three dimensional model of the plane, and

communicating the object path data and the projection matrix to one or more client devices.

18. A method as claimed in claim 17, comprising

generating a three dimensional representation of the scene, which includes a synthesised representation of each of the plurality of objects disposed on the plane, by projecting the position of the objects according to the object path data into the plane of the three dimensional model of the scene using the projection matrix and a predetermined assumption of a height of each of the objects.

19. A method as claimed in claim 17, comprising

communicating the video images with the object path data and the projection matrix to the one or more client devices,

identifying, at the client devices each of the objects, using the object path data and the projection matrix received from the distribution server, and

in response to a user selection, displaying a part of the video images within which one of more of the identified objects are located.

20. A non-transitory computer-readable storage medium having computer readable program codes embodied in the computer readable storage medium that, when executed cause a computer to execute the method of claim 17.

21. An apparatus for communicating a representation of a scene which includes a plurality of objects disposed on a plane to one or more client devices, the representation being generated from one or more video images of the scene, which include the objects on the plane produced by a video camera, the apparatus comprising

means for processing the captured video images so as to extract one or more image features from each object,

means for comparing the one or more image features with sample image features from a predetermined set of possible example objects which the video images may contain,

means for identifying the objects from the comparison of the image features with the predetermined image features of the possible example objects,

means for generating object path data for each object, which identifies the respective object;

and provides a position of the identified object on a three dimensional model of the plane in the video images with respect to time;

means for calculating a projection matrix for projecting the position of each of the objects according to the object path data from the plane in the video image into the three dimensional model of the plane, and

means for communicating the object path data and the projection matrix to one or more client devices.