VIDEO PROCESSING SYSTEM AND METHOD

• US 20090066696A1
• Filed: 08/12/2008
• Published: 03/12/2009
• Est. Priority Date: 09/07/2007
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Williams, Michael John
• Granted Patent: US 8,648,857 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An image processing method introduces a graphical feature or effect into a part of one or more video images of a scene. The scene includes a view of a plurality of objects disposed on an object plane, the part of the video images including one of the plurality of objects on the object plane. The method includes forming a three dimensional model view of the object plane, calculating a transform matrix for transforming points in the model of the object plane into the view of the object plane in the video image, forming the graphical feature or effect in the model of the object plane, transforming the graphical feature or effect from the model to the image view of the object plane using the transform matrix, identifying the one of the plurality of object which is to be viewed within the part of the video image, in which the object is present, and extracting the part of the image view from the video images, into which the graphical feature has been transformed, the extracted part including that part of the feature of effect, which has been transformed from the model. The image processing method provides an improved way of adding a graphical feature or effect to a plane on which a plurality of objects are disposed, so that when a part of the video image, in which one of the plurality of objects, is viewed, the graphical feature or effect appears to the viewer to be a natural scene as if the graphical feature or effect is actually present.

Claims

1. A method of introducing a graphical feature or effect into a part of one or more video images of a scene, the scene including a view of a plurality of objects disposed on an object plane, the part of the video images including one of the plurality of objects on the object plane, the method comprising

forming a three dimensional model view of the object plane,

calculating a transform matrix for transforming points in the model of the object plane into the view of the object plane in the video image,

forming the graphical feature or effect in the model of the object plane,

transforming the graphical feature or effect from the model to the image view of the object plane using the transform matrix,

identifying the one of the plurality of object which is to be viewed within the part of the video image, in which the object is present, and

extracting the part of the image view from the video images, into which the graphical feature has been transformed, the extracted part including that part of the feature or effect, which has been transformed from the model.

2. A method as claimed in claim 1, wherein the video images provide a two dimensional view of the object plane, the object plane including a plurality of lines or edges defining features on the object plane, and the calculating the transform matrix, includes

identifying a plurality of triplets of points on the lines or edges of the object plane in the image from the camera, each of the triplets providing three or more non co-linear points on the lines of the object in the plane of the object,

identifying a sample triplet of points in the model,

for each of the identified plurality of triplets, calculating a transform matrix which will transform the sample triplet from the model onto the identified triplet from the camera image,

for each of the identified transform matrices, calculating a transformation of a plurality of points on the lines of the object plane in the model,

calculating a comparison metric between lines formed between the triplet of points transformed from the lines of the object plane in the model and corresponding lines between the triplet of points in the image view of the object plane, and

identifying the transform matrix which produces the best comparison metric, the identified transform matrix being for use in transforming points on the model into the image produced by the camera, or points in the image onto the model.

3. A method as claimed in claim 1, wherein the video image are high definition video images, the video camera being a high definition video camera.

4. A method as claimed in claim 1, wherein the identifying the one of the plurality of objects which is to be viewed within the part of the video image, includes

identifying each of the plurality of objects on the object plane,

generating object path data which represents a position of each of the plurality of objects which have been identified on the object plane with respect to time, and

receiving an indication from a user of a desired view of one of the objects, and the extracting the part of the image view from the video images, includes

identifying the one of the plurality of objects, which is to be viewed from the user indication and the object path data, which provides a position of the user indicated object on the object plane with respect to time, and

extracting the part of the video images which includes the user indicated object as identified by the object path data.

5. A method as claimed in claim 1, wherein the objects are players on a sports field, the object plane being the sports field.

6. A method as claimed in claim 1, wherein the video camera is a high definition video camera, which is positioned to provide a view of the scene, to generate the video signals of the plurality of object on the object plane.

7. A method as claimed in claim 1, comprising

generating the video signals by capturing video signals from each of two respective high definition video cameras, and

combining the captured video signals from each of the cameras to form a view of the scene, and

generating the video signals of the plurality of objects on the object plane from the video signals which have been combined from each camera.

16. A data carrier having a recordable medium having recorded thereon information signals representing a computer program, which when loaded onto a data processor performs the method according to claim 1.

8. An image processing apparatus operable to introduce a graphical feature or effect into a part of one or more video images of a scene, the scene including a view of a plurality of objects disposed on an object plane, the part of the video images including one of the plurality of objects on the object plane, the image processing apparatus including a data processor, which is operable

to form a three dimensional model view of the object plane,

to calculate a transform matrix for transforming points in the model of the object plane into the view of the object plane in the video image,

to form the graphical feature or effect in the model of the object plane,

to transform the graphical feature or effect from the model to the image view of the object plane using the transform matrix,

to identify the one of the plurality of object which is to be viewed within the part of the video image, in which the object is present, and

to extract the part of the image view from the video images, into which the graphical feature has been transformed, the extracted part including that part of the feature or effect, which has been transformed from the model.

9. An image processing apparatus as claimed in claim 8, wherein the video images provide a two dimensional view of the object plane, the object plane including a plurality of lines or edges defining features on the object plane, and the calculating the transform matrix, and the data processor is operable

to identify a plurality of triplets of points on the lines or edges of the object plane in the image from the camera, each of the triplets providing three or more non co-linear points on the lines of the object in the plane of the object,

to identify a sample triplet of points in the model,

for each of the identified plurality of triplets, to calculate a transform matrix which will transform the sample triplet from the model onto the identified triplet from the camera image,

for each of the identified transform matrices, to calculate a transformation of a plurality of points on the lines of the object in the model,

to calculate a comparison metric between lines formed between the triplet of points transformed from the lines of the object in the model and corresponding lines between the triplet of points in the image view of the object, and

to identify the transform matrix which produces the best comparison metric, the identified transform matrix being for use in transforming points on the model into the image produced by the camera, or points in the image onto the model.

10. An image processing apparatus as claimed in claim 8, wherein the video images are high definition video images, the video camera being a high definition video camera.

11. An image processing apparatus as claimed in claim 9, wherein the video images are high definition video images, the video camera being a high definition video camera.

12. An image processing apparatus as claimed in claim 8, wherein the data processor is operable to identify the one of the plurality of objects which is to be viewed within the part of the video image by

identifying each of the plurality of objects on the object plane,

generating object path data which represents a position of each of the plurality of objects which have been identified on the object plane with respect to time, and

receiving an indication from a user of a desired view of one of the objects, and the extracting the part of the image view from the video images, includes

identifying the one of the plurality of objects, which is to be viewed from the user indication and the object path data, which provides a position of the user indicated object on the object plane with respect to time, and

extracting the part of the video images which includes the user indicated object as identified by the object path data.

13. An image processing apparatus as claimed in claim 8, wherein the objects are players on a sports field, the object plane being the sports field.

14. An image processing apparatus as claimed in claim 8, wherein the video camera is a high definition video camera, which is positioned to provide a view of the scene, to generate the video signals of the plurality of objects on the object plane.

15. An image processing apparatus as claimed in claim 8, wherein the video images are generated by capturing video signals from each of two respective high definition video cameras, the data processor being operable

to combine the captured video signals from each of the cameras to form a view of the scene, and

to generate the video signals of the plurality of object on the object plane from the video signals which have been combined from each camera.

17. An apparatus for introducing a graphical feature or effect into a part of one or more video images of a scene, the scene including a view of a plurality of objects disposed on an object plane, the part of the video images including one of the plurality of objects on the object plane, the apparatus comprising

means for forming a three dimensional model view of the object plane,

means for calculating a transform matrix for transforming points in the model of the object plane into the view of the object plane in the video image,

means for forming the graphical feature or effect in the model of the object plane,

means for transforming the graphical feature or effect from the model to the image view of the object plane using the transform matrix,

means for identifying the one of the plurality of object which is to be viewed within the part of the video image, in which the object is present, and

means for extracting the part of the image view from the video images, into which the graphical feature has been transformed, the extracted part including that part of the feature or effect, which has been transformed from the model.