Video processing system and method for introducing graphical features into video images in a scene

• US 8,648,857 B2
• Filed: 08/12/2008
• Issued: 02/11/2014
• Est. Priority Date: 09/07/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Williams, Michael John
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US12/190,106
• Publication Number:
  US 20090066696A1
• Time in Patent Office: 2,009 Days
• Field of Search: None
• US Class Current: 345/427
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An image processing method introduces a graphical feature or effect into a part of one or more video images of a scene. The scene includes a view of a plurality of objects disposed on an object plane, the part of the video images including one of the plurality of objects on the object plane. The method includes forming a three dimensional model view of the object plane, calculating a transform matrix for transforming points in the model of the object plane into the view of the object plane in the video image, forming the graphical feature or effect in the model of the object plane, transforming the graphical feature or effect from the model to the image view of the object plane using the transform matrix, identifying the one of the plurality of object which is to be viewed within the part of the video image, in which the object is present, and extracting the part of the image view from the video images, into which the graphical feature has been transformed, the extracted part including that part of the feature of effect, which has been transformed from the model. The image processing method provides an improved way of adding a graphical feature or effect to a plane on which a plurality of objects are disposed, so that when a part of the video image, in which one of the plurality of objects, is viewed, the graphical feature or effect appears to the viewer to be a natural scene as if the graphical feature or effect is actually present.

Claims

1. A method of introducing a graphical feature or effect into one or more video image frames, the method comprising:

forming a three dimensional model view of an object plane;

calculating a transform matrix for transforming points in the three dimensional model view of the object plane into an image view of the object plane, the image view of the object plane being represented in the one or more video image frames, the one or more video image frames including images of a scene that includes a view of a plurality of objects disposed on the object plane;

forming the graphical feature or effect in the three dimensional model view of the object plane;

transforming the graphical feature or effect from the three dimensional model view to the image view of the object plane using the transform matrix;

identifying each of the plurality of objects within the one or more video image frames; and

extracting plural parts of each of the one or more video image frames, each part including a different one of the plurality of objects on the object plane, into which the graphical feature or effect has been transformed, each of the extracted parts including a portion of the graphical feature or effect, wherein

calculating the transform matrix includes

calculating a comparison metric between lines formed between a plurality of points transformed from lines of the object plane in the three dimensional model view and corresponding lines between a triplet of points in an identified plurality of triplets of points obtained from the image view of the object plane, and

identifying the transform matrix, the transform matrix being an optimal transform matrix which produces an optimal comparison metric.

2. The method as claimed in claim 1, wherein the video image frames provide a two dimensional view of the object plane, the object plane including a plurality of lines or edges defining features on the object plane, and

wherein the calculating the transform matrix, includes

identifying a plurality of triplets of points on the lines or edges of the object plane in the image view, the image view being obtained from a camera, each of the triplets providing three or more non co-linear points on the lines or edges of the object plane,

identifying a sample triplet of points in the three dimensional model view,

for each of the identified plurality of triplets, calculating a transform matrix which transforms the sample triplet from the three dimensional model view into the respective one of the identified plurality of triplets obtained from the image view,

for each of the calculated transform matrices, calculating a transformation of a plurality of points on the lines of the object plane in the three dimensional model view,

calculating a comparison metric between lines formed between the plurality of points transformed from the lines of the object plane in the three dimensional model view and corresponding lines between a triplet of points in the identified plurality of triplets of points in the image view of the object plane, and

identifying an optimal transform matrix which produces an optimal comparison metric, the optimal transform matrix being used for transforming points on the three dimensional model view into the image view obtained from the camera, or for transforming points in the image view into the three dimensional model view.

3. The method as claimed in claim 1, wherein the video image frames are high definition video image frames obtained from a high definition video camera.

4. The method as claimed in claim 1, wherein

the identifying each of the plurality of objects includes

identifying each of the plurality of objects on the object plane,

generating object path data which represents a position of each of the plurality of objects, which have been identified on the object plane, with respect to time, and

receiving an indication from a user of a desired view of one of the plurality of objects, and

wherein the extracting the plural parts of the image view includes

identifying the one of the plurality of objects, which is to be viewed from the desired view and based on the object path data, which provides a position of the user-indicated object on the object plane with respect to time, and

wherein extracting the plural parts of the video images includes extracting the one of the plurality of objects, the one of the plurality of objects being indicated by the user and identified by the object path data.

5. The method as claimed in claim 1, wherein the plurality of objects are players on a sports field, the object plane being the sports field.

6. The method as claimed in claim 1, wherein a high definition video camera is positioned to provide a view of the scene, and to generate video signals corresponding to the plurality of objects on the object plane.

7. The method as claimed in claim 1, further comprising:

capturing video signals from each of two high definition video cameras; and

combining the captured video signals from each of the high definition video cameras to form the scene that includes the view; and

generating video signals of the plurality of objects on the object plane from the video signals which have been combined from each of the high definition video cameras.

15. A non-transitory computer readable storage medium having computer readable program codes embodied in the medium, which when executed, cause a data processor to execute the method according to claim 1.

17. The method as claimed in claim 1,

wherein each extracted part including one of the plurality of objects and the portion of the graphical feature or effect is displayed at a new location, in the image view of the object plane, that is different from an initial location at which the extracted part was extracted.

18. The method as claimed in claim 1,

wherein the identifying includes determining a team association of each of the plurality of objects.

19. The method as claimed in claim 1,

wherein each extracted part including a different one of the plurality of objects includes a different portion of the graphical feature or effect.

20. The method as claimed in claim 1,

wherein the one or more video image frames are obtained by a single imaging device.

8. An image processing apparatus operable to introduce a graphical feature or effect into one or more video image frames, the image processing apparatus comprising:

a data processor, which is operable to form a three dimensional model view of an object plane, to calculate a transform matrix for transforming points in the three dimensional model view of the object plane into an image view of the object plane, the image view of the object plane being represented in the one or more video image frames, the one or more video image frames including images of a scene that includes a view of a plurality of objects disposed on the object plane, to form the graphical feature or effect in the three dimensional model view of the object plane, to transform the graphical feature or effect from the three dimensional model view to the image view of the object plane using the transform matrix, to identify each of the plurality of objects within the one or more video image frames, and to extract plural parts of each of the one or more video image frames, each part including a different one of the plurality of objects on the object plane, into which the graphical feature or effect has been transformed, each of the extracted parts including a portion of the graphical feature or effect, wherein

in calculating the transform matrix, the data processor is further operable to

calculate a comparison metric between lines formed between a plurality of points transformed from lines of the object plane in the three dimensional model view and corresponding lines between a triplet of points in an identified plurality of triplets of points obtained from the image view of the object plane, and

identify the transform matrix, the transform matrix being an optimal transform matrix which produces an optimal comparison metric.

9. The image processing apparatus as claimed in claim 8,

wherein the video image frames provide a two dimensional view of the object plane, the object plane including a plurality of lines or edges defining features on the object plane, and

wherein, in calculating the transform matrix, the data processor is further operable to identify a plurality of triplets of points on the lines or edges of the object plane in the image view, the image view being obtained from a camera, each of the triplets providing three or more non co-linear points on the lines or edges of the object plane, to identify a sample triplet of points in the three dimensional model view, to calculate, for each of the identified plurality of triplets, a transform matrix which transforms the sample triplet from the three dimensional model view into the respective one of the identified plurality of triplets obtained from the image view, for each of the identified plurality of triplets, to calculate a transformation of a plurality of points on the lines of the object plane in the three dimensional model view, for each of the calculated identified transform matrices, to calculate a comparison metric between lines formed between the plurality of points transformed from the lines of the object plane in the three dimensional model view and corresponding lines between a triplet of points in the identified plurality of triplets of points in the image view of the object plane, and to identify an optimal transform matrix which produces an optimal comparison metric, the optimal transform matrix being used for transforming points on the three dimensional model view into the image view obtained from the camera, or for transforming points in the image view into the three dimensional model view.

10. The image processing apparatus as claimed in claim 8, wherein the video image frames are high definition video image frames obtained from a high definition video camera.

11. The image processing apparatus as claimed in claim 8, wherein

the data processor is operable to identify each of the plurality of objects which is to be viewed within the part of the video image by

identifying each of the plurality of objects on the object plane,

generating object path data which represents a position of each of the plurality of objects, which have been identified on the object plane, with respect to time, and

receiving an indication from a user of a desired view of one of the plurality of objects, and

wherein the extracting the plural parts of the image view includes

identifying the one of the plurality of objects, which is to be viewed from the desired view and based on the object path data, which provides a position of the user indicated object on the object plane with respect to time, and

wherein extracting the plural parts of the video images includes extracting the one of the plurality of objects, the one of the plurality of objects being indicated by the user and identified by the object path data.

12. The image processing apparatus as claimed in claim 8, wherein the plurality of objects are players on a sports field, the object plane being the sports field.

13. The image processing apparatus as claimed in claim 8, wherein a high definition video camera is positioned to provide a view of the scene, and to generate video signals corresponding to the plurality of objects on the object plane.

14. The image processing apparatus as claimed in claim 8, wherein the video images are generated by capturing video signals from each of two respective high definition video cameras, the data processor being operable to:

combine the captured video signals from each of the high definition video cameras to form the scene that includes the view; and

generate video signals of the plurality of objects on the object plane from the video signals which have been combined from each of the high definition video cameras.

21. The image processing apparatus as claimed in claim 8, wherein, in calculating the transform matrix, the data processor is further operable to calculate a transform matrix which transforms a sample triplet in the three dimensional model view into one of an identified plurality of triplets obtained from the image view, for each of the identified plurality of triplets.

22. The image processing apparatus as claimed in claim 8, wherein, in calculating the transform matrix, the data processor is further operable to calculate a transformation of a plurality of points on lines of the object plane in the three dimensional model view.

23. The image processing apparatus as claimed in claim 8, wherein

the transform matrix further transforms points on the image view into the three dimensional model view.

16. An apparatus for introducing a graphical feature or effect into one or more video image frames, the apparatus comprising:

means for forming a three dimensional model view of an object plane;

means for calculating a transform matrix for transforming points in the three dimensional model view of the object plane into an image view of the object plane, the image view of the object plane being represented in the one or more video image frames, the one or more video image frames including images of a scene that includes a view of a plurality of objects disposed on the object plane;

means for forming the graphical feature or effect in the three dimensional model view of the object plane;

means for transforming the graphical feature or effect from the three dimensional model view to the image view of the object plane using the transform matrix;

means for identifying each of the plurality of objects within the one or more video image frames; and

means for extracting plural parts of each of the one or more video image frames, each part including a different one of the plurality of objects on the object plane, into which the graphical feature or effect has been transformed, each of the extracted parts including a portion of the graphical feature or effect, wherein

the means for calculating the transform matrix

calculates a comparison metric between lines formed between a plurality of points transformed from lines of the object plane in the three dimensional model view and corresponding lines between a triplet of points in an identified plurality of triplets of points obtained from the image view of the object plane, and

identifies the transform matrix, the transform matrix being an optimal transform matrix which produces an optimal comparison metric.