Camera information coding/decoding method for synthesizing stereoscopic real video and a computer graphic image

• US 6,965,699 B2
• Filed: 04/15/2002
• Issued: 11/15/2005
• Est. Priority Date: 12/17/2001
• Status: Expired due to Fees

• Current Assignee: Uniloc 2017 LLC (SoftBank Group Corp.)
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Han, Kyu Seo, Chun, Byung Tae, Jeong, Se Yoon, Bae, Young Lae, Yoon, Ho Sub
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US10/121,583
• Publication Number:
  US 20030112327A1
• Time in Patent Office: 1,310 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/232, 348/42, 348/47, 348/50, 348/51, 348/584, 348/586, 348/587, 348/589, 348/590, 348/591, 348/722, 345/418, 345/419
• US Class Current: 382/232
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A camera information coding/decoding method for synthesizing stereoscopic real video and a computer graphic image is disclosed. The method stores the camera information generated when picturing, which is not contained in the existing multimedia contents coding standard, that is, the location motion information of the camera and the camera viewpoint displacement information, together with contents as the camera motion information stream. The camera information obtained through the camera position sensor and the camera viewpoint displacement sensor when picturing image is coded and stored together with the contents, thereby easily and precisely synthesizing with the computer graphic image. In addition, since the information may be easily transformed to the descriptor of the camera motion information of MPEG-7 format, there is another advantage of easily carrying out camera motion information indexing operation among searching methods of the multimedia contents.

Claims

1. A camera information coding method for synthesizing a stereoscopic real video and a computer graphic image, the method comprising the steps of:

detecting and inputting a camera information on a corresponding frame of an image being pictured at present;

calculating each component difference between the camera information on an inputted present (t) frame and a camera information on a just previous (t−

1) frame to acquire a camera displacement information between the frames;

coding the camera displacement information between the frames to compress and store pictured image contents and the coded camera information; and

updating the coded camera information on the corresponding frame as the camera information on the previous frame.

2. The camera information coding method as claimed in claim 1, wherein the camera information comprises at least one of a camera location information (x, y, z) and a camera viewpoint information.

3. The camera information coding method as claimed in claim 2, wherein the camera viewpoint information comprises at least one of a tilting angle information of the camera, a panning angle information of the camera, a zooming angle of the camera and a rolling information of the camera.

4. The camera information coding method as claimed in claim 2, wherein the camera location information comprises at least one of a tracking information of the camera, a dolling information of the camera, and a booming angle of the camera.

5. The camera information coding method as claimed in claim 1, wherein the coding step employs a Huffman or LZW coding method, or an arbitrary entropy coding method.

6. A camera information decoding method for synthesizing a stereoscopic real video and a computer graphic image, the method comprising the steps of:

if a coded camera information of a current frame at a camera information stream is inputted, decoding the inputted camera to acquire a camera displacement information between the frames;

adding the camera displacement information and a camera information of a previous frame to calculate a location vector, viewpoint vector, and upward vector information of the camera of the current frame;

updating the calculated location vector, viewpoint vector, and upward vector information of the camera of the current frame as the camera information on the previous frame; and

synthesizing the stereoscopic real video and the computer graphic image by use of the location vector, viewpoint vector, and upward vector information of the camera of the current frame.

7. The camera information decoding method as claimed in claim 6, wherein the step of calculating the location vector information of the camera comprises the steps of:

determining whether a bReset value of the camera location information (CameraLocationInfo) of the current frame is true, and if the bReset value is false, adding a DeltaVector of the CameraLocationInfo to a location vector of the previous frame, to acquire a location vector of the camera;

updating the location vector of the camera as a previous value for a next calculation;

carrying out coordinate system compensation transform of the location vector of the camera for compensating an origin location variation between origins upon implementing the computer graphic synthesis, supposing that the origin is the location of an initial location of the camera; and

acquiring locations of left and right cameras which are required for setting the viewpoint of the current frame by carrying out the coordinate system compensation transform.

8. The camera information decoding method as claimed in claim 7, further comprising the steps of:

if the bReset value of CameraLocationInfo of the current frame is true, using the CameraLocation information as the location of the left camera as, and calculating a normal vector (Dnormal) by carrying out the coordinate system compensation transform of the location information of the camera to acquire a location vector of the camera as the location of the right camera;

updating the location vector of the camera as a previous value for a next calculation;

carrying out coordinate system compensation transform of the location vector of the camera for compensating an origin location variation between origins upon implementing the computer graphic synthesis, supposing that the origin is the location of an initial location of the camera; and

acquiring locations of left and right cameras which are required for setting the viewpoint of the current frame by carrying out the coordinate system compensation transform.

9. The camera information decoding method as claimed in claim 8, wherein the normal vector (Dnormal) is parallel with a normal vector of a plane formed by a left viewpoint vector Center and an upward direction vector Up, and has a magnitude corresponding to a distance D between lenses of two cameras.

10. The camera information decoding method as claimed in claim 6, further comprising the step of:

to conduct zooming when synthesizing the stereoscopic real video and the computer graphic image, storing a camera constant information used when matching spaces of the actual image and the computer graphic image, a distance information between two lenses used when the right camera information is acquired from the left camera information, and a lens information for determining a rotating amount of a zoom controller upon changing from the minimum distance to the maximum distance of the lens.

11. The camera information decoding method as claimed in claim 10, wherein the camera constant information comprises at least one of the number of sensors in an x-axis direction of the camera, the number of pixels in an x-axis direction of the camera, a size per sensor in an x-axis direction of the camera, a size per sensor in a y-axis direction of the camera, a size per x-axis pixel, a size per y-axis pixel, a center point of x-axis, a center point of y-axis, and scale element information.

12. A recording medium for executing a camera information coding method for synthesizing a stereoscopic real video and a computer graphic image, the recoding medium capable of being read by a digital processor, and storing a program of commands executed by the digital processor, the program being implemented by types, with the program comprising the steps of:

detecting and inputting a camera information on a corresponding frame of an image being pictured at present;

calculating each component difference between the camera information on an inputted present (t) frame and a camera information on a just previous (t−

1) frame to acquire a camera displacement information between the frames;

coding the camera displacement information between the frames to compress and store pictured image contents and the coded camera information; and

updating the coded camera information on the corresponding frame as the camera information on the previous frame.

14. The program as claimed in claim 12, wherein the camera information comprises at least one of a camera location information (x, y, z) and a camera viewpoint information.

15. The camera information coding method as claimed in claim 14, wherein the camera viewpoint information comprises at least one of a tilting angle information of the camera, a panning angle information of the camera, a zooming angle of the camera and a rolling information of the camera.

16. The camera information coding method as claimed in claim 14, wherein the camera location information comprises at least one of a tracking information of the camera, a dolling information of the camera, and a booming angle of the camera.

17. The camera information coding method as claimed in claim 12, wherein the coding step employs a Huffman or LZW coding method, or an arbitrary entropy coding method.

13. A recording medium for executing a camera information decoding method for synthesizing a stereoscopic real video and a computer graphic image, the recoding medium capable of being read by a digital processor, and storing a program of commands executed by the digital processor, the program being implemented by types, with the program comprising the steps of:

if a coded camera information of a current frame at a camera information stream is inputted, decoding the inputted camera to acquire a camera displacement information between the frames;

adding the camera displacement information and a camera information of a previous frame to calculate a location vector, viewpoint vector, and upward vector information of the camera of the current frame;

updating the calculated location vector, viewpoint vector, and upward vector information of the camera of the current frame as the camera information on the previous frame; and

synthesizing the stereoscopic real video and the computer graphic image by use of the location vector, viewpoint vector, and upward vector information of the camera of the current frame.

18. The program as claimed in claim 13, wherein the step of calculating the location vector information of the camera comprises the steps of:

determining whether a bReset value of the camera location information (CameraLocationInfo) of the current frame is true, and if the bReset value is false, adding a DeltaVector of the CameraLocationInfo to a location vector of the previous frame, to acquire a location vector of the camera;

updating the location vector of the camera as a previous value for a next calculation;

carrying out coordinate system compensation transform of the location vector of the camera for compensating an origin location variation between origins upon implementing the computer graphic synthesis, supposing that the origin is the location of an initial location of the camera; and

acquiring locations of left and right cameras which are required for setting the viewpoint of the current frame by carrying out the coordinate system compensation transform.

19. The camera information decoding method as claimed in claim 18, further comprising the steps of:

if the bReset value of CameraLocationInfo of the current frame is true, using the CameraLocation information as the location of the left camera as, and calculating a normal vector (Dnormal) by carrying out the coordinate system compensation transform of the location information of the camera to acquire a location vector of the camera as the location of the right camera;

updating the location vector of the camera as a previous value for a next calculation;

carrying out coordinate system compensation transform of the location vector of the camera for compensating an origin location variation between origins upon implementing the computer graphic synthesis, supposing that the origin is the location of an initial location of the camera; and

acquiring locations of left and right cameras which are required for setting the viewpoint of the current frame by carrying out the coordinate system compensation transform.

20. The camera information decoding method as claimed in claim 19, wherein the normal vector (Dnormal) is parallel with a normal vector of a plane formed by a left viewpoint vector Center and an upward direction vector Up, and has a magnitude corresponding to a distance D between lenses of two cameras.

21. The camera information decoding method as claimed in claim 13, further comprising the step of:

to conduct zooming when synthesizing the stereoscopic real video and the computer graphic image, storing a camera constant information used when matching spaces of the actual image and the computer graphic image, a distance information between two lenses used when the right camera information is acquired from the left camera information, and a lens information for determining a rotating amount of a zoom controller upon changing from the minimum distance to the maximum distance of the lens.

22. The program as claimed in claim 21, wherein the camera constant information comprises at least one of the number of sensors in an x-axis direction of the camera, the number of pixels in an x-axis direction of the camera, a size per sensor in an x-axis direction of the camera, a size per sensor in a y-axis direction of the camera, a size per x-axis pixel, a size per y-axis pixel, a center point of x-axis, a center point of y-axis, and scale element information.