SYSTEM AND METHOD FOR REAL-TIME PROCESSING OF ULTRA-HIGH RESOLUTION DIGITAL VIDEO

• US 20160205341A1
• Filed: 08/20/2014
• Published: 07/14/2016
• Est. Priority Date: 08/20/2013
• Status: Active Application

• Current Assignee: Smarter Tv Ltd.
• Original Assignee: Smarter Tv Ltd.
• Inventors: HOLLANDER, Elad Moshe, SHENKAR, Victor
• Application Number:
  US14/913,276
• Publication Number:
  US 20160205341A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 7/015 High-definition television systems

Abstract

A method for encoding a video stream generated from at least one ultra-high resolution camera capturing sequential image frames from a fixed viewpoint of a scene includes decomposing the sequential image frames into quasi-static background and dynamic image features; distinguishing between different objects represented by the dynamic image features by recognizing characteristics and tracking movement of the objects in the sequential image frames. The dynamic image features are formatted into a sequence of miniaturized image frames that reduces at least one of: inter-frame movement of the objects; and high spatial frequency data. The sequence is compressed into a dynamic data layer and the quasi-static background into a quasi-static data layer. The dynamic data layer and the quasi-static data layer are encoded with setting metadata pertaining to the scene and the at least one ultra-high resolution camera, and corresponding consolidated formatting metadata pertaining to the decomposing and formatting procedures.

Claims

1. A method for encoding a video stream generated from at least one ultra-high resolution camera capturing a plurality of sequential image frames from a fixed viewpoint of a scene, the method comprising the procedures of:

decomposing said sequential image frames into quasi-static background and dynamic image features;

distinguishing between different objects represented by said dynamic image features by recognizing characteristics of said objects and by tracking movement of said objects in said sequential image frames;

formatting said dynamic image features into a sequence of miniaturized image frames that reduces at least one of;

inter-frame movement of said objects in said sequence of miniaturized image frames; and

high spatial frequency data in said sequence of miniaturized image frames;

compressing said sequence of miniaturized image frames into a dynamic data layer and said quasi-static background into a quasi-static data layer; and

encoding said dynamic data layer and said quasi-static data layer with setting metadata pertaining to said scene and said at least one ultra-high resolution camera, and corresponding consolidated formatting metadata pertaining to said decomposing procedure and said formatting procedure.

2. The method according to claim 1, further comprising an initial procedure of calibrating the respective position and orientation of each of said at least one ultra-high resolution camera in relation to a global coordinate system associated with said scene, thereby defining said setting metadata.

3. The method according to claim 2, further comprising a preliminary procedure of determining said setting metadata which includes a scene model describing spatial characteristics pertaining to said scene, a camera model describing respective extrinsic and intrinsic parameters of each of said at least one ultra-high resolution camera, and data yielded from said calibrating procedure.

4. The method according to claim 3, wherein said calibrating procedure facilitates generation of back-projection functions that transform from respective image coordinates of said sequential image frames captured from said at least one ultra-high resolution camera to said global coordinate system.

5. The method according to claim 1, wherein said consolidated formatting metadata includes information that describes data contents of formatted said dynamic image features.

6. The method according to claim 1, wherein a miniaturized image frame in said sequence of miniaturized image frames includes a respective miniature image of said object, recognized from said dynamic image features.

7. The method according to claim 5, wherein said consolidated formatting metadata includes at least one of:

correspondence data that associates a particular identified said object with its position in said sequence of miniaturized image frames, specifications of said sequence of miniaturized image frames, and data specifying reduction of said high spatial frequency data.

8. The method according to claim 1, further comprising the procedure of transmitting encoded said dynamic data layer and said quasi-static data layer with said setting metadata and encoded said consolidated formatting metadata.

9. The method according to claim 1, further comprising a procedure of completing said quasi-static background in areas of said sequential image frames where former positions of said dynamic images features were assumed prior to said procedure of decomposition.

10. The method according to claim 1, wherein said sequence of miniaturized image frames and said quasi-static background are compressed separately in said compressing procedure.

11. The method according to claim 1, further comprising a procedure of decoding the encoded said quasi-static data layer, and the encoded said dynamic data layer with corresponding encoded said consolidated formatting metadata, and with said setting metadata, so as to respectively generate a decoded quasi-static data layer, a decoded dynamic data layer, and decoded consolidated formatting metadata.

12. The method according to claim 11, further comprising a procedure of decompressing said decoded quasi-static layer, said decoded dynamic data layer, and said decoded consolidated formatting metadata.

13. The method according to claim 1, wherein each of said at least one ultra-high resolution camera has a different said fixed viewpoint of said scene.

14. The method according to claim 4, further comprising a procedure of receiving as input a user-selected virtual camera viewpoint of said scene that is different from said fixed viewpoint captured from said at least one ultra-high resolution camera, said user-selected virtual camera viewpoint is associated with a virtual camera coordinate system in relation to said global coordinate system.

15. The method according to claim 14, further comprising a procedure of generating from said sequential image frames a rendered output video stream that includes a plurality of rendered image frames, using said setting metadata and given input relating to said user-selected virtual camera viewpoint.

16. The method according to claim 15, wherein said rendered output video stream is generated in particular, by mapping each of said back-projection functions each associated with a respective said at least one ultra-high resolution camera onto said virtual camera coordinate system, thereby creating a set of three-dimensional (3-D) data points that are projected onto a two-dimensional surface so as to yield said rendered image frames.

17. The method according to claim 15, wherein said rendered image frames include at least one of:

a representation of at least part of said quasi-static data layer, and a representation of at least part of said dynamic data layer respectively corresponding to said dynamic image features, wherein said consolidated formatting metadata determines the positions and orientations of said dynamic image features in said rendered image frames.

18. The method according to claim 17, further comprising a procedure of incorporating graphics content into said rendered image frames.

19. The method according to claim 17, further comprising a procedure of displaying said rendered image frames.

20. The method according claim 19, further comprising a procedure of providing information about a particular said object exhibited in displayed said rendered image frames, in response to user input.

21. The method according to claim 19, further comprising a procedure of providing a selectable viewing mode of displayed said rendered image frames.

22. The method according to claim 21, wherein said selectable viewing mode is selected from a list consisting of:

zoom-in viewing mode;

zoom-out viewing mode;

object tracking viewing mode;

viewing mode where imaged said scene matches said fixed viewpoint generated from one of said ultra-high resolution cameras;

user-selected manual display viewing mode;

follow-the-anchor viewing mode;

user-interactive viewing mode; and

simultaneous viewing mode.

23. The method according to claim 1, further comprising a procedure of synchronizing each of said at least one ultra-high resolution camera to a reference time.

24. The method according to claim 11, wherein said encoding and said decoding are performed in real-time.

25. The method according to claim 1, wherein at least two of said at least one ultra-high resolution camera is configured as adjacent pairs, where each of said at least one ultra-high resolution camera in a pair is operative to substantially capture said sequential image frames from different complementary areas of said scene.

26. The method according to claim 25, further comprising a procedure of calibrating between said adjacent pairs so as to minimize the effect of parallax.

27. The method according to claim 25, wherein at least two of said at least one ultra-high resolution camera is configured so as to provide stereoscopic image capture of said scene.

28. The method according to claim 1, wherein said sequential image frames of said video stream have a resolution of at least 8 megapixels.

29. The method according to claim 1, wherein said scene includes a sport playing ground/pitch.

30. The method according to claim 29, wherein said sport playing ground/pitch is selected from a list consisting of:

soccer/football field;

Gaelic football/rugby pitch;

basketball court;

baseball field;

tennis court;

cricket pitch;

hockey filed;

ice hockey rink;

volleyball court;

badminton court;

velodrome;

speed skating rink;

curling rink;

equine sports track;

polo field;

tag games fields;

archery field;

fistball field;

handball field;

dodgeball court;

swimming pool;

combat sports rings/areas;

cue sports tables;

flying disc sports fields;

running tracks;

ice rink;

snow sports areas;

Olympic sports stadium;

golf field;

gymnastics arena;

motor racing track/circuit;

card games tables/spaces;

board games boards;

table sports tables;

casino games areas;

gambling tables;

performing arts stages;

auction areas; and

dancing ground.

31. A system for providing ultra-high resolution video, the system comprising:

at least one ultra-high resolution camera that captures a plurality of sequential image frames from a fixed viewpoint of a scene;

a server node comprising;

a server processor coupled with said at least one ultra-high resolution camera, said server processor decomposes said sequential image frames into quasi-static background and dynamic image features thereby yielding decomposition metadata, said server processor distinguishes between different objects represented by said dynamic image features by recognizing characteristics of said objects and by tracking movement of said objects in said sequential image frames, said server processor formatting said dynamic image features into a sequence of miniaturized image frames that reduces at least one of;

inter-frame movement of said objects in said sequence of miniaturized image frames; and

high spatial frequency data in said sequence of miniaturized image frames, thereby yielding formatting metadata;

said server processor compresses said sequence of miniaturized image frames into a dynamic data layer and said quasi-static background into a quasi-static data layer, said server processor encodes said dynamic data layer and said quasi-static data layer with metadata that includes setting metadata pertaining to said scene and said at least one ultra-high resolution camera, and consolidated formatting metadata that includes said decomposition metadata and said formatting metadata; and

a server communication module, coupled with said server processor, for transmitting encoded said dynamic data layer and encoded said quasi-static data layer; and

at least one client node communicatively coupled with said server node, said at least one client node comprising;

a client communication module for receiving encoded said metadata, encoded said dynamic data layer and encoded said quasi-static data layer; and

a client processor, coupled with said client communication module, said client processor decodes and combines encoded said dynamic data layer and encoded said quasi-static data layer, according to said metadata that includes said consolidated formatting metadata, so as to generate an output video stream that reconstructs said sequential image frames.

32. The system according to claim 31, wherein the position and orientation of each of said at least one ultra-high resolution camera in relation to a global coordinate system associated with said scene are calibrated and recorded by said server node, thereby defining said setting metadata.

33. The system according to claim 32, wherein said setting metadata includes a scene model describing spatial characteristics pertaining to said scene, a camera model describing respective extrinsic and intrinsic parameters of each of said at least one ultra-high resolution camera, and data yielded from calibration.

34. The system according to claim 33, wherein said server node generates, via said calibration, back-projection functions that transform from respective said image coordinates of said sequential image frames captured from said at least one ultra-high resolution camera to said global coordinate system.

35. The system according to claim 31, wherein said consolidated formatting metadata includes information that describes data contents of formatted said dynamic image features.

36. The system according to claim 31, wherein a miniaturized image frame in said sequence of miniaturized image frames includes a respective miniature image of said object, recognized from said dynamic image features.

37. The system according to claim 35, wherein said consolidated formatting metadata includes at least one of:

correspondence data that associates a particular identified said object with its position in said sequence of miniaturized image frames, specifications of said sequence of miniaturized image frames, and data specifying reduction of said high spatial frequency data.

38. The system according to claim 31, wherein said server node completes said quasi-static background in areas of said sequential image frames where former positions of said dynamic images features were assumed prior to decomposition.

39. The system according to claim 31, wherein said sequence of miniaturized image frames and said quasi-static background are compressed separately.

40. The system according to claim 31, wherein said client node generates decoded quasi-static data layer from received said encoded quasi-static data layer, and decoded dynamic data layer from received said encoded dynamic data layer, with corresponding said consolidated formatting metadata.

41. The system according to claim 40, wherein said client node decompresses said decoded quasi-static layer, said decoded dynamic data layer, and said decoded metadata.

42. The system according to claim 31, wherein each of said at least one ultra-high resolution camera has a different said fixed viewpoint of said scene.

43. The system according to claim 34, wherein said client node receives as input a user-selected virtual camera viewpoint of said scene that is different from said fixed viewpoint captured from said at least one ultra-high resolution camera, said user-selected virtual camera viewpoint is associated with a virtual camera coordinate system in relation to said global coordinate system.

44. The system according to claim 43, wherein said client node generates from said sequential image frames a rendered output video stream that includes a plurality of rendered image frames, using said setting metadata and given input relating to said user-selected virtual camera viewpoint.

45. The system according to claim 44, wherein said rendered output video stream is generated in particular, by mapping each of said back-projection functions each associated with a respective said at least one ultra-high resolution camera onto said virtual camera coordinate system, thereby creating a set of three-dimensional (3-D) data points that are projected onto a two-dimensional surface so as to yield said rendered image frames.

46. The system according to claim 44, wherein said rendered image frames include at least one of:

a representation of at least part of said quasi-static data layer, and a representation of at least part of said dynamic data layer respectively corresponding to said dynamic image features, wherein said consolidated formatting metadata determines the positions and orientations of said dynamic image features in said rendered image frames.

47. The system according to claim 46, wherein said client node incorporates graphics content into said rendered image frames.

48. The system according to claim 46, further comprising a client display coupled with said client processor for displaying said rendered image frames.

49. The system according claim 48, wherein said client node provides information about a particular said object exhibited in displayed said rendered image frames, in response to user input.

50. The system according to claim 48, further comprising a procedure of providing a selectable viewing mode of displayed said rendered image frames.

51. The system according to claim 50, wherein said selectable viewing mode is selected from a list consisting of:

zoom-in viewing mode;

zoom-out viewing mode;

object tracking viewing mode;

viewing mode where imaged said scene matches said fixed viewpoint generated from one of said ultra-high resolution cameras;

user-selected manual display viewing mode;

follow-the-anchor viewing mode;

user-interactive viewing mode; and

simultaneous viewing mode.

52. The system according to claim 31, wherein said server node synchronizes each of said at least one ultra-high resolution camera to a reference time.

53. The system according to claim 41, wherein said encoding and said decoding are performed in real-time.

54. The system according to claim 31, wherein at least two of said at least one ultra-high resolution camera is configured as adjacent pairs, where each of said at least one ultra-high resolution camera in a pair is operative to substantially capture said sequential image frames from different complementary areas of said scene.

55. The system according to claim 54, wherein said adjacent pairs are calibrated so as to minimize the effect of parallax.

56. The system according to claim 55, wherein at least two of said at least one ultra-high resolution camera is configured so as to provide stereoscopic image capture of said scene.

57. The system according to claim 31, wherein said sequential image frames of said video stream have a resolution of at least 8 megapixels.

58. The system according to claim 31, wherein said scene includes a sport playing ground/pitch.

59. The system according to claim 58, wherein said sport playing ground/pitch is selected from a list consisting of:

soccer/football field;

Gaelic football/rugby pitch;

basketball court;

baseball field;

tennis court;

cricket pitch;

hockey filed;

ice hockey rink;

volleyball court;

badminton court;

velodrome;

speed skating rink;

curling rink;

equine sports track;

polo field;

tag games fields;

archery field;

fistball field;

handball field;

dodgeball court;

swimming pool;

combat sports rings/areas;

cue sports tables;

flying disc sports fields;

running tracks;

ice rink;

snow sports areas;

Olympic sports stadium;

golf field;

gymnastics arena;

motor racing track/circuit;

card games tables/spaces;

board games boards;

table sports tables;

casino games areas;

gambling tables;

performing arts stages;

auction areas; and

dancing ground.