METHOD AND SYSTEM FOR CREATING THREE-DIMENSIONAL VIEWABLE VIDEO FROM A SINGLE VIDEO STREAM

• US 20130278727A1
• Filed: 11/24/2011
• Published: 10/24/2013
• Est. Priority Date: 11/24/2010
• Status: Abandoned Application

• Current Assignee: Stergen High-Tech Ltd.
• Original Assignee: Stergen High-Tech Ltd.
• Inventors: Tamir, Michael, Birnboim, Michael, Wilf, Itzhak, Sabag, Shai, Littman, Rotem
• Application Number:
  US13/824,470
• Publication Number:
  US 20130278727A1
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Generating 3D representations of a scene represented by a first video stream captured by video cameras. Identifying a transition between cameras, retrieving parameters of a first set of viewing configurations, providing 3D video representations representing the scene at several sets of viewing configurations different from the first set of viewing configurations, and generating an integrated video stream enabling 3D display of the scene by integration of at least two video streams having respective sets of viewing configurations, which are mutually different. Another provided process is for synthesizing an image of an object from a first image, captured by a certain camera at a first viewing configuration. Assigning a 3D model to a portion of a segmented object, calculating a modified image of the portion of the object from a viewing configuration different from the first viewing configuration, and embedding the modified image in a frame for stereoscopy.

Claims

1-39. -39. (canceled)

40. A method for generating a three-dimensional representation of a scene, the scene being represented by a first video stream captured by a certain camera at a first set of viewing configurations, the method comprising:

(a) segmenting a portion of an object from a rest portion of a frame;

(b) processing the segmented portion of the object; and

(c) embedding the processed portion of the object in a frame of an integrated video stream enabling three-dimensional display of the scene.

41. The method of claim 40 wherein the method includes a process for generating one or more three-dimensional representations of a scene, the first video stream captured by two or more video cameras, the process comprising:

(A) identifying a transition between a first camera and a second camera of the two or more video cameras;

(B) retrieving parameters of a certain set of viewing configurations associated with said second camera;

(C) based on at least the retrieved parameters of the certain set of viewing configurations, providing one or more video streams representing the scene at one or more respective sets of viewing configurations different from said certain set of viewing configurations; and

(D) generating an integrated video stream enabling three-dimensional display of the scene by integration of at least two video streams selected from the group of video streams consisting of the first video stream and the one or more provided video streams, the sets of viewing configurations related to the selected video streams being mutually different.

42. The method of claim 40 wherein the method includes a process for synthesizing an image of at least one portion of an object from a first image of the at least one portion of the object, the first image being at least a part of a frame captured by a certain camera at a first viewing configuration, the process comprising:

(A) segmenting at least the at least one portion of the object from a rest portion of a frame;

(B) assigning a three-dimensional model to the at least one portion of the object;

(C) in accordance with said three-dimensional model, calculating a modified image of said at least one portion of the object from a viewing configuration different from said first viewing configuration; and

(D) embedding said modified image in a frame of an integrated video stream enabling three-dimensional display of the scene.

43. The method of claim 42 wherein said three-dimensional model is selected from the group of three-dimensional models consisting of:

(i) a flat surface;

(ii) a cylinder;

(iii) an elongated body having an uniform elliptical cross-section; and

(iv) a three dimensional human shape model.

44. The method of claim 42 wherein said three-dimensional model is a three-dimensional shape model represented as a collection of surface patches.

45. The method of claim 40 wherein the method includes a process for synthesizing an image of a on-field object captured in two or more frames by a certain camera at a first set of viewing configuration of a sports scene, the on-field object being identified in a first certain frame of said two or more frames, the first certain frame being transformed to a first respective frame associated with a different set of viewing configurations, the first viewing configuration and said different set of viewing configurations being suitable for two eye stereoscopy, the process comprising:

(A) identifying the on-field object in a second certain frame of said two or more frames;

(B) transforming at least a portion of said second frame to a second respective frame associated with the different set of viewing configurations; and

(C) embedding said on-field object in said second respective frame such that;

(i) said second certain frame of the two or more frames and said second respective frame fitting two eye stereoscopy; and

(ii) the resulted second respective frame being different from a frame obtained by transforming the whole second frame in accordance with said different set of viewing configurations.

46. The method of claim 45 wherein said identifying the on-field object in a second certain frame is facilitated by at least one method of a group of method consisting of:

(I) footing locations in both said first certain frame and said second certain frame;

(II) object tracking between subsequent frames; and

(III) identifying a feature associated with said first object in both said first certain frame and said second certain frame.

47. The method of claim 45 wherein a disparity value distribution of the embedded on-field object is determined in accordance with a calculated disparity value distribution of an surface underlying said on-field object.

48. The method of claim 47 wherein the disparity value distribution of the embedded said on-field object is perturbed in a series of frames having said different set of viewing configurations around a calculated disparity value distribution of the underlying surface, the perturbations are by a small differential disparity value such as to visually separate said first object from said underlying surface, and the disparity value distribution of the embedded on-field object is modified continuously between a frame having separated on-field objects and a frame where the on-field objects are not separated.

49. The method of claim 40 wherein the method includes a process for presenting a playing object in a sports scene from a first series of images of the sports scene, the first series of images captured at a respective first set of viewing configurations of the sports scene, the process comprising:

(A) identifying the playing object in the first series of images to get identified playing objects in respective images;

(B) segmenting an identified playing object from the rest of a respective image;

(C) calculating at least one depth value associated with the segmented playing object; and

(D) synthesizing a second series of images of the playing object fitting a second set of viewing configurations using for each image of the second series the respective calculated at least one depth value, said second set of viewing configuration being different from the first set of viewing configurations, the different viewing configuration supporting a three-dimensional display of the sports scene.

50. The method of claim 49 wherein said playing object is identified using at least one method of a group of methods consisting of color based detection, shape based detection and motion based detection.

51. The method of claim 49 wherein the process includes transforming a first representation of an air trajectory of said playing object as captured in the first series of images to a second representation of said air trajectory in accordance with the second set of the viewing configurations.

52. The method of claim 51 wherein the process includes:

(i) based on said first representation of said air trajectory, determining world representation of a plane disposed vertical to an horizontal plane and hosting said air trajectory;

(ii) calculating world representation of said air trajectory, based on said world representation of the plane; and

(iii) calculating disparity values along said air trajectory in accordance with the second set of viewing configurations based on the calculated world representation of said air trajectory.

53. The method of claim 52 wherein the process includes:

(iv) determining on-field endpoints of said air trajectory.

54. The method of claim 49 wherein the process includes at least one step of:

(i) measuring a size of said playing object in the first representation of the playing object;

(ii) determining the depth and the disparity of said object based on its size;

(iii) measuring said size of the playing object in perpendicular to a motion vector associated with said air trajectory; and

(iv) smoothing the measurements of said size of said playing object based on a monotonous change along the air trajectory.

55. The method of claim 49 wherein a graphic element is embedded within the first and second series of images by:

(i) selecting an object of interest in the first series of images; and

(ii) rendering the graphic element in accordance with a depth value of said object of interest within the first and second series of images.

56. The method of claim 40 wherein the method includes a process for presenting a static object in a sports scene based on a first image of the sports scene captured at a first viewing configuration, the static object residing in part on a plane different from the field surface, the process comprising:

(A) based on a model of the static object and its position relative to other static object, transforming a first representation of the static object in the first series of images to a second representation fitting a second viewing configuration different from said first viewing configuration; and

(B) identifying a part of said static object as being absent in said first representation of said static object, and as being present in said second representation of said static object.

57. The method of claim 56 wherein the method further includes:

(C) in-painting said part of said static object.

58. The method of claim 56 wherein the method includes in-painting said part based on at least one source of a group of sources consisting of:

(i) an image captured at a viewing configuration different from the first viewing configuration;

(ii) a prior model of the static object; and

(iii) a similar object located in a other field location.

59. The method of claim 56 wherein the static object is selected from a group of objects consisting of a goal post, a tennis net, a basket poll, a billboard, a gallery, a balcony and a tribune.