SYSTEMS AND METHODS FOR THE AUTONOMOUS PRODUCTION OF VIDEOS FROM MULTI-SENSORED DATA

• US 20120057852A1
• Filed: 05/07/2010
• Published: 03/08/2012
• Est. Priority Date: 05/07/2009
• Status: Active Grant

• Current Assignee: Keemotion S.A.
• Original Assignee: Université catholique de Louvain
• Inventors: Devleeschouwer, Christophe, Chen, Fan
• Granted Patent: US 8,854,457 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/268 Signal distribution or switching

Abstract

An autonomous computer based method and system is described for personalized production of videos such as team sport videos such as basketball videos from multi-sensored data under limited display resolution. Embodiments of the present invention relate to the selection of a view to display from among the multiple video streams captured by the camera network. Technical solutions are provided to provide perceptual comfort as well as an efficient integration of contextual information, which is implemented, for example, by smoothing generated viewpoint/camera sequences to alleviate flickering visual artefacts and discontinuous story-telling artefacts. A design and implementation of the viewpoint selection process is disclosed that has been verified by experiments, which shows that the method and system of the present invention efficiently distribute the processing load across cameras, and effectively selects viewpoints that cover the team action at hand while avoiding major perceptual artefacts.

Claims

1-43. -43. (canceled)

44. A computer based method for autonomous production of an edited video from multiple video streams captured by a plurality of cameras distributed around a scene of interest, the method comprising:

detecting objects in the images of the video streams,

selecting for each camera, a field of view based on joint processing of positions of multiple objects that have been detected,

building the edited video by selecting and concatenating video segments provided by one or more individual cameras, wherein the building is done in a way that maximizes completeness and closeness metrics along the time, while smoothing out the sequence of rendering parameters associated to concatenated segments, wherein completeness measures to which extent objects-of-interest are included and visible within the displayed viewpoint, and closeness refers to the fineness of details when rendering the objects-of-interest, balance completeness and closeness, as a function of individual user preferences.

45. The method of claim 44, further comprising rating the viewpoint selected in each camera view according to the quality of its completeness/closeness trade-off, and to its degree of occlusions.

46. The method of claim 44, further comprising selecting the optimal field of view in each camera, at a given time instant, wherein a field of view v_k in the k^th camera view is defined by the size S_k and the center c_k of the window that is cropped in the k^th view for actual display and is selected to include the objects of interest and to provide a high resolution description of the objects, and an optimal field of view v_k* is selected to maximize a weighted sum of object interests as follows

47. The method of claim 46, wherein α

( . . . ) decreases with Sic and the function α

( . . . ) is equal to one when S_k<

u_res, and decrease afterwards, and

wherein α

( . . . ) is defined optionally by;

48. The method of claim 45, wherein the highest rate correspond to a view that makes most object of interest visible, and is close to the action.

49. The method of claim 45, wherein, given the interest I_n of each player, the rate I_k(v_k, u) associated to the k^th camera view is defined as follows:

50. The method of claim 49, wherein β

(.) is defined as

β

_k(S, u)=u_k·

α

(S, u),

where u_k denotes the weight assigned to the k^th camera, and α

(S, u) is defined as in claim 47.

51. The method of claim 44 further comprising smoothing the sequence of camera indices and corresponding viewpoint parameters, wherein the smoothing process is for example implemented based on two Markov Random Fields, linear or non-linear low-pass filtering mechanism, or via a graph model formalism, solved based on conventional Viterbi algorithm.

63. A non-transitory machine readable signal storage medium storing a computer program product that comprises code segments which when executed on a processing engine execute the method of claim 44 or implement the system according to claim 52.

52. Computer based system comprising a processing engine and memory for autonomous production of an edited video from multiple video streams captured by a plurality of cameras distributed around a scene of interest, the system comprising:

detector for detecting objects in the images of the video streams,

first means for selecting one or more camera viewpoints based on joint processing of positions of multiple objects that have been detected,

second means for selecting rendering parameters that maximize and smooth out closeness and completeness metrics by concatenating segments in the video streams provided by one or more individual cameras, wherein the building is done in a way that maximizes completeness and closeness metrics along the time, while smoothing out the sequence of rendering parameters associated to concatenated segments, wherein completeness measures to which extent objects-of-interest are included and visible within the displayed viewpoint, and

closeness refers to the fineness of details when rendering the objects-of-interest, balance completeness and closeness, as a function of individual user preferences.

53. The system of claim 52, further comprising third means for selecting camera and image parameter variations for the camera view that render action as a function of time for a set of joint closeness and completeness metrics,

the third means being optionally for selecting camera and image parameter variations is adapted to crop in the camera view of a static camera or to control the control parameters of a dynamic camera.

54. The system of claim 52 further comprising means for mapping images from all views of all cameras to the same absolute temporal coordinates based a common unique temporal reference for all camera views.

55. The system of claim 52 further comprising fourth means for selecting the variations of parameters that optimize the trade-off between completeness and closeness at each time instant, and for each camera view,

wherein the completeness/closeness trade-off is optionally measured as a function of the user preferences.

56. The system of claim 52, further comprising means for rating the viewpoint selected in each camera view according to the quality of its completeness/closeness trade-off, and to its degree of occlusions.

57. The system of claim 56, further comprising means for computing the parameters of an optimal virtual camera that pans, zooms and switches across views to preserve high ratings of selected viewpoints while minimizing the amount of virtual camera movements, for the temporal segment at hand,

58. The system of claim 56, further comprising fifth means for selecting the optimal viewpoint in each camera view, at a given time instant,

wherein the fifth means for selecting the optimal viewpoint is adapted, for a viewpoint v_k in the k^th camera view is defined by the size S_k and the center c_k of the window that is cropped in the k^th view for actual display and is selected to include the objects of interest and to provide a high resolution, is adapted to select a description of the objects and an optimal viewpoint v_k* to maximize a weighted sum of object interests as follows

59. The system of claim 58, wherein α

( . . . ) decreases with Sk and the function α

( . . . ) is equal to one when S_k<

u_res, and decrease afterwards,

wherein α

( . . . ) is optionally defined by;

60. The system of claim 57, further comprising sixth means for selecting the camera at a given time instant that makes most object of interest visible, and is close to the action, whereby an optimal camera index k* is selected according to an equation that is similar or equivalent to:

61. The system of claim 60, wherein β

_k(.) is defined as

β

_k(S, u)=u_k·

α

(S, u),

where u_k denotes the weight assigned to the k^th camera, and α

(S, u) is defined as in claim 59.

62. The system of claim 59 further comprising means for smoothing the sequence of camera indices and corresponding viewpoint parameters, wherein the means for smoothing is adapted to smooth based on two Markov Random Fields, by a linear or non-linear low-pass filtering mechanism, by a graph model formalism, solved based on conventional Viterbi algorithm.