A METHOD FOR GENERATING AN IMMERSIVE VIDEO OF A PLURALITY OF PERSONS

• US 20160150187A1
• Filed: 06/20/2014
• Published: 05/26/2016
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Provenance Asset Group LLC
• Inventors: Fadili, Moulay, Abou-Chakra, Rabih
• Granted Patent: US 9,729,825 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A method for generating an immersive video of a plurality of persons in a computer server, the method comprising:—receiving a plurality of video streams on a plurality of video channels from a plurality of client devices, each video stream including a silhouette of a person;—extracting the person'"'"'s silhouette from each incoming video stream to generate a filtered video stream representing only the silhouette of the person;—generating a video stream of a virtual scene comprising a plurality of—the silhouettes extracted from the plurality of filtered video streams the method further comprising receiving silhouette data carried on a metadata channel from a client device in addition to the video stream, the silhouette data representing a position of a face of the person within a frame of the video stream;—analyzing the silhouette data performing the silhouette extraction as a function of the silhouette data analyzed.

Claims

1. A method for generating an immersive video of a plurality of persons in a computer server, the method comprising:

receiving a plurality of video streams on a plurality of video channels from a plurality of client devices, each video stream including a silhouette of a person;

extracting the person'"'"'s silhouette from each incoming video stream to generate a filtered video stream representing only the silhouette of the person;

generating a video stream of a virtual scene comprising a plurality of the silhouettes extracted from the plurality of filtered video streams;

the method further comprising receiving silhouette data carried on a metadata channel from a client device in addition to the video stream, the silhouette data representing a position of a face of the person within a frame of the video stream, the silhouette data being selected in the group consisting of coarse silhouette information representing coordinates of a characteristic point of the face of the person and fine silhouette information representing a mask corresponding to the person'"'"'s silhouette;

analyzing the silhouette data;

performing the silhouette extraction as a function of the silhouette data analyzed.

2. A method according to claim 1, further including:

listening for CPU allocation requests from the client devices, wherein a CPU allocation request defines a type of the silhouette data to be sent by the client device;

determining an available CPU capacity of the computer server in order to reply to a requesting first client device according to the CPU load.

3. A method according to claim 2, further including accepting the allocation request from the first client device, allocating CPU capacity to the first client device and registering an ID of the first client device.

4. A method according to claim 3 further including:

sending a message to the first client device to ask if the first client device has CPU capacity to generate fine silhouette information;

unregistering said registered first client device in response to receiving a positive reply from the first client device;

allocating CPU capacity freed from the first client device to perform the silhouette extraction for a requesting second client device;

Registering the requesting second client device.

5. A method according to claim 3 further including:

sending a message to the first client device to ask if the first client device has CPU capacity to generate coarse silhouette information;

disconnecting the extraction of coarse information for the first client device in response to receiving a positive reply from the first client device;

allocating CPU capacity freed from the first client device to perform the silhouette extraction for a requesting second client device;

Registering the requesting second client device.

6. A method according to claim 1, wherein the coarse silhouette information comprises coordinates of a face center, a height and a width of the face.

7. A method according to claim 1, wherein the received silhouette data comprises coarse silhouette information, the method further comprising transmitting the coarse silhouette information to a silhouette extraction processing block which computes the filtered video stream of the silhouette as function of the coarse silhouette information and the incoming video stream.

8. A method according to claim 7, wherein the silhouette extraction processing block applies a prediction algorithm to perform the silhouette extraction according to coarse silhouette information.

9. A method according to claim 1, wherein the received silhouette data comprises the fine silhouette information representing the mask, the method further including comparing the fine silhouette information to a threshold to detect the mask and applying the mask to the corresponding frame of the incoming video stream to obtain the filtered video stream representing only the silhouette of the person.

10. A method according to claim 9, wherein the fine silhouette information is a picture composed from pixels of a first color and/or pixels of a second color, the method further including:

comparing a value for each pixel in the picture to a mean value;

attributing a standard value to each pixel as a function of a result of the comparison, wherein the standard value is a first standard value attributed if the pixel value is lower than the mean value and a second standard value attributed if the pixel value is higher than the mean value.

11. A method of generating a video stream in a client device including:

capturing a video stream from a capture device;

performing a silhouette extraction process from the captured video stream to generate silhouette data;

encoding the silhouette data generated by the silhouette extraction process;

sending the video stream in a four channels video format including three channels of raw video data using a color space for the video stream and one metadata channel for the silhouette data generated by the silhouette extraction process.

12. Method according to claim 11, wherein the metadata channel is selected in the group consisting of an alpha channel, a RTP extension channel and a RTP channel.

13. A video processing Server including:

a video decoder block able to receive video streams from a plurality of client devices,

a user processing detection block able to detect a metadata channel in an incoming video stream;

a first silhouette extraction sub-processing block able to perform a silhouette extraction process in the incoming video stream to generate coarse silhouette information;

a second silhouette extraction sub-processing block able to perform a silhouette extraction process to generate fine silhouette information as a function of the coarse silhouette information and the incoming video stream;

a immersive rendering block able to mix a plurality of silhouette video streams to generate a virtual scene,

a video encoder block able to encode and send a video stream comprising the virtual scene to a client device.

14. The Video processing server according to claim 13 further including a Resource manager module including:

a listener module adapted to listen for CPU request for extraction step,

a CPU capacity determination module able to determine a CPU capacity available in the video processing server;

a communication module able to send messages to registered client devices for which the video processing server performs silhouette extraction, and

A memory able to register a client device.