SYSTEM FOR AND METHOD OF GENERATING USER-SELECTABLE NOVEL VIEWS ON A VIEWING DEVICE

• US 20160182894A1
• Filed: 04/01/2015
• Published: 06/23/2016
• Est. Priority Date: 04/30/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Replay Technologies Inc. (Intel Corporation)
• Inventors: Gilat, Adi, Haimovitch-Yogev, Oren, Shapira, Matteo, Shapira, Aviv, Prilusky, Diego, Ben Zvi, Yaniv
• Granted Patent: US 10,063,851 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G06T 11/003 Reconstruction from projections, e.g. tomography
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2207/20152 Watershed segmentation
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of generating user-selectable novel views of an event on a viewing device includes reconstructing by a server system for each camera image data into at least one foreground model for the respective camera and an environment model for the respective camera; joining by the server system the foreground model for each camera to create a visual atlas of all foreground models; creating by the server system foreground mapping data for foreground image data in the visual atlas; creating by the server system environment mapping data for environment image data in each respective environment model; transmitting by the server system each compressed data in a sequence it was compressed; receiving by the viewing device all compressed data; uncompressing by the viewing device all compressed data; selecting by a user the novel view; and rendering by the viewing device each novel view.

Claims

1. A method of generating user-selectable novel views of an event on a viewing device, the method comprising the steps of:

(a) reconstructing by a server system for each camera of a plurality of cameras image data into at least one foreground model for the respective camera and at least one environment model for the respective camera;

(b) joining by the server system the at least one foreground model for each camera to create a visual atlas of all foreground models;

(c) creating by the server system foreground mapping data for foreground image data in the visual atlas to a 3D coordinate in a 3D data representation;

(d) projecting by the server system environment image data of all cameras for each camera onto each respective environment model;

(e) creating by the server system environment mapping data for environment image data in each respective environment model to a 3D coordinate in a 3D data representation;

(f) saving by the server system for each camera the respective environment model as an environment image;

(g) compressing by the server system the foreground mapping data, the environment mapping data, and a depth map;

(h) inserting by the server system the visual atlas into each respective environment model as a new image frame in an image sequence by projecting background rasters on each respective environment model;

(i) compressing by the server system the visual atlas;

(j) compressing by the server system the respective environment images and the respective environment models;

(k) transmitting by the server system each compressed data in a sequence it was compressed;

(l) receiving by the viewing device all compressed data;

(m) uncompressing by the viewing device all compressed data;

(n) selecting by a user on the viewing device the novel view; and

(o) rendering by the viewing device the respective environment images onto the respective environment models for each novel view.

2. The method of claim 1, further comprising before step (a), the steps of

imaging with the plurality of cameras the event in a plurality of image frames, the event comprising a foreground and an environment;

extracting by the server system image data from the plurality of image frames;

3. The method of claim 2, further comprising the step of

saving the image data as the 3D data representation.

4. The method of claim 1, further comprising after step (e) the step of

calculating by the server system a median model value for each respective environment model.

5. The method of claim 4, further comprising after step (f) the step of

calculating by the server system a median image value for each respective environment image;

determining a difference between the median image value and the median model value.

6. The method of claim 1, further comprising the step of

inserting by the server system a brightness image into the environment image.

7. The method of claim 6, further comprising the step of

compressing by the server system the brightness image.

8. The method of claim 1, wherein step (h) requires using an exponential matrix to decide the weight of a projection.

9. The method of claim 1, wherein step (h) requires looping through each camera and referencing image data in the visual atlas.

10. An interactive-player system for generating user-selectable novel views of an event on a viewing device, the multi-view reconstruction system comprising:

a server system and a viewing device;

the server system

(a) reconstructing for each camera of a plurality of cameras image data into at least one foreground model for the respective camera and at least one environment model for the respective camera;

(b) joining the at least one foreground model for each camera to create a visual atlas of all foreground models;

(c) creating foreground mapping data for foreground image data in the visual atlas to a 3D coordinate in a 3D data representation;

(d) projecting environment image data of all cameras for each camera onto each respective environment model;

(e) creating environment mapping data for environment image data in each respective environment model to a 3D coordinate in a 3D data representation;

(f) saving for each camera the respective environment model as an environment image;

(g) compressing the foreground mapping data, the environment mapping data, and a depth map;

(h) inserting the visual atlas into each respective environment model as a new image frame in an image sequence by projecting background rasters on each respective environment model;

(i) compressing the visual atlas;

(j) compressing the respective environment images and the respective environment models;

(k) transmitting each compressed data in a sequence it was compressed;

the viewing device

receiving all compressed data;

uncompressing all compressed data;

selecting by a user on the viewing device the novel view; and

rendering by the viewing device the respective environment images onto the respective environment models for each novel view.

11. The system of claim 10, wherein the viewing device comprises a path selection module for selecting a path to display a plurality of novel views.

12. The system of claim 10, wherein the server side comprises a configuration engine for executing (a)-(k).

13. The system of claim 10, further comprising

the plurality of cameras for imaging the event in a plurality of image frames, the event comprising a foreground and an environment;

an FES module for separating the foreground from the environment in the image frames.

14. The system of claim 10, further comprising a plurality of viewing devices and a social messaging system for communicating among a plurality of users of viewing devices.