Creation of virtual cameras for viewing real-time events

• US 10,419,788 B2
• Filed: 09/30/2015
• Issued: 09/17/2019
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Nathan Dhilan Arimilli, Nevin Kumar Arimilli, Shivani Kumar Arimilli
• Original Assignee: Nathan Dhilan Arimilli, Nevin Kumar Arimilli, Shivani Kumar Arimilli
• Inventors: Arimilli, Nathan Dhilan, Arimilli, Nevin Kumar, Arimilli, Shivani Kumar
• Primary Examiner(s): Johnson, Motilewa Good
• Application Number:
  US14/871,392
• Publication Number:
  US 20170094326A1
• Time in Patent Office: 1,448 Days
• Field of Search: None
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A video processing system receives a custom view request specifying a virtual camera perspective from which a dynamic scene in a physical environment is to be viewed. The video processing system generates a custom view of the scene from the virtual camera perspective, where generating the custom view includes generating the custom view based on both video content captured by one or more physical cameras and a dynamic three dimensional scene model of the scene residing within data storage. The video processing system records the custom view within on-demand video content and transmits the on-demand video content to a video presentation system.

Claims

1. A method of video processing in a video processing system, the method comprising:

the video processing system maintaining, within data storage, a dynamic three dimensional scene model of a dynamic scene in a real world physical environment, wherein the dynamic three dimensional scene model includes a plurality of dynamic elements individually corresponding to dynamic physical components of the dynamic scene in the physical environment, and wherein the maintaining includes the video processing system applying a rule set to at least a particular dynamic element of the plurality of dynamic elements to determine its dynamically changing position and movement within the scene model;

the video processing system receiving first video content of the dynamic scene captured by one or more physical cameras;

in response to the video processing system receiving a custom view request specifying a virtual camera perspective from which the dynamic scene is to be viewed, the video processing system automatically generating a custom view of the dynamic scene from the virtual camera perspective, wherein the automatically generating includes generating each of a plurality of video frames from the virtual camera perspective by;

initializing a video frame among the plurality of video frames;

filling the at least one coverage gap with second video content from the virtual camera perspective generated from the dynamic three dimensional scene model of the dynamic scene residing within data storage, wherein the second video content includes a surface of the particular dynamic element for which no video content captured by physical cameras is available that shows the surface of the particular dynamic element at its current position, wherein the filling includes rendering the surface based on application of the at least one rule set to the particular dynamic element;

the video processing system recording the custom view within on-demand video content; and

the video processing system transmitting the on-demand video content to a video presentation system.

2. The method of claim 1, wherein generating the custom view includes generating the custom view in substantially real time.

3. The method of claim 1, and further comprising building the three dimensional scene model within the data storage by processing video content captured by the one or more physical cameras.

4. The method of claim 1, wherein the transmitting the on-demand video content comprises transmitting the on-demand video content interspersed within a transmission of a video program depicting the scene.

5. The method of claim 1, wherein:

the three dimensional scene model comprises a plurality of different states each corresponding to a respective one of a plurality of different chronological times;

the custom view request further includes a timing indication; and

the video processing system generates the custom view based upon a particular one of the plurality of different states of the three dimensional scene model corresponding to the timing indication.

6. The method of claim 1, wherein generating each of a plurality of video frames from the virtual camera perspective further includes:

preferentially filling the video frame with the first video content captured by the one or more physical cameras except for the at least one coverage gap.

7. The method of claim 6, wherein preferentially filling the video frame with the first video content comprises preferentially filling the video frame with the first video content without use of the dynamic three dimensional scene model.

8. The method of claim 6, wherein preferentially filling the video frame with the first video content comprises filling the video frame with the first video content after applying a mathematical transformation to the first video content to provide the virtual camera perspective requested by the custom view request.

9. The method of claim 1, wherein generating each of a plurality of video frames from the virtual camera perspective further includes

preferentially filling the video frame with second video content from the virtual camera perspective generated from the dynamic three dimensional scene model rather than the first video content.

10. The method of claim 9, wherein:

the method further comprises testing to determine a variance over time between surfaces predicted by the dynamic three dimensional scene model and the first video content captured by the physical cameras; and

the preferentially filling includes preferentially filling the video frame to represent those surfaces predicted by dynamic three dimensional scene model that have a greater than threshold conformance with (or less than threshold deviation from) those depicted in the first video content.

11. The method of claim 1, wherein the at least one rule set includes at least one of a set consisting of (1) rules specifying physical dimensions of dynamic elements the dynamic scene, (2) physical laws of motion governing movement of the dynamic elements, and (3) biological constraints on ranges of motion of dynamic elements representing living entities.

12. A program product for video processing, the program product comprising:

a data storage device; and

program code stored within the data storage device, wherein the program code, when executed by a processor of a video processing system, causes the video processing system to perform;

the video processing system maintaining, within data storage, a dynamic three dimensional scene model of a dynamic scene in a real world physical environment, wherein the dynamic three dimensional scene model includes a plurality of dynamic elements individually corresponding to dynamic physical components of the dynamic scene in the physical environment, and wherein the maintaining includes the video processing system applying a rule set to at least a particular dynamic element of the plurality of dynamic elements to determine its dynamically changing position and movement within the scene model;

the video processing system receiving first video content of the dynamic scene captured by one or more physical cameras;

in response to the video processing system receiving a custom view request specifying a virtual camera perspective from which the dynamic scene is to be viewed, the video processing system automatically generating a custom view of the dynamic scene from the virtual camera perspective, wherein the automatically generating includes generating each of a plurality of video frames from the virtual camera perspective by;

initializing a video frame among the plurality of video frames;

filling at least one coverage gap for which no video content captured by physical cameras is available with second video content from the virtual camera perspective generated from the dynamic three dimensional scene model of the dynamic scene residing within data storage, wherein the second video content includes a surface of the particular dynamic element for which no video content captured by physical cameras is available that shows the surface of the particular dynamic element at its current position, wherein the filling includes rendering the surface based on application of the at least one rule set to the particular dynamic element;

the video processing system recording the custom view within on-demand video content; and

the video processing system transmitting the on-demand video content to a video presentation system.

13. The program product of claim 12, wherein generating the custom view includes generating the custom view in substantially real time.

14. The program product of claim 12, wherein the program code, when executed, further causes the video processing system to perform building the three dimensional scene model within the data storage by processing video content captured by the one or more physical cameras.

15. The program product of claim 12, wherein the transmitting the on-demand video content comprises transmitting the on-demand video content interspersed within a transmission of a video program depicting the scene.

16. The program product of claim 12, wherein:

the three dimensional scene model comprises a plurality of different states each corresponding to a respective one of a plurality of different chronological times;

the custom view request further includes a timing indication; and

the program code, when executed, further causes the video processing system to generate the custom view based upon a particular one of the plurality of different states of the three dimensional scene model corresponding to the timing indication.

17. The program product of claim 12, wherein generating each of a plurality of video frames from the virtual camera perspective further includes:

preferentially filling the video frame with the first video content captured by the one or more physical cameras except for the at least one coverage gap.

18. The program product of claim 17, wherein preferentially filling the video frame with the first video content comprises preferentially filling the video frame with the first video content without use of the dynamic three dimensional scene model.

19. The program product of claim 17, wherein preferentially filling the video frame with the first video content comprises filling the video frame with the first video content after applying a mathematical transformation to the first video content to provide the virtual camera perspective requested by the custom view request.

20. A video processing system, the video processing system comprising:

a processor;

data storage communicatively coupled to the processor; and

program code stored within the data storage, wherein the program code, when executed by the processor, causes the video processing system to perform;

the video processing system maintaining, within data storage, a dynamic three dimensional scene model of a dynamic scene in a real world physical environment, wherein the dynamic three dimensional scene model includes a plurality of dynamic elements individually corresponding to dynamic physical components of the dynamic scene in the physical environment, and wherein the maintaining includes the video processing system applying a rule set to at least a particular dynamic element of the plurality of dynamic elements to determine its dynamically changing position and movement within the scene model;

the video processing system receiving first video content of the dynamic scene captured by one or more physical cameras;

in response to the video processing system receiving a custom view request specifying a virtual camera perspective from which the dynamic scene is to be viewed, the video processing system automatically generating a custom view of the dynamic scene from the virtual camera perspective, wherein the automatically generating includes generating each of a plurality of video frames from the virtual camera perspective by;

initializing a video frame among the plurality of video frames;

filling the at least one coverage gap with second video content from the virtual camera perspective generated from the dynamic three dimensional scene model of the dynamic scene residing within data storage, wherein the second video content includes a surface of the particular dynamic element for which no video content captured by physical cameras is available that shows the surface of the particular dynamic element at its current position, wherein the filling includes rendering the surface based on application of the at least one rule set to the particular dynamic element;

the video processing system recording the custom view within on-demand video content; and

the video processing system transmitting the on-demand video content to a video presentation system.

21. The video processing system of claim 20, wherein generating the custom view includes generating the custom view in substantially real time.

22. The video processing system of claim 20, wherein the program code, when executed, further causes the video processing system to perform building the three dimensional scene model within the data storage by processing video content captured by the one or more physical cameras.

23. The video processing system of claim 20, wherein the transmitting the on-demand video content comprises transmitting the on-demand video content interspersed within a transmission of a video program depicting the scene.

24. The video processing system of claim 20, wherein:

the three dimensional scene model comprises a plurality of different states each corresponding to a respective one of a plurality of different chronological times;

the custom view request further includes a timing indication; and

the program code, when executed, further causes the video processing system to generate the custom view based upon a particular one of the plurality of different states of the three dimensional scene model corresponding to the timing indication.

25. The video processing system of claim 20, wherein generating each of a plurality of video frames from the virtual camera perspective further includes:

preferentially filling the video frame with the first video content captured by the one or more physical cameras except for the at least one coverage gap.

26. The video processing system of claim 25, wherein preferentially filling the video frame with the first video content comprises preferentially filling the video frame with the first video content without use of the dynamic three dimensional scene model.

27. The video processing system of claim 25, wherein preferentially filling the video frame with the first video content comprises filling the video frame with the first video content after applying a mathematical transformation to the first video content to provide the virtual camera perspective requested by the custom view request.