Embedding a real time video into a virtual environment

• US 8,339,418 B1
• Filed: 06/25/2007
• Issued: 12/25/2012
• Est. Priority Date: 06/25/2007
• Status: Active Grant

• Current Assignee: Pacific Arts Corporation
• Original Assignee: Pacific Arts Corporation
• Inventors: Nesmith, Michael
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US11/823,100
• Time in Patent Office: 2,010 Days
• Field of Search: 345/633
• US Class Current: 345/633
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

Generating combined content suitable for including in a virtual environment is disclosed. A background associated with the virtual environment is inputted. A real time video inputted. The background is combined with the real time video to generate combined content. The combined content is suitable for including in the virtual environment.

Claims

1. A method of generating combined content suitable for including in a virtual environment, comprising:

inputting a background associated with the virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by a user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

capturing a live subject, including gestures and movements of the live subject, in a real time video;

using a processor to combine the background with the real time video to generate combined content suitable for including in the virtual environment;

placing at least a portion of the combined content at a location and an orientation within the computer generated three-dimensional environment in a manner that allows the live subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject;

receiving a communication directed to the live subject from the user, wherein the communication includes communication that is communicated through the avatar; and

providing a screen for the live subject captured in the real time video to view the communication as the real time video is being captured, wherein the live subject is enabled to communicate back to the user in response to the communication via the real time video capturing the gestures and movements of the live subject that appears as the virtual representation, and to view simultaneously the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment.

2. The method as recited in claim 1, wherein the real time video includes the subject against a chroma key background.

3. The method as recited in claim 2, wherein combining the background with the real time video includes replacing a chroma key background portion of the real time video with at least a portion of the background.

4. The method as recited in claim 3, wherein the chroma key background portion of the real time video is detected at least in part by determining which portion of the real time video is associated with a specified range of chroma key colors.

5. The method as recited in claim 1, wherein each of one or more subjects of the real time video appear to the user of the virtual environment as a separate virtual object existing in the virtual environment.

6. The method as recited in claim 1, wherein the virtual environment user is able to interact live with the life subject of the real time video.

7. The method as recited in claim 1, wherein the real time video includes a musical performance.

8. The method as recited in claim 1, wherein the combined content is encoded in a format suitable for streaming to one or more clients rendering the virtual environment.

9. The method as recited in claim 1, wherein the real time video includes a chroma key colored object occluding the subject of the real time video.

10. The method as recited in claim 9, wherein the occluding object in the live video is processed to be replaced with a replacement content.

11. The method as recited in claim 1, wherein the subject in the real time video is lit by a real life light source configured in a manner that substantially reproduces lighting effects of a virtual environment light source on a virtual environment location associated with a virtual environment placement location of the combined content.

12. The method as recited in claim 1, wherein the background is at least in part obtained from the virtual environment.

13. The method as recited in claim 1, wherein the background includes as least a portion of an image captured from one or more of the following:

the virtual environment and a virtual object to be included in the virtual environment.

14. The method as recited in claim 1, wherein the background includes one or more of the following:

an image content, a video content, an animation content, a text content, and a virtual environment content.

15. The method as recited in claim 1, wherein combining the background image with the real time video includes adjusting on the real live video or the background one or more of the following:

a scale, a color, and a cropping.

16. The method as recited in claim 1, wherein combining the background image with the real time video includes compositing the background image with the real time video.

17. The method as recited in claim 1, wherein the combined content is associated with a virtual environment object.

18. The method as recited in claim 17, wherein associating the combined content with the virtual environment object includes adjusting a scale, a position, a color, or a cropping of the combined content with respect to the virtual environment object.

19. The method as recited in claim 17, wherein associating the combined content with the virtual environment object includes adjusting the combined content to substantially match the background of the combined content with at least a portion of the virtual environment object.

20. The method as recited in claim 1, wherein at least a portion of an edge of the combined content is processed to better blend in with the virtual environment or a virtual environment object.

21. The method as recited in claim 20, wherein processing the portion of the edge includes blurring the edge.

22. The method as recited in claim 1, wherein the subject of the real time video appears to be immersed in the virtual environment at least in part due to one or more of the following:

a determined camera angle used to produce the live video, a determined magnification used to produce the live video, a virtual environment location associated at least a portion of the combined content, and an arrangement of one or more objects in the virtual environment.

23. The method as recited in claim 1, wherein a communication between the avatar and the subject includes one or more of the following:

a gesture by the avatar, a gesture by the subject, a text based chat, and a voice based chat.

24. The method as recited in claim 1, wherein the screen displays one or more of the following:

the virtual environment, a virtual environment object, a representation of the virtual environment user, and a chat session.

25. A system for generating combined content suitable for including in a virtual environment, comprising:

a processor configured to;

receive a background associated with the virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by a user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

capture a live subject, including gestures and movements of the live subject, in a real time video;

combine the background with the real time video to generate combined content suitable for including in the virtual environment;

place at least a portion of the combined content at a location and an orientation within the computer generated three-dimensional environment in a manner that allows the live subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject;

receive a communication directed to the live subject from the user, wherein the communication includes communication that is communicated through the avatar; and

provide a screen for the live subject captured in the real time video to view the communication as the real time video is being captured, wherein the live subject is enabled to communicate back to the user in response to the communication via the real time video capturing the gestures and movements of the live subject that appears as the virtual representation, and to view simultaneously the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions.

26. A computer program product for generating combined content suitable for including in a virtual environment, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:

inputting a background associated with the virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by a user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

capturing a live subject, including gestures and movements of the live subject, in a real time video;

combining the background with the real time video to generate combined content suitable for including in the virtual environment;

placing at least a portion of the combined content at a location and an orientation within the computer generated three-dimensional environment in a manner that allows the live subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject;

receiving a communication directed to the live subject from the user, wherein the communication includes communication that is communicated through the avatar; and

providing a screen for the live subject captured in the real time video to view the communication as the real time video is being captured, wherein the live subject is enabled to communicate back to the user in response to the communication via the real time video capturing the gestures and movements of the live subject that appears as the virtual representation, and to view simultaneously the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment.

27. A method of providing communication between a user and an embedded video, comprising:

capturing a real time video to be at least in part embedded substantially in real time into a virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by the user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

placing at least a portion of the real time video at a location and an orientation within the computer generated three-dimensional environment in a manner that allows a subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject;

providing a screen for the subject captured in the real time video to view as the real time video is being captured the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment;

receiving a communication directed to the embedded video from the user viewing in the virtual environment the embedded video, wherein the communication is sent via a communication channel provided in the virtual environment, and wherein the communication includes communication that is communicated through the avatar; and

using a processor to forward the communication to one or more subjects of the embedded video, wherein the one or more subjects are enabled to communicate back to the user in response to the communication via the real time video that appears as the virtual representation.

28. The method as recited in claim 27, wherein the user sends the communication directed to the embedded video in an attempt to communicate with the one or more subjects of the embedded video.

29. The method as recited in claim 27, wherein the communication includes a real time communication.

30. The method as recited in claim 27, wherein the communication includes one or more of the following:

a text communication, a voice communication, and a gesture communication.

31. The method as recited in claim 27, wherein forwarding the communication includes displaying at least a portion of the communication on the screen viewable by any subject being filmed for the real time video.

32. The method as recited in claim 31, wherein the screen displays a field of view of the virtual environment, and the field of view includes one or more representations of the user and any other users in the field of view.

33. The method as recited in claim 31, wherein the screen displays a chat session.

34. A system for providing communication between a user and an embedded video, comprising:

a processor configured to;

capture a real time video to be at least in part embedded substantially in real time into a virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by the user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

place at least a portion of the real time video at a location and an orientation within the computer generated three-dimensional environment in a manner that allows a subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject; and

provide a screen for the subject captured in the real time video to view as the real time video is being captured the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment; and

a communication interface configured to receive a communication directed to the embedded video from the user viewing in the virtual environment the embedded video, wherein the communication is sent via a communication channel provided in the virtual environment, and wherein the communication includes communication that is communicated through the avatar;

wherein the communication is forwarded to one or more subjects of the embedded video, and wherein the one or more subjects are enabled to communicate back to the user in response to the communication via the real time video that appears as the virtual representation.

35. A computer program product for providing communication between a user and an embedded video, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:

capturing a real time video to be at least in part embedded substantially in real time into a virtual environment, wherein the virtual environment includes a computer generated three-dimensional graphical environment that can be navigated by the user, wherein navigating in the virtual environment by the user includes navigating as an avatar in the virtual environment;

placing at least a portion of the real time video at a location and an orientation within the computer generated three-dimensional environment in a manner that allows a subject captured in the real time video to appear immersed in the computer generated three-dimensional graphical environment as a virtual representation of the subject;

providing a screen for the subject captured in the real time video to view as the real time video is being captured the computer generated three-dimensional graphical environment from a point-of-view of the virtual representation of the subject appearing immersed in the computer generated three-dimensional graphical environment, including viewing the avatar navigating in the virtual environment;

receiving a communication directed to the embedded video from the user viewing in the virtual environment the embedded video, wherein the communication is sent via a communication channel provided in the virtual environment; and

forwarding the communication to one or more subjects of the embedded video, wherein the one or more subjects are enabled to communicate back to the user in response to the communication via the real time video that appears as the virtual representation, and wherein the communication includes communication that is communicated through the avatar.