Method and system for real-time rendering within a gaming environment

• US 20050085296A1
• Filed: 10/17/2003
• Published: 04/21/2005
• Est. Priority Date: 10/17/2003
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Malzbender, Thomas, Gelb, Daniel G.
• Granted Patent: US 7,285,047 B2
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: G06T 15/02 Non-photorealistic rendering

Abstract

A method and system for real-time rendering within a gaming environment. Specifically, one embodiment of the present invention discloses a method of rendering a local participant within an interactive gaming environment. The method begins by capturing a plurality of real-time video streams of a local participant from a plurality of camera viewpoints. From the plurality of video streams, a new view synthesis technique is applied to generate a rendering of the local participant. The rendering is generated from a perspective of a remote participant located remotely in the gaming environment. The rendering is then sent to the remote participant for viewing.

Claims

1. A method of real-time rendering in a gaming environment to create an interactive experience, comprising:

capturing a plurality of real-time video streams of a local participant from a plurality of camera viewpoints;

generating real-time renderings of said local participant based on said plurality of real-time video streams by applying a new view synthesis technique, said real-time renderings taken from a perspective of a remote participant located remotely in said gaming environment; and

sending said real-time renderings to said remote participant for viewing within said gaming environment.

2. The method of claim 1, further comprising:

incorporating said real-time renderings comprising real-time photo-realistic features of said local participant into a graphically generated avatar, said graphically generated avatar representing said local participant in said gaming environment; and

sending said graphically generated avatar incorporating said real-time renderings to said local participant for viewing within said gaming environment.

3. The method as described in claim 1, wherein said generating real-time renderings of said local participant further comprises:

generating image based renderings of said local participant as said real-time renderings, wherein said new-view synthesis technique comprises an image-based visual hull technique.

4. The method as described in claim 1, wherein said generating real-time renderings of said local participant further comprises:

generating a three-dimensional geometric model of said local participant from said plurality of real-time video streams as said real-time renderings.

5. The method as described in claim 1, wherein said generating real-time renderings of said local participant further comprises:

rendering said real-time renderings from a second location of said remote participant with respect to a first location of said local participant within a coordinate space of said gaming environment.

6. The method as described in claim 5, further comprising:

enabling a change in at least one of said first and second locations by allowing said local participant and said remote participant to navigate through said coordinate space; and

modifying said real-time renderings to reflect said change in at least one of said first and second locations.

7. The method as described in claim 1, further comprising:

blending said real-time renderings within a synthetic rendering of said gaming environment; and

displaying said real-time renderings that is blended within said gaming environment at a display viewable by said second participant to enable real-time video communication for interactive gaming between said local participant and said remote participant within said gaming environment.

8. The method as described in claim 1, wherein said gaming environment comprises a three dimensional gaming environment.

9. The method as described in claim 1, further comprising:

personifying said graphically generated avatar to said local participant by illustrating real-time emotion of said local participant through said video image stream comprising representative said real-time photo-realistic features of said local participant.

10. The method as described in claim 1, further comprising:

generating another real-time renderings of said local participant based on said plurality of real-time video streams by applying said new view synthesis technique, said another real-time renderings taken from a perspective of another participant located remotely in said gaming environment; and

sending said another real-time renderings to said another participant for viewing within said gaming environment.

11. A method of real-time rendering in a gaming environment to create an interactive experience, comprising:

capturing a plurality of real-time video streams of a local participant from a plurality of camera viewpoints;

generating a video image stream in real-time of said local participant based on said plurality of real-time video streams by applying a new view synthesis technique, said video image stream rendered from a perspective of a remote participant located remotely in said gaming environment; and

incorporating said video image stream comprising real-time photo-realistic features of said local participant into a graphically generated avatar, said graphically generated avatar representing said local participant in said gaming environment.

12. The method as described in claim 11, wherein said generating said video image stream further comprises:

rendering said video image stream from a second location of said remote participant with respect to a first location of said local participant within a coordinate space of said gaming environment.

13. The method as described in claim 12, further comprising:

enabling a change in at least one of said first and second locations by allowing said local participant and said remote participant to navigate through said coordinate space; and

modifying said video image stream to reflect said change in at least one of said first and second locations.

14. The method as described in claim 11, further comprising:

blending said graphically generated avatar incorporating said video image stream within a synthetic rendering of said gaming environment; and

displaying said graphically generated avatar incorporating said video image stream within said gaming environment at a display viewable by said remote participant to enable real-time video communication for interactive gaming between said local participant and said remote participant within said gaming environment.

15. The method as described in claim 11, further comprising:

personifying said graphically generated avatar to said local participant by illustrating real-time emotion of said local participant through said video image stream comprising representative said real-time photo-realistic features of said local participant.

16. The method as described in claim 11, wherein said new-view synthesis technique comprises an image-based visual hull technique.

17. The method as described in claim 11, wherein said generating said video image stream further comprises:

generating a three dimensional geometric model of said local participant based on said plurality of real time video streams by applying said new view synthesis technique; and

generating said video image stream of said local participant from said three dimensional geometric model.

18. The method as described in claim 11, further comprising:

generating a real-time audio output from audio of said local participant to enable real-time audio communication in said gaming environment between said local participant and said remote participant.

19. The method as described in claim 11, further comprising:

varying a total of said plurality of camera viewpoints when capturing said plurality of real-time video streams of said local participant to vary image quality in said plurality of video image streams.

20. The method as described in claim 11, further comprising:

performing non-photorealistic processing of said plurality of video image streams resulting in a non-photorealistic cartoon rendering of said plurality of video image streams.

21. A gaming station comprising:

a plurality of video cameras for recording real-time video streams of a local participant from multiple sample viewpoints surrounding said local participant;

a new view synthesis module for generating a plurality of output real-time renderings of said local participant, by applying a new view synthesis technique to said real-time video streams, wherein said output real-time renderings are taken from a plurality of locations within a coordinate space of a three dimensional gaming environment.

22. The gaming station of claim 21, further comprising:

an interface module for incorporating a plurality of input video image streams comprising real-time photo-realistic features of a plurality of observing participants into a plurality of graphically generated avatars representing said plurality of observing participants, and blending said plurality of graphically generated avatars within a synthetic rendering of said three dimensional gaming environment.

23. The gaming station as described in claim 22, wherein said interface module receives a plurality of input audio streams in real-time associated with said plurality of observing participants for mixing into a single audio stream that is broadcast through a local speaker.

24. The gaming station as described in claim 22, further comprising:

a display for viewing said plurality of graphically generated avatars representing said plurality of observing participants to enable real-time view communication between said local participant and said plurality of observing participants within said three dimensional gaming environment.

25. The gaming station as described in claim 21, further comprising a contour extraction module for extracting said local participant from a physical background.

26. The gaming station as described in claim 21, further comprising a means for capturing an audio stream in real-time of said local participant.

27. A computer system comprising:

a processor; and

a computer readable memory coupled to said processor and containing program instructions that, when executed, implement a method of real-time rendering in a gaming environment to create an interactive experience, comprising;

capturing a plurality of real-time video streams of a local participant from a plurality of camera viewpoints;

generating real-time renderings of said local participant based on said plurality of real-time video streams by applying a new view synthesis technique, said real-time renderings taken from a perspective of a remote participant located remotely in said gaming environment; and

sending said real-time renderings to said remote participant for viewing within said gaming environment.

28. The computer system of claim 27, wherein said method further comprises:

incorporating said real-time renderings comprising real-time photo-realistic features of said local participant into a graphically generated avatar, said graphically generated avatar representing said local participant in said gaming environment; and

sending said graphically generated avatar incorporating said real-time renderings to said local participant for viewing within said gaming environment.

29. The computer system as described in claim 27, wherein said generating real-time renderings of said local participant in said method further comprises:

generating image based renderings of said local participant as said real-time renderings, wherein said new-view synthesis technique comprises an image-based visual hull technique.

30. The computer system as described in claim 27, wherein said generating real-time renderings of said local participant in said method further comprises:

generating a three-dimensional geometric model of said local participant from said plurality of real-time video streams as said real-time renderings.

31. The computer system as described in claim 27, wherein said generating real-time renderings of said local participant in said method further comprises:

rendering said real-time renderings from a second location of said remote participant with respect to a first location of said local participant within a coordinate space of said gaming environment.

32. The computer system as described in claim 31, wherein said method further comprises:

enabling a change in at least one of said first and second locations by allowing said local participant and said remote participant to navigate through said coordinate space; and

modifying said real-time renderings to reflect said change in at least one of said first and second locations.

33. The computer system as described in claim 27, wherein said method further comprises:

blending said real-time renderings within a synthetic rendering of said gaming environment; and

displaying said real-time renderings that is blended within said gaming environment at a display viewable by said second participant to enable real-time video communication for interactive gaming between said local participant and said remote participant within said gaming environment.

34. The computer system as described in claim 27, wherein said gaming environment comprises a three dimensional gaming environment.

35. The computer system as described in claim 27, wherein said method further comprises:

personifying said graphically generated avatar to said local participant by illustrating real-time emotion of said local participant through said video image stream comprising representative said real-time photo-realistic features of said local participant.

36. The computer system as described in claim 27, wherein said method further comprises:

generating another real-time renderings of said local participant based on said plurality of real-time video streams by applying said new view synthesis technique, said another real-time renderings taken from a perspective of another participant located remotely in said gaming environment; and

sending said another real-time renderings to said another participant for viewing within said gaming environment.