Method and system for real-time video communication within a virtual environment

• US 20030234859A1
• Filed: 06/21/2002
• Published: 12/25/2003
• Est. Priority Date: 06/21/2002
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Malzbender, Thomas, Culbertson, W. Bruce, Gelb, Daniel G., Sobel, Irwin, Tanguay, Donald O., Goss, Michael E., Baker, H. Harlyn
• Granted Patent: US 6,853,398 B2
• Time in Patent Office: Days
• US Class Current: 348/14.05
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A method for real-time video communication. Specifically, one embodiment of the present invention discloses a method of video conferencing that captures a plurality of real-time video streams of a local participant from a plurality of sample viewpoints. From the plurality of video streams, a new view synthesis technique can be applied to generate a video image stream in real-time of the local participant rendered from a second location of a second participant with respect to a first location of the local participant in a coordinate space of a virtual environment. A change in either of the locations leads to the modifying of the video image stream, thereby enabling real-time video communication from the local participant to the second participant.

Claims

1. A method of video conferencing comprising:

a) creating a virtual environment in three dimensions;

b) capturing a plurality of real-time video streams of a local participant from a plurality of sample viewpoints;

c) generating a video image stream in real-time of said local participant to be displayed to a second participant based on said plurality of real-time video streams by applying a new view synthesis technique, said video image stream rendered from a second location of said second participant with respect to a first location of said local participant within a coordinate space of said virtual environment;

d) enabling a change in at least one of said first and second locations;

e) modifying said video image stream to reflect said change in c); and

f) enabling real-time video communication from said local participant to said second participant.

2. The method of video conferencing as described in claim 1, wherein c) comprises:

c1) generating a three dimensional video model of said local participant based on said plurality of real time video streams by applying said new view synthesis technique; and

c2) generating said video image stream of said local participant from said three dimensional model, wherein said video image stream is in two dimensions.

3. The method of video conferencing as described in claim 1, wherein d) comprises:

d1) enabling said local participant and said second participant to navigate through said coordinate space of said virtual environment effecting said change in at least one of said first and second locations.

4. The method of video conferencing as described in claim 1, wherein f) further comprises:

f1) sending said video image stream to said second participant for blending within a synthetic rendering of said three dimensional virtual environment from said second location.

5. The method of video conferencing as described in claim 1, further comprising:

g) receiving a second video image stream in real-time of said second participant;

h) receiving an input audio stream of said second participant; and

i) blending said second video image stream within a synthetic rendering of said virtual environment.

6. The method of video conferencing as described in claim 5, wherein said second video image stream in real-time is generated by applying said new view synthesis technique on a second plurality of real-time video streams of said second participant, said second video image stream rendered from said first location of said local participant with respect to said second location of said second participant within said coordinate space of said virtual environment in order to enable real-time video conferencing between said local participant and said second participant.

7. The method of video conferencing as described in claim 5, further comprising:

j) portraying an eye contact view of said local participant on a display viewed by said second participant during periods when said local participant is directly viewing an image of said second participant from said second video image stream.

8. The method of video conferencing as described in claim 1, wherein f) further comprises:

f1) capturing an output audio stream in real-time of said local participant; and

f2) sending said output audio stream to said second participant.

9. The method of video conferencing as described in claim 1, wherein said virtual environment is a conference room.

10. The method of video conferencing as described in claim 1, further comprising:

g) generating a second video image stream in real-time of said local participant based on said plurality of real-time video streams by applying said new view synthesis technique, said second video image stream rendered from a third location of a third participant with respect to said local participant within said coordinate space of said virtual environment;

h) enabling a change in at least one of said first and third locations;

i) modifying said second video image stream to reflect said change in h); and

j) enabling real-time video conferencing from said local participant to said third participant.

35. The computer system as described in claim 1, wherein d) in said method comprises:

d1) enabling said local participant and said second participant to navigate through said coordinate space of said virtual environment effecting said change in at least one of said first and second locations.

11. A method of communication comprising:

a) creating a virtual environment in three dimensions;

b) capturing a plurality of real-time video streams of a local participant from a plurality of sample viewpoints;

c) generating a plurality of output video image streams in real-time of said local participant, to be displayed to a plurality of observing participants, based on said plurality of real-time video streams by applying a new view synthesis technique, said plurality of output video image streams rendered from a plurality of observing locations of said plurality of observing participants with respect to a first location of said local participant within a coordinate space of said virtual environment;

d) enabling changes in at least one of said first location and said plurality of observing locations;

e) modifying corresponding output video image streams in said plurality of output video image streams to reflect said changes in c); and

f) enabling real-time video communication in a communication session from said local participant to said plurality of observing participants.

12. The method of communication as described in claim 11, wherein c) comprises:

generating a three dimensional video model of said local participant based on said plurality of real-time video streams for at least one of said plurality of output video image streams, a first output video image stream, said first output video image stream rendered from a first observing location of a first observing participant with respect to said first location; and

generating said first output video image stream in two dimensions from said three dimensional video model.

13. The method of communication as described in claim 11, wherein f) further comprises:

f1) sending said plurality of output video image streams to said plurality of observing participants for blending within a plurality of synthetic renderings corresponding to said plurality of observing participants within said virtual environment.

14. The method of communication as described in claim 11, wherein f) further comprises:

f1) capturing an output audio stream in real-time corresponding to said local participant; and

f2) sending said output audio stream to said plurality of observing participants.

15. The method of communication as described in claim 11, further comprising:

g) receiving a plurality of input audio streams from said plurality of observing participants;

h) receiving a plurality of input video image streams in real-time of said plurality of observing participants, wherein at least one of said input video image streams, a first input video image stream, is generated by applying said new view synthesis technique on a second plurality of real-time video streams of a first observing participant, said first input video image stream a first observing location of said first observing participant with respect to said first location of said local participant within said coordinate space of said virtual environment; and

i) blending said plurality of input video image streams within a synthetic rendering of said three dimensional virtual environment.

16. The method of communication as described in claim 11, wherein said plurality of output video image streams focus on a portion of said local participant.

17. The method of communication as described in claim 11, wherein said local participant is an object.

18. The method of communication as described in claim 14, further comprising:

modulating strengths of said plurality of input audio streams as a function of distance between said local participant and said plurality of observing participants to diminish audio interference in separate and independent side communications in said communication session.

19. The method of communication as described in claim 11, wherein a plurality of participants represented in said virtual environment includes said local participant and said observing participants, and wherein said plurality of participants are physically located in independent and remote physical locations coupled together through a communication network.

20. The method of communication as described in claim 11, wherein at least one of said plurality of observing participants is located within a localized area surrounding said local participant within said coordinate space of said virtual environment.

21. The method of communication as described in claim 12, wherein at least one of said plurality of observing participants requests participation within said communication session including said local participant and said plurality of observing participants.

22. A video conferencing station comprising:

a plurality of video cameras for recording real-time video streams of a local participant at multiple sample viewpoints surrounding said user;

a new view synthesis module for generating a plurality of output video image streams by applying a new view synthesis technique, based on said real-time video streams, of said local participant rendered from a plurality of locations of a plurality of observing participants from a first location of said local participant within a coordinate space of a virtual environment;

an interface module for blending a plurality of input video image streams from said plurality of observing participants within a synthetic rendering of said three dimensional virtual environment; and

a display for viewing said plurality of input video image streams to enable real-time video communication in a communication session between said local participant and said plurality of observing participants within said virtual environment.

23. The video conferencing station as described in claim 22, further comprising a contour extraction module for extracting said local participant from a physical background.

24. The video conferencing station as described in claim 22, further comprising a microphone for capturing an audio stream in real-time of said local participant.

25. The video conferencing station as described in claim 22, wherein said interface module receives a plurality of input audio streams in real-time associated with said plurality of observing participants for mixing into a single audio stream that is broadcast through a local speaker.

26. A method of communication comprising:

a) simultaneously generating a first video image stream in real-time of a first participant and a second video image stream in real-time of a second participant by applying a new view synthesis technique, said first and second participants represented within a virtual environment,

b) allowing for navigation within said virtual environment by said first and second participants;

c) modifying said first and second video image streams to reflect said navigation in b)

d) sending said first video image stream to said second participant;

e) sending said second video image stream to said first participant; and

f) enabling real-time video communication between said first and second participant within said virtual environment.

27. The method of communication as described in claim 26, wherein generating said first video image stream in real-time in a) comprises:

a1) capturing a plurality of real-time video streams of said first participant from a plurality of sample viewpoints;

a2) generating said first video image stream in real-time based on said plurality of real-time video streams by applying said new view synthesis technique, said first video image stream a second location of said second participant with respect to a first location of said first participant within a coordinate space of said virtual environment; and

wherein b) comprises;

b1) enabling a change in at least one of said first and second locations.

28. The method of communication as described in claim 27, further comprising:

sending said first video image stream to said second participant for blending within a synthetic rendering of said three dimensional virtual environment.

29. The method of communication as described in claim 26, wherein generating said second video image stream in real-time in a) comprises:

a1) capturing a plurality of real-time video streams of said second participant from a plurality of sample viewpoints;

a2) generating said second video image stream in real-time based on said plurality of real-time video streams by applying said new view synthesis technique, said second video image stream a first location of said first participant with respect to a second location of said second participant within a coordinate space of said virtual environment; and

wherein b) comprises;

b1) enabling a change in at least one of said first and second locations.

30. The method of communication as described in claim 29, further comprising:

sending said second video image stream to said first participant for blending within a synthetic rendering of said three dimensional virtual environment.

31. The method of communication as described in claim 26, wherein said first and second participants are located in independent and remote locations coupled together through a communication network.

32. The method of communication as described in claim 26, wherein said virtual environment is a conference room.

33. A computer system comprising:

a processor; and

a computer readable memory coupled to said processor and containing program instructions that, when executed, implement a method of video conferencing comprising;

a) creating a virtual environment in three dimensions;

b) capturing a plurality of real-time video streams of a local participant from a plurality of sample viewpoints;

c) generating a video image stream in real-time of said local participant to be displayed to a second participant based on said plurality of real-time video streams by applying a new view synthesis technique, said video image stream rendered from a second location of said second participant with respect to a first location of said local participant within a coordinate space of said virtual environment;

d) enabling a change in at least one of said first and second locations;

e) modifying said video image stream to reflect said change in c); and

f) enabling real-time video communication from said local participant to said second participant.

34. The computer system as described in claim 33, wherein c) in said method comprises:

c1) generating a three dimensional video model of said local participant based on said plurality of real time video streams by applying said new view synthesis technique; and

c2) generating said video image stream of said local participant from said three dimensional model, wherein said video image stream is in two dimensions.

36. The computer system as described in claim 33, wherein f) in said method further comprises:

f1) sending said video image stream to said second participant for blending within a synthetic rendering of said three dimensional virtual environment from said second location.

37. The computer system as described in claim 33, wherein said method further comprises:

g) receiving a second video image stream in real-time of said second participant;

h) receiving an input audio stream of said second participant; and

i) blending said second video image stream within a synthetic rendering of said virtual environment.

38. The computer system as described in claim 37, wherein said second video image stream in real-time is generated by applying said new view synthesis technique on a second plurality of real-time video streams of said second participant, said second video image stream rendered from said first location of said local participant with respect to said second location of said local participant within said coordinate space of said virtual environment in order to enable real-time video conferencing between said local participant and said second participant.

39. The computer system as described in claim 37, wherein said method further comprises:

j) portraying an eye contact view of said local participant on a display viewed by said second participant during periods when said local participant is directly viewing an image of said second participant from said second video image stream.

40. The computer system as described in claim 33, wherein f) in said method further comprises:

f1) capturing an output audio stream in real-time of said local participant; and

f2) sending said output audio stream to said second participant.

41. The computer system as described in claim 33, wherein said virtual environment is a conference room.

42. The computer system as described in claim 33, wherein said method further comprises:

g) generating a second video image stream in real-time of said local participant based on said plurality of real-time video streams by applying said new view synthesis technique, said second video image stream rendered from a third location of a third participant with respect to said local participant within said coordinate space of said virtual environment;

h) enabling a change in at least one of said first and third locations;

i) modifying said second video image stream to reflect said change in h); and

j) enabling real-time video conferencing from said local participant to said third participant.