Method and system for videoconferencing between parties at N sites

• US 20060244817A1
• Filed: 04/29/2005
• Published: 11/02/2006
• Est. Priority Date: 04/29/2005
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Harville, Michael, Baker, Henry Harlyn, Culbertson, W. Bruce, Tanguay, Donald O. Jr.
• Granted Patent: US 7,528,860 B2
• Time in Patent Office: Days
• US Class Current: 348/14.80
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

A method and system for videoconferencing between parties at N sites. Specifically, a virtual space arrangement of N sites is created. Video sequences of N−1 remote sites are simultaneously displayed on a non-planar display arranged around a local party at a local site consistent with the virtual space arrangement. The displayed images of the remote parties are scaled consistently with their true physical sizes at their distances from the local party in the virtual space arrangement. Real-time video streams of the local party are captured from a plurality of viewpoints and are used to generate N−1 video view streams representing views of the local party from approximately the local display locations of the N−1 remote parties.

Claims

1. A method for videoconferencing between parties at N sites, comprising:

creating a virtual space arrangement of N sites including N−

1 remote sites and a local site;

simultaneously displaying N−

1 video sequences of representations of parties at said N−

1 remote sites at display locations on a non-planar display arrangement arranged around a local party at said local site, wherein said N−

1 remote sites are displayed in a manner that is directionally consistent with said virtual space arrangement to provide reciprocal site-accurate eye contact between a local party at said local site and N−

1 remote parties at said N−

1 remote sites;

scaling displayed images of said remote parties in said N−

1 video sequences consistently with their true physical sizes in said virtual space arrangement;

capturing a plurality of real-time video streams of said local party from a plurality of viewpoints; and

generating N−

1 video view streams providing said reciprocal site-accurate eye contact in real-time of said local party representing views of said local party from display locations corresponding to said N−

1 remote parties.

2. The method of claim 1, further comprising:

adapting to a new virtual space arrangement that accommodates a change in N without physical reconfiguration of said non-planar display arrangement and without physical reconfiguration of a video capturing mechanism that captures said plurality of real-time video streams.

3. The method of claim 1, further comprising:

creating an arrangement of said N sites in a virtual coordinate system that corresponds to said virtual space arrangement.

4. The method of claim 1, further comprising:

adapting to geometric changes in locations of said sites in said virtual arrangement where N remains constant.

5. The method of claim 1, further comprising:

receiving from said N−

1 remote sites said N−

1 video sequences of said N−

1 remote parties, wherein each of said N−

1 video sequences representing a view of a corresponding remote party for said local party consistent with said virtual arrangement, and wherein each of said N−

1 video view streams represent a view of said local party from a display location corresponding to one of said N−

1 remote parties.

6. The method of claim 1, further comprising:

equally spacing said N sites around said virtual space arrangement that comprises a circular arrangement.

7. The method of claim 1, wherein at least one of said N−

1 video view streams of said local site is generated from one video capturing device at one of said plurality of sample viewpoints.

8. The method of claim 1, wherein said video capturing mechanism comprises a plurality of video capturing devices that is fixed in an array located at a surface of said non-planar display arrangement.

9. The method of claim 1, further comprising:

performing background removal to extract parties at said N−

1 remote sites from their physical backgrounds; and

compositing said parties of said N−

1 remote sites on a single continuous background on said non-planar display arrangement.

10. The method of claim 1, further comprising:

constructing three-dimensional models of said local party at said local site; and

rendering said N−

1 video view streams as graphic representations of said three-dimensional models as viewed from the corresponding directions at which said remote parties are displayed on said display surface at said local site.

11. The method of claim 1, wherein at least one of said N−

1 video view streams of said local site is generated using data from more than one of said real-time video streams of said local party captured at said plurality of viewpoints.

12. A system for videoconferencing between parties at N sites, comprising:

a virtual arrangement selector for creating a virtual space arrangement of N sites including N−

1 remote sites and a local site, wherein each site accommodates a party comprising at least one person;

a non-planar display for simultaneously displaying N−

1 video sequences of representations of said N−

1 remote sites at display locations on a non-planar display arrangement arranged around a local party at said local site, wherein said display locations are directionally consistent with said virtual space arrangement, and wherein said N−

1 remote sites are displayed to provide reciprocal site-accurate eye contact between a local party at said local site and N−

1 remote parties at said N−

1 remote sites;

a life-size scaler for scaling displayed images of said remote parties in said N−

1 video sequences consistently with their true physical sizes at their distances from said local party in said virtual space arrangement;

a plurality of video capturing devices for capturing a plurality of real-time video streams of said local party from a plurality of viewpoints;

a view generator for generating N−

1 video view streams in real-time of said local party, each of said N−

1 video view streams representing a view of said local party from a display location corresponding to one of said N−

1 remote parties, wherein said N−

1 video view streams provide said reciprocal site-accurate eye contact; and

a controller for adapting to a new virtual space arrangement that accommodates a change in N without physical reconfiguration of said non-planar display arrangement and without physical reconfiguration of said plurality of video capturing devices at said plurality of sample viewpoints.

13. The system of claim 12, wherein said virtual arrangement selector creates said virtual space arrangement according to physical limitations associated with each of said N sites.

14. The system of claim 12, further comprising:

a receiver for receiving from said N−

1 remote sites said N−

1 video sequences of said N−

1 remote parties, each of said N−

1 video sequences representing a view of a corresponding remote party for said local party consistent with said virtual arrangement.

15. The system of claim 12, further comprising:

a background removal module for performing background removal to extract parties at said N−

1 remote sites from their physical backgrounds; and

a compositor for compositing said parties of said N−

1 remote sites on a single continuous background on said non-planar display arrangement

16. The system of claim 12, further comprising:

an audio spatializer for rendering spatialized audio received from said N−

1 remote sites consistently with said virtual space arrangement at said local site.

17. The system of claim 12, wherein said virtual space arrangement is taken from a group consisting of:

a one dimensional arrangement;

a two dimensional arrangement; and

a three dimensional arrangement.

18. The system of claim 12, wherein said non-planar display arrangement encompasses at least 180 degrees of the field-of-view of said local party at said local site.

19. The system of claim 12, wherein said non-planar display arrangement is a curved wall upon which said N−

1 video sequences are projected.

20. The system of claim 12, wherein said non-planar display arrangement is a plurality of displays positioned in an arc.

21. The system of claim 12, wherein at least one party comprises at least two persons.

22. A computer readable medium containing program instructions that implement a method for videoconferencing between parties at N sites, comprising:

creating a virtual space arrangement of N sites including N−

1 remote sites and a local site;

simultaneously displaying N−

1 video sequences of representations of parties at said N−

1 remote sites at display locations on a non-planar display arrangement arranged around a local party at said local site, wherein said N−

1 remote sites are displayed in a manner that is directionally consistent with said virtual space arrangement to provide reciprocal site-accurate eye contact between a local party at said local site and N−

1 remote parties at said N−

1 remote sites;

scaling displayed images of said remote parties in said N−

1 video sequences consistently with their true physical sizes in said virtual space arrangement;

capturing a plurality of real-time video streams of said local party from a plurality of viewpoints; and

generating N−

1 video view streams providing said reciprocal site-accurate eye contact in real-time of said local party representing views of said local party from display locations corresponding to said N−

1 remote parties.

23. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

adapting to a new virtual space arrangement that accommodates a change in N without physical reconfiguration of said non-planar display arrangement and without physical reconfiguration of a video capturing mechanism that captures said plurality of real-time video streams.

24. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

creating an arrangement of said N sites in a virtual coordinate system that corresponds to said virtual space arrangement.

25. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

receiving from said N−

1 remote sites said N−

1 video sequences of said N−

1 remote parties, wherein each of said N−

1 video sequences representing a view of a corresponding remote party for said local party consistent with said virtual arrangement, and wherein each of said N−

1 video view streams represent a view of said local party from a display location corresponding to one of said N−

1 remote parties.

26. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

equally spacing said N sites around said virtual space arrangement that comprises a circular arrangement.

27. The computer readable medium of claim 22, wherein at least one of said N−

1 video view streams of said local site is generated from one video capturing device at one of said plurality of sample viewpoints.

28. The computer readable medium of claim 22, wherein said capturing a plurality of real-time video streams further comprises program instruction comprising capturing said plurality of real-time video streams with a plurality of video capturing devices that is fixed in an array located at a surface of said non-planar display arrangement.

29. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

performing background removal to extract parties at said N−

1 remote sites from their physical backgrounds; and

compositing said parties of said N−

1 remote sites on a single continuous background on said non-planar display arrangement.

30. The computer readable medium of claim 22, wherein said program instructions in said method further comprise:

constructing three-dimensional models of said local party at said local site; and

rendering said N−

1 video view streams as graphic representations of said three-dimensional models as viewed from the corresponding directions at which said remote parties are displayed on said display surface at said local site.