Systems and methods for implementing augmented reality and/or virtual reality

• US 10,659,733 B1
• Filed: 11/30/2018
• Issued: 05/19/2020
• Est. Priority Date: 12/15/2016
• Status: Active Grant

• Current Assignee: Steelcase Incorporated
• Original Assignee: Steelcase Incorporated
• Inventors: Goetzinger, Stephen E.
• Primary Examiner(s): Nguyen, Khai N.
• Application Number:
  US16/205,683
• Time in Patent Office: 536 Days
• Field of Search: 348 1407
• CPC Class Codes: H04L 65/1086 session scope modification
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A system and method for presenting remote attendee representations at remote conference session locations where the attendee representations are presented as directional videos such that each representation has an apparent sight trajectory aimed at a target of interest (e.g., typically a content window, another attendee representation, etc.) that is an instance of the same target of interest for the attendee associated with the representation at a different station.

Claims

1. An enhanced reality workstation for use by an electronic conference attendee, the workstation comprising:

an emissive surface assembly that includes at least first and second emissive surface portions that are arranged about an attendee location to be occupied by an attendee using the workstation;

at least a first camera for obtaining images of a first attendee at the workstation;

at least a first sensor for sensing the first attendee'"'"'s sight trajectory at the workstation;

a processor programmed to perform the steps of;

(i) identifying a target of interest (TOI) presented at some location on the emissive surface assembly subtended by the attendee'"'"'s sight trajectory;

(ii) using the images from the at least a first camera to generate at least first and second angled video representations of the first attendee to be presented at other workstations.

2. The workstation of claim 1 further including a substantially horizontal worksurface having a rear edge, the emissive surface located adjacent the rear edge of the worksurface.

3. The workstation of claim 2 further including a leg structure for supporting the worksurface at a worksurface height.

4. The workstation of claim 2 wherein each of the first and second emissive surface portions is substantially flat.

5. The workstation of claim 1 wherein the at least a first camera and the first sensor are a single device.

6. The workstation of claim 1 wherein the at least a first camera includes a plurality of cameras arranged about the first attendee, the step of using images including using images from a plurality of the cameras at the workstation to generate the video representations.

7. The workstation of claim 6 wherein at least the first video representation shows the first attendee head on and wherein at least the second video representation shows the first attendee from a partial side view.

8. The workstation of claim 1 wherein the processor is further programmed to present content on different portions of the emissive surface wherein each presented content set comprises at least one TOI.

9. The workstation of claim 8 further including a database storing information usable to generate views of the first attendee from different angles based on images collected by the at least a first camera.

10. The workstation of claim 1 wherein a second attendee is located at a second workstation that includes an emissive surface with first and second non-parallel emissive surface portions and wherein a video representation of the first attendee and a representation of the TOI are presented on the first and second emissive surface portions at the second workstation, respectively, and, wherein, the processor is further programmed to receive information related to the relative positions of the video representation of the first attendee and the TOI representation at the second workstation and use that information to generate at least one of the first and second angled video representations of the first user to present at the second workstation.

11. The workstation of claim 1 wherein the processor further uses the TOI subtended by the first attendee'"'"'s sight trajectory to generate the at least first and second angled video representations.

12. The workstation of claim 1 wherein the first and second angled images are different and are presented on emissive surfaces at second and third remote workstations, respectively.

13. The workstation of claim 12 wherein the TOI is presented at second and third locations on emissive surfaces at the second and third stations, respectively, and wherein the first and second angled video representations are presented at fourth and fifth locations at the second and third stations, respectively, and wherein the first angled video representation shows the first attendee looking toward the TOI at the second location and the second angled video representation shows the first attendee looking toward the TOI at the third location.

14. The workstation of claim 1 wherein the processor is further programmed to present an angled video representation of the first attendee at a second location on an emissive surface at a second workstation that is remote from the first workstation and, wherein, the processor selects which of the angled video representations of the first attendee to present at the second workstation as a function of the location at which the angled video representation is presented on the emissive surface at the second workstation.

15. The workstation of claim 14 wherein the processor is further programmed to present a representation of the TOI at a location on the emissive surface at the second workstation and wherein the processor selects which of the angled video representations of the first attendee to present at the second workstation as a function of the location at which the TOI is presented on the emissive surface at the second workstation.

16. The workstation of claim 15 wherein the location of at least one of the TOI and the angled video representation is changeable at the second station and wherein the processor selects a different one of the angled video representations as the location of the at least one of the TOI and the angled video representation is changed.

17. The workstation of claim 1 wherein the processor is programmed to present the first and second angled video representations on emissive surfaces at second and third workstations, respectively.

18. An enhanced reality conferencing system for use by at least first and second attendees, the system comprising:

a first workstation including a first emissive surface arrangement and at least a first station camera;

a second workstation including a second emissive surface arrangement that is linked via a communication link to the first workstation and at least a second station camera;

at least one processor programmed to drive the first and second emissive surfaces to;

present a first target of interest (TOI) at a first location on the first emissive surface and at a second location on the second emissive surface;

present at least a first angled video representation of the second attendee located at the second workstation on the first emissive surface at a third location;

present at least a second angled video representation of the first attendee located at the first workstation on the second emissive surface at a fourth location; and

wherein, the at least one processor is also programmed (i) to use images from the first camera to generate the second angled video showing the first attendee looking toward the location of the first TOI on the second emissive surface when the first attendee is looking at the first TOI at the first workstation and (ii) to use images from the second camera to generate the first angled video showing the second attendee looking toward the location of the first TOI on the first emissive surface when the second attendee is looking at the first TOI at the second workstation.

19. The conferencing system of claim 18 wherein the first workstation further includes a first interface usable to change the location of at least one of the first TOI and the second angled video on the first emissive surface and the second workstation further includes a second interface usable to change the location of at least one of the first TOI and the first angled video on the first emissive surface and wherein the processor is further programmed to track the locations of the first TOI and first and second angled videos and to modify the angled videos so that regardless of the relative juxtapositions of the first TOI and the angled videos, whenever the first attendee is looking at the first TOI at the first station, the second angled video shows the first attendee looking at the first TOI on the second emissive surface and, whenever the second attendee is looking at the first TOI at the second station, the first angled video shows the second attendee looking at the first TOI on the first emissive surface.

20. The conferencing system of claim 19 wherein the processor also presents a second TOI on each of the first and second emissive surfaces and wherein when the first attendee'"'"'s sight trajectory is aimed at the second TOI, the processor generates the second angled video showing the first user looking at the second TOI at the second station regardless of the relative locations of the second TOI and the second angled video at the second station and, when the second attendee'"'"'s sight trajectory is aimed at the second TOI, the processor generates the first angled video showing the second user looking at the second TOI at the first station regardless of the relative locations of the second TOI and the first angled video at the first station.

21. The conferencing system of claim 18 wherein the at least a first processor includes multiple distributed processors.

22. An enhanced reality conferencing system for use by at least first and second attendees, the system comprising:

a first workstation including a first emissive surface arrangement and at least a first camera;

a second workstation including a second emissive surface arrangement;

at least one processor programmed to perform the steps of;

(i) presenting a first target of interest at a first location on the first emissive surface and at a second location on the second emissive surface wherein each target of interest includes at least one of a content set and a video representation of a remote attendee to the conference;

(ii) using images form the at least a first camera to detect a sight trajectory (ST) of a first attendee at the first station;

(iii) determining when the first attendee'"'"'s ST is directed at

the TOI presented on the first emissive surface

(iv) using images from the at least a first camera to generate a first angled video representation showing the first attendee; and

(v) presenting the first angled video representation at a third location on the second emissive surface wherein the first angled video representation shows the first attendee looking toward the TOI presented at the second location on the second emissive surface.

23. The conferencing system of claim 22 wherein the first workstation includes a first interface, the processor is programmed to present a plurality of TOIs on each of the first and second emissive surfaces, many of the TOIs presented on the first emissive surface are common with the TOIs presented on the second emissive surface and, wherein the first interface enables the first attendee to move at least one of the common TOIs simultaneously on both the first and second emissive surfaces.

24. The conferencing system of claim 22 wherein the first emissive surface includes at least first and second non-parallel emissive surface portions and wherein the second emissive surface includes at least first and second non-parallel emissive surface portions.

25. The conferencing system of claim 24 wherein the first angled video representation is presented on the first emissive surface portion and the TOI is presented on the second emissive surface portion.

26. The conferencing system of claim 22 wherein the first emissive surface includes first, second and third non-parallel emissive surface portions arranged about a space occupied by the first attendee and wherein the second emissive surface includes first, second and third non-parallel emissive surface portions arranged about a space occupied by the second attendee.

27. The conferencing system of claim 22 wherein the processor is further programmed to present angled video representations of other remote attendees in addition to the first attendee on the second emissive surfaces, wherein each of the angled video representations is another TOI.

28. The conferencing system of claim 22 wherein the processor includes a plurality of distributed system processors including at least one processor integrated with the first workstation and a second process integrated with the second workstation.

29. The conferencing system of claim 22 wherein the at least a first camera includes a plurality of cameras.

30. The conferencing system of claim 22 wherein a plurality of TOIs are presented on each of the first and second emissive surfaces and wherein the processor is further programmed to determine when the first attendee'"'"'s ST is directed at any one of the TOI presented on the first emissive surface and to generate and present the first angled video representation on the second emissive surface such that the first angled video representation shows the first attendee looking toward the TOI at the second workstation that the first attendee is looking at the first workstation.

31. The conferencing system of claim 22 wherein the processor is further programmed to present a representation of the second attendee on the first emissive surface, detect when the first attendee'"'"'s ST is directed at the representation of the second attendee on the first emissive surface and, when the first attendee'"'"'s ST is directed at the representation of the second attendee on the first emissive surface, generate and present the first angled video representation on the second emissive surface showing the first attendee looking toward the second attendee at the second workstation.

32. The conferencing system of claim 22 wherein at least one of the first and second workstations includes an interface useable to change the location of the TOI on at least one of the first and second emissive surfaces.

33. The conferencing system of claim 32 wherein each of the first and second workstations includes an interface wherein each of the interfaces is usable to change the location of the TOI on at least one of the first and second emissive surfaces.

34. The conferencing system of claim 32 wherein the second workstation interface is useable to move the first angled video representation to different locations on the second emissive surface and wherein the processor is programmed to modify the first angled video representation so that the first angled video representation continues to show the first attendee looking toward the TOI while the first attendee persistently looks toward the TOI at the first station as the first angled video representation is moved to different locations on the second emissive surface.

35. The conferencing system of claim 22 wherein the processor is further programmed to perform the steps of:

(i) presenting at least a second TOI at a fourth location on the first emissive surface and at a fifth location on the second emissive surface;

(ii) determining when the first attendee'"'"'s ST is directed at the second TOI presented on the first emissive surface; and

(iii) using images from the at least a first camera to generate the first angled video representation showing the first attendee and presenting the first angle video representation on the second emissive surface showing the first attendee looking toward the second TOI at the fifth location on the second emissive surface.

36. The conferencing system of claim 22 wherein the second workstation further includes at least a second station camera, the at least one processor further programmed to perform the steps of:

(i) using images form the at least a second camera to detect a sight trajectory (ST) of a second attendee at the second station;

(iii) determining when the second attendee'"'"'s ST is directed at the TOI presented on the second emissive surface;

(iv) using images from the at least a second camera to generate a second angled video representation showing the second attendee; and

(v) presenting the second angled video representation at a fourth location on the first emissive surface wherein the second angled video representation shows the second attendee looking toward the presented at the first location on the first emissive surface.