Systems and methods for implementing augmented reality and/or virtual reality

• US 10,182,210 B1
• Filed: 12/13/2017
• Issued: 01/15/2019
• Est. Priority Date: 12/15/2016
• Status: Active Grant

• Current Assignee: Steelcase Incorporated
• Original Assignee: Steelcase Incorporated
• Inventors: Goetzinger, Jr., Stephen E.
• Primary Examiner(s): Nguyen, Khai N
• Application Number:
  US15/840,703
• Time in Patent Office: 398 Days
• Field of Search: 348 1407
• CPC Class Codes: H04L 65/1086 session scope modification
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A system and method for presenting remote attendee representations at remote conference session locations where the attendee representations are presented as directional videos such that each representation has an apparent sight trajectory aimed at a target of interest (e.g., typically a content window, another attendee representation, etc.) that is an instance of the same target of interest for the attendee associated with the representation at a different station.

Claims

1. An enhanced reality collaboration system comprising:

a first affordance configuration for use by at least a first collaboration session attendee, the first affordance configuration including at least a first emissive surface, at least a first camera and a first sensor, the first camera having a field of view (FOV) that includes a location to be generally occupied by the first attendee, the first emissive surface presenting a first representation of a first target of interest (TOI), the first camera generating images of the first attendee at the first location and the first sensor generating data useable to identify a sight trajectory (ST) of the first attendee at the first affordance configuration;

a second affordance configuration remote from the first affordance configuration and for use by at least a second collaboration session attendee, the second affordance configuration including at least a second emissive surface and presenting a second representation of the first TOI and a first video representation of the first attendee at non-overlapping locations;

a system processor programmed to perform the steps of;

(i) using sensor data from the first sensor to detect the first attendee'"'"'s ST at the first affordance configuration;

(ii) determining when the first attendee'"'"'s ST is aimed at the first representation of the first TOI;

(iii) identifying the location of the second representation of the first TOI at the second affordance configuration;

(iv) identifying the location of the first video representation of the first attendee at the second affordance configuration; and

(iv) when the first attendee'"'"'s ST is aimed at the first representation of the first TOI at the first affordance configuration, using images from the first camera to generate the first video representation of the first attendee where the first video representation of the first attendee shows the first attendee with an ST aimed at the location of the second representation of the first TOI at the second affordance configuration.

2. The system of claim 1 wherein the first camera includes the first sensor and wherein images from the first camera are used to determine the first attendee'"'"'s ST at the first affordance configuration.

3. The system of claim 1 wherein the first emissive surface further presents a first representation of a second target of interest (TOI), the second affordance configuration presenting a second representation of the second TOI, the system processor further performing the steps of, determining when the first attendee'"'"'s ST is aimed at the first representation of the second TOI, identifying the location of the second representation of the second TOI at the second affordance configuration, and, when the first attendee'"'"'s ST is aimed at the first representation of the second TOI at the first affordance configuration, using images from the first camera to generate the first video representation of the first attendee where the first video representation of the first attendee shows the first attendee with an ST aimed at the location of the second representation of the second TOI at the second affordance configuration.

4. The system of claim 3 wherein the first video representation of the first attendee is presented between the first representations of the first and second TOIs.

5. The system of claim 3 wherein the second affordance configuration includes an interface enabling the second attendee to move the second representation of the first TOI among at least first and second different locations on the second emissive surface and where, when the second representation of the first TOI is moved from the first location to the second location while the first attendee'"'"'s ST is directed at the first representation of the first TOI, the processor uses the images from at least the first camera to generate the first video representation where the first video representation shows the first attendee'"'"'s ST aimed at the second representation of the first TOI at the second location.

6. The system of claim 1 wherein the second affordance configuration includes an interface enabling the second attendee to move the first representation of the first attendee among at least first and second different locations on the second emissive surface and where, when the first representation of the first attendee is moved from the first location to the second location while the first attendee'"'"'s ST is directed at the first representation of the first TOI, the processor uses the images from at least the first camera to generate the first video representation where the first video representation shows the first attendee'"'"'s ST aimed at the second representation of the first TOI when the first representation of the first attendee is at the first location and also when the first representation of the first attendee is at the second location.

7. The system of claim 6 wherein a plurality of first representations of TOIs are presented at different locations on the first emissive surface and a plurality of second representations of the TOIs are presented at different locations on the second emissive surface and wherein the first video representation of the first attendee always shows the first attendee with an ST directed at a second representation of the TOI at the second affordance configuration that corresponds to the first representation of the same TOI at the first affordance configuration that the first attendee'"'"'s ST is aligned with.

8. The system of claim 7 wherein each of at least a subset of the TOIs represented at the first affordance configuration includes a directional video of an associated attendee at a different location.

9. The system of claim 1 wherein the enhanced reality system is an augmented reality system where the emissive surface is at least partially transparent.

10. The system of claim 1 wherein first and second emissive surfaces are built into virtual reality headsets.

11. The system of claim 1 wherein the at least a second emissive surface includes at least first and second emissive surface portions that are located in non-parallel planes and wherein the second representation of the first TOI and the first representation of the first attendee are presented on the first and second emissive surface portions.

12. The system of claim 11 wherein the first emissive surface forms an arc.

13. The system of claim 12 wherein the first emissive surface substantially surrounds the location occupied by the first attendee on at least three sides.

14. The system of claim 1 wherein the second affordance configuration further includes a second camera and a second sensor, the second camera having a field of view (FOV) that includes a location to be generally occupied by the second attendee, the second emissive surface presenting a first representation of a second target of interest (TOI), the second camera generating images of the second attendee at the second location and the second sensor generating data useable to identify a sight trajectory (ST) of the second attendee at the second affordance configuration;

the first affordance configuration presenting a second representation of the second TOI and a first video representation of the second attendee at non-overlapping locations, the system processor further programmed to perform the steps of;

(i) using sensor data from the second sensor to detect the second attendee'"'"'s ST at the second affordance configuration;

(ii) determining when the second attendee'"'"'s ST is aimed at the first representation of the second TOI;

(iii) identifying the location of the second representation of the second TOI at the first affordance configuration;

(iv) identifying the location of the first video representation of the second attendee at the first affordance configuration; and

(v) when the second attendee'"'"'s ST is aimed at the first representation of the second TOI at the second affordance configuration, using images from the second camera to generate the first video representation of the second attendee where the first video representation of the second attendee shows the second attendee with an ST aimed at the location of the second representation of the second TOI at the first affordance configuration.

15. The system of claim 1 further including at least a third affordance configuration for use by at least a third collaboration session attendee, the third affordance configuration including at least a third emissive surface and presenting a third representation of the first TOI, a third representation of the second TOI, a second video representation of the first attendee and a second video representation of the second attendee, the processor further programmed to perform the steps of:

(i) identifying the locations of the third representations of the first and second TOIs at the third affordance configuration;

(ii) identifying the locations of the first video representations of each of the first and second attendees at the third affordance configuration; and

(iii) when the first attendee'"'"'s ST is aimed at the first representation of the first TOI at the first affordance configuration, using images from the first camera to generate the second video representation of the first attendee where the second video representation of the first attendee shows the first attendee with an ST aimed at the location of the third representation of the first TOI at the third affordance configuration;

(iv) when the first attendee'"'"'s ST is aimed at the second representation of the second TOI at the first affordance configuration, using images from the first camera to generate the second video representation of the first attendee where the second video representation of the first attendee shows the first attendee with an ST aimed at the location of the third representation of the second TOI at the third affordance configuration;

(v) when the second attendee'"'"'s ST is aimed at the second representation of the first TOI at the second affordance configuration, using images from the second camera to generate the second video representation of the second attendee where the second video representation of the second attendee shows the second attendee with an ST aimed at the location of the third representation of the first TOI at the third affordance configuration; and

(vi) when the second attendee'"'"'s ST is aimed at the first representation of the second TOI at the second affordance configuration, using images from the second camera to generate the second video representation of the second attendee where the second video representation of the second attendee shows the second attendee with an ST aimed at the location of the third representation of the second TOI at the third affordance configuration.

16. The system of claim 1 wherein at least the first affordance configuration includes a personal workstation.

17. The system of claim 1 wherein the second representation of the first TOI is visually distinguished when from other TOIs presented at the second affordance configuration when the first attendee'"'"'s ST is aimed at the first representation of the first TOI at the first affordance configuration.

18. The system of claim 1 wherein at least the second affordance configuration includes a multi-attendee conferencing configuration for use by at least the second attendee and a third attendee positioned at first and second locations within a conference space, the second affordance configuration further including at least a third emissive surface that is non-planar with the second emissive surface, the second emissive surface arranged to substantially face the first location and the third emissive surface arranged to substantially face the second location within the conference space, the processor programmed to present differently angled video representations of the first attendee on the second and third emissive surfaces, each of the differently angled video representations presenting the first attendee with an ST aimed at the same general location within the conference space.

19. An enhanced reality collaboration system comprising:

a plurality of remotely located affordance configurations, each of the affordance configurations for use by at least one collaboration session attendee, each affordance configuration including at least a first emissive surface, at least a first camera and at least a first sensor, each camera having a field of view (FOV) that includes a location to be occupied by an attendee, at least a subset of the emissive surfaces presenting representations of each of a plurality of targets of interest (TOIs) at locations selected by an occupying attendee, at least a subset of the emissive surfaces presenting video representations of at least a subset of the attendees at other affordance configurations, each camera generating images of an occupying attendee at the affordance configuration and each sensor generating data useable to identify a sight trajectory (ST) of the occupying attendee at an associated affordance configuration;

a system processor programmed to perform the steps of;

(i) identifying locations of each TOI representation at each affordance configuration;

(ii) identifying locations of each attendee video representation at each affordance configuration;

(iii) using sensor data from the sensors at each affordance configuration to detect each attendee'"'"'s ST at associated affordance configurations;

(ii) determining when any attendee'"'"'s ST is aimed at one of the plurality of representations of TOIs at the affordance configuration occupied by the attendee;

(iii) when an attendee'"'"'s ST is aimed at a TOI representation at an occupied affordance configuration, using images from the camera at the occupied affordance configuration to generate at least first and second video representations of the attendee where the first and second video representations are presented at first and second other affordance configurations and each shows the attendee with an ST aimed at a location of the TOI representation at the affordance configuration that replicates the TOI representation that the attendee'"'"'s ST is aimed at the occupied affordance configuration, and wherein the first and second video representations show the attendee from first and second different perspectives.

20. The system of claim 19 wherein, each time an attendee changes ST to aim at a different TOI, the system processor generates at least some signal indicating an instantaneously new TOI.

21. An enhanced reality collaboration system comprising:

a plurality of remotely located affordance configurations, each of the affordance configurations for use by at least one collaboration session attendee, each affordance configuration including at least a first emissive surface that includes at least first and second different emissive surface portions that are non-coplanar, at least a first camera and at least a first sensor, each camera having a field of view (FOV) that includes a location to be occupied by an attendee, at least a subset of the emissive surfaces presenting representations of each of a plurality of targets of interest (TOIs) at locations selected by an occupying attendee, at least a subset of the emissive surfaces presenting video representations of at least a subset of the attendees at other affordance configurations, each camera generating images of an occupying attendee at the affordance configuration and each sensor generating data useable to identify a sight trajectory (ST) of the occupying attendee at an associated affordance configuration, at least some of the TOIs and attendee video representations on different emissive surface portions at each affordance configuration;

a system processor programmed to perform the steps of;

(i) identifying locations of each TOI representation at each affordance configuration;

(ii) identifying locations of each attendee video representation at each affordance configuration;

(iii) using sensor data from the sensors at each affordance configuration to detect each attendee'"'"'s ST at associated affordance configurations;

(ii) determining when any attendee'"'"'s ST is aimed at one of the plurality of representations of TOIs at the affordance configuration occupied by the attendee;

(iii) when an attendee'"'"'s ST is aimed at a TOI representation at an occupied affordance configuration, using images from the camera at the occupied affordance configuration to generate at least first and second video representations of the attendee where the first and second video representations are presented at first and second other affordance configurations and each shows the attendee with an ST aimed at a location of the TOI representation at the affordance configuration that replicates the TOI representation that the attendee'"'"'s ST is aimed at the occupied affordance configuration, and wherein the first and second video representations show the attendee from first and second different perspectives; and

wherein the processor further augments content on the emissive surfaces to provide at least some indication regarding attendee ST and subtended TOIs.