Wireless immersive experience capture and viewing

• US 9,615,177 B2
• Filed: 03/06/2015
• Issued: 04/04/2017
• Est. Priority Date: 03/06/2014
• Status: Active Grant

• Current Assignee: Sphere Optics Company, Llc
• Original Assignee: Sphere Optics Company, Llc
• Inventors: Englert, Robert S., Giordano, Meyer J.
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/640,197
• Publication Number:
  US 20150254882A1
• Time in Patent Office: 760 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04R 2499/15 Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence

Abstract

Aspects capture, view, present and display photo-realistic, fully-immersive panoramic virtual environments, displaying to a user via a display screen, in a first layer at a first level of opacity, an image of current surroundings of the user from image data captured from the current surroundings of the user by an image capture device; and in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the user as a function of time or of place, wherein the second layer is located coextensive with the first layer. The first level of opacity is selected to present image data of the panoramic image within the first layer as transparent relative to image data of the panoramic image within the second layer; or as opaque relative to image data of the panoramic image within the second layer.

Claims

1. A system, comprising:

a processor;

a computer readable memory in circuit communication with the processor;

a computer readable storage medium in circuit communication with the processor;

at least one image capture device in circuit communication with the processor; and

at least one display screen in circuit communication with the processor; and

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

displays via the at least one display screen, in a first layer at first level of opacity, an image of current surroundings of the system from image data captured by the at least one image capture device from the current surroundings of the system, wherein the image within the first layer is image data of a surrounding real time physical environment of the user while wearing the system;

displays via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the system as a function of time or of place, wherein the second layer is located coextensive with the first layer;

selects the first level of opacity to present image data of the image within the first layer from the group consisting of transparent relative to image data of the image within the second layer, thereby blending a streaming of real-time visual images of the surrounding real time physical environment of the user wearing the system with the second layer image data of the another environment, and opaque relative to image data of the image within the second layer;

in response to determining that the user is walking, presents the image data within the second layer at a first level of transparency; and

in response to determining that the user is stationary, presents the image data within the second layer at a second level of transparency that is more opaque than the first level of transparency.

2. The system of claim 1, wherein the at least one image capture device is a front facing camera.

3. The system of claim 2, wherein the image of the another environment is a 360 degree panoramic image construct;

and

wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

syncs turning movements of a head of the user to the left or to the right with corresponding movement through geographic features of the 360 degree panoramic image construct of the another environment; and

displays via the at least one display screen, in the second layer at the second level of opacity, portions of the image of another environment that correspond with a viewpoint of the user during the turning movements of the head of the user to the left or to the right.

4. The system of claim 3, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby:

selects the first level of transparency to present real-time visual images of the surrounding real time physical environment of the user wearing the system to enable the user to see and walk through the physical environment of the user while simultaneously viewing the second layer image data of the 360 degree panoramic image construct of the another environment.

5. The system of claim 4, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby:

in response to a determination that the user is walking and moving toward a collision with another object, presents the image data within the second layer at a third level of transparency that is more transparent than the first level of transparency.

6. A system, comprising:

a processor;

a computer readable memory in circuit communication with the processor;

a computer readable storage medium in circuit communication with the processor;

at least one image capture device in circuit communication with the processor;

at least one display screen in circuit communication with the processor;

at least one directional microphone in circuit communication with the processor; and

at least one speaker in circuit communication with the processor;

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

displays via the at least one display screen, in a first layer at first level of opacity, an image of current surroundings of the system from image data captured by the at least one image capture device from the current surroundings of the system;

displays via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the system as a function of time or of place, wherein the second layer is located coextensive with the first layer; and

selects the first level of opacity to present image data of the image within the first layer from the group consisting of transparent relative to image data of the image within the second layer, and opaque relative to image data of the image within the second layer;

drives the at least one speaker to reproduce first sound signals that are detected from the current surroundings of the system by the at least one directional microphone, as a first surround sound representation of the first sound signals in a first volume level;

drives the at least one speaker to reproduce second sound signals that are detected from the another environment, as a second surround sound representation of the second sound signals in a second volume level;

sets the first volume level to be greater than the second volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as opaque relative to the image data of the image within the second layer; and

sets the second volume level to be greater than the first volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as transparent relative to the image data of the image within the second layer.

7. A computer program product for capturing, viewing, presenting and displaying photo-realistic, fully-immersive panoramic virtual environments, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a processor that cause the processor to;

display to a user via at least one display screen, in a first layer at a first level of opacity, an image of current surroundings of the user from image data captured by at least one image capture device from the current surroundings of the system, wherein the image within the first layer is image data of a surrounding real time physical environment of the user while wearing the system;

display via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the user as a function of time or of place, wherein the second layer is located coextensive with the first layer;

select the first level of opacity to present image data of the panoramic image within the first layer from the group consisting of transparent relative to image data of the panoramic image within the second layer, thereby blending a streaming of real-time visual images of the surrounding real time physical environment of the user wearing the system with the second layer image data of the another environment, and opaque relative to image data of the panoramic image within the second layer;

in response to determining that the user is walking, present the image data within the second layer at a first level of transparency; and

in response to determining that the user is stationary, present the image data within the second layer at a second level of transparency that is more opaque than the first level of transparency.

8. The computer program product of claim 7, wherein the at least one image capture device is a front facing camera.

9. The computer program product of claim 8, wherein the image of the another environment is a 360 degree panoramic image construct;

and

wherein the computer readable program code instructions for execution by the processor further cause the processor to;

sync turning movements of a head of the user to the left or to the right with corresponding movement through geographic features of the 360 degree panoramic image construct of the another environment; and

display via the at least one display screen, in the second layer at the second level of opacity, portions of the image of another environment that correspond with a viewpoint of the user during the turning movements of the head of the user to the left or to the right.

10. The computer program product of claim 9, wherein the computer readable program code instructions for execution by the processor further cause the processor to:

select the first level of transparency to present real-time visual images of the surrounding real time physical environment of the user wearing the system to enable the user to see and walk through the physical environment of the user while simultaneously viewing the second layer image data of the 360 degree panoramic image construct of the another environment.

11. The computer program product of claim 10, wherein the computer readable program code instructions for execution by the processor further cause the processor to:

in response to a determination that the user is walking and moving toward a collision with another object, present the image data within the second layer at a third level of transparency that is more transparent than the first level of transparency.

12. A computer program product for capturing, viewing, presenting and displaying photo-realistic, fully-immersive panoramic virtual environments, the computer program product, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a processor that cause the processor to;

display to a user via at least one display screen, in a first layer at a first level of opacity, an image of current surroundings of the user from image data captured by at least one image capture device from the current surroundings of the system;

display via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the user as a function of time or of place, wherein the second layer is located coextensive with the first layer;

select the first level of opacity to present image data of the image within the first layer from the group consisting of transparent relative to image data of the image within the second layer, and opaque relative to image data of the image within the second layer;

drive at least one speaker to reproduce first sound signals that are detected from the current surroundings of the system by at least one directional microphone, as a first surround sound representation of the first sound signals in a first volume level;

drive the at least one speaker to reproduce second sound signals that are detected from the another environment, as a second surround sound representation of the second sound signals in a second volume level;

set the first volume level to be greater than the second volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as opaque relative to the image data of the image within the second layer; and

set the second volume level to be greater than the first volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as transparent relative to the image data of the image within the second layer.

13. A method for capturing, viewing, presenting and displaying photo-realistic, fully-immersive panoramic virtual environments, the method comprising:

displaying to a user via at least one display screen, in a first layer at a first level of opacity, an image of current surroundings of the user from image data captured from the current surroundings of the user by at least one image capture device, wherein the image within the first layer is image data of a surrounding real time physical environment of the user while wearing the system;

displaying via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the user as a function of time or of place, wherein the second layer is located coextensive with the first layer;

selecting the first level of opacity to present image data of the panoramic image within the first layer from the group consisting of transparent relative to image data of the panoramic image within the second layer, thereby blending a streaming of real-time visual images of the surrounding real time physical environment of the user wearing the system with the second layer image data of the another environment, and opaque relative to image data of the panoramic image within the second layer;

in response to determining that the user is walking, presenting the image data within the second layer at a first level of transparency; and

in response to determining that the user is stationary, presenting the image data within the second layer at a second level of transparency that is more opaque than the first level of transparency.

14. The method of claim 13, wherein the at least one image capture device is a front facing camera.

15. The method of claim 14, wherein the image of the another environment is a 360 degree panoramic image construct, the method further comprising:

syncing turning movements of a head of the user to the left or to the right with corresponding movement through geographic features of the 360 degree panoramic image construct of the another environment; and

displaying via the at least one display screen, in the second layer at the second level of opacity, portions of the image of the another environment that correspond with a viewpoint of the user during the turning movements of the head of the user to the left or to the right.

16. The method of claim 15, further comprising:

selecting the first level of transparency to present real-time visual images of the surrounding real time physical environment of the user wearing the system to enable the user to see and walk through the physical environment of the user while simultaneously viewing the second layer image data of the 360 degree panoramic image construct of the another environment; and

in response to a determination that the user is walking and moving toward a collision with another object, presenting the image data within the second layer at a third level of transparency that is more transparent than the first level of transparency.

17. A method for capturing, viewing, presenting and displaying photo-realistic, fully-immersive panoramic virtual environments, the method comprising:

displaying via at least one display screen, in a first layer at first level of opacity, an image of current surroundings of the system from image data captured by the at least one image capture device from the current surroundings of the system;

displaying via the at least one display screen, in a second layer at a second level of opacity, an image of another environment that differs from the current surroundings of the system as a function of time or of place, wherein the second layer is located coextensive with the first layer;

selecting the first level of opacity to present image data of the image within the first layer from the group consisting of transparent relative to image data of the image within the second layer, and opaque relative to image data of the image within the second layer;

driving at least one speaker to reproduce first sound signals that are detected from the current surroundings of the user by at least one directional microphone, as a first surround sound representation of the first sound signals in a first volume level;

driving the at least one speaker to reproduce second sound signals that are detected from the another environment, as a second surround sound representation of the second sound signals in a second volume level;

setting the first volume level to be greater than the second volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as opaque relative to the image data of the image within the second layer; and

setting the second volume level to be greater than the first volume level in response to selecting the first level of opacity to present the image data of the image within the first layer as transparent relative to the image data of the image within the second layer.