Virtual/augmented reality transition system and method

• US 9,690,374 B2
• Filed: 04/27/2015
• Issued: 06/27/2017
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Clement, Manuel Christian, Faaborg, Alexander James
• Primary Examiner(s): EURICE, MICHAEL
• Application Number:
  US14/696,716
• Publication Number:
  US 20160313790A1
• Time in Patent Office: 792 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method of operating an audio visual system generating a virtual immersive experience may include an electronic user device in communication with a tracking device that may track a user'"'"'s physical movement in a real world space and translate the tracked physical movement into corresponding movement in the virtual world generated by the user device. The system may detect when a user and the user device are approaching a boundary of a tracking area and automatically initiate a transition out of the virtual world and into the real world. A smooth, or graceful, transition between the virtual world and the real world as the user encounters this boundary may avoid disorientation which may occur as a user continues to move in the real world, while motion appears to have stopped upon reaching the tracking boundary.

Claims

1. A method of operating a system configured to generate a virtual environment, the method comprising:

activating a tracking device and tracking a position of a user electronic device in an ambient environment;

detecting a transition condition, including

detecting an external input and generating a transition command when the detected external input is recognized as a request from an external party to initiate a transition of the user electronic device between the virtual environment and the ambient environment, or

comparing a distance between the user electronic device and at least one boundary of a tracking area tracked by the tracking device within the ambient environment and generating a transition command when the distance between the user electronic device and the at least one boundary of the tracking area is less than or equal to a preset threshold distance; and

performing a transition process to transition between the virtual environment generated by the user electronic device and the ambient environment in response to the detected transition condition.

2. The method of claim 1, wherein an area of the ambient environment is greater than a tracking area of the tracking device, such that a system boundary corresponds to a boundary of the tracking area of the tracking device within the ambient environment, detecting a transition condition including:

comparing a distance between the user electronic device and the boundary of the tracking area of the tracking device within the ambient environment; and

generating the transition command to initiate the transition process when the distance between the user electronic device and the boundary of the tracking area is less than or equal to a preset threshold distance.

3. The method of claim 1, wherein an area of the ambient environment is less than a tracking area of the tracking device, such that a system boundary corresponds to a physical boundary of the ambient environment, detecting a transition condition including:

comparing a distance between the user electronic device and the physical boundary of the ambient environment; and

generating the transition command to initiate the transition process when the distance between the user electronic device and the physical boundary of the ambient environment is less than or equal to a preset threshold distance.

4. The method of claim 1,

detecting an external input including detecting a physical gesture implemented by an external party, an audio command generated by an external party, an approach of an external party, or an eye gaze of an external party, and

comparing a distance between the user electronic device and at least one boundary of a tracking area tracked by the tracking device including determining at least one of a direction, a speed, or an acceleration of movement of the user electronic device based on a comparison of a current position to at least one previous position of the electronic device detected by the tracking device, and

performing a transition process including:

replacing virtual image output displayed on a display of the user electronic device in the virtual environment with transition image output displayed on the display, the transition image output including at least one of a monochromatic display, a pass through image of the ambient environment captured by a camera of the user electronic device, or a fading out of the virtual image output of the virtual environment and a fading in of the transition image output, or

replacing virtual audio output generated by the user electronic device in the virtual environment with transition audio output, the transition audio output including at least one of a preset signal or tone or a pass through of ambient sound captured by a microphone of the user electronic device, or a fading out of the audio output of the virtual environment and a fading in of transition audio output, or

generating at least one of an externally directed visual indicator, an externally directed audible indicator, or an externally directed physical indicator indicating to an external party that the transition process is complete.

5. The method of claim 1, comparing a distance between the user electronic device and at least one boundary of a tracking area tracked by the tracking device also including:

comparing a current position of the user electronic device detected by the tracking device to at least one previous position of the user electronic device detected by the tracking device; and

determining at least one of a direction of movement of the user electronic device, a speed of movement of the user electronic device, or an acceleration of movement of the user electronic device based on the comparison of the current position to the at least one previous position.

6. The method of claim 5, performing a transition process to transition between the virtual environment generated by the user electronic device and the ambient environment including:

selecting a transition profile based on at least one of the determined direction, speed or acceleration of movement of the user electronic device, environmental parameters of the virtual environment generated by the user electronic device, or preset user preferences.

7. The method of claim 1, wherein performing a transition process includes:

initiating the transition process in response to the transition command and performing the transition process such that the transition between the virtual environment and the ambient environment is complete at or before the user electronic device reaches the at least one boundary of the tracking area.

8. The method of claim 7, performing the transition process also including:

detecting that the user electronic device has crossed the at least one boundary and is outside of the tracking area of the tracking device;

detecting that the user electronic device has moved from the outside of the tracking area and across the at least one boundary of the tracking area and has re-entered the tracking area; and

transitioning back into the virtual environment when the tracking device detects that the user electronic device has re-entered the tracking area.

9. The method of claim 1, detecting a transition condition including:

detecting physical obstacle in a movement path of the user electronic device in the ambient environment, within a tracking area of the tracking device within the ambient environment;

comparing a distance between the user electronic device and the detected obstacle; and

generating a transition command to initiate the transition process when the distance between the user electronic device and the detected obstacle is less than or equal to a preset threshold distance.

10. The method of claim 9, detecting a physical obstacle in a movement path of the user electronic device in the ambient environment, within a tracking area of the tracking device within the ambient environment including:

intermittently scanning the tracking area and capturing an image of the tracking area;

comparing a current image of the tracking area to at least one previous image of the tracking area captured by the tracking device; and

detecting the physical obstacle in the movement path of the user electronic device in the ambient environment based on at least one change detected when comparing the current image of the tracking area to the at least one previous image of the tracking area.

11. The method of claim 1, detecting an external input including at least one of:

detecting a physical gesture implemented by an external party,

detecting an audio command generated by an external party,

detecting an approach of an external party, or

detecting an eye gaze of an external party; and

wherein performing the transition process includes generating at least one of a visual indicator, an audible indicator, or a physical indicator directed to the external party indicating completion of the transition process.

12. The method of claim 11, detecting an audio command corresponding to an externally generated transition command including at least one of detecting a voice command including at least one previously set keyword, or detecting an audio tone, corresponding to the externally generated transition command.

13. The method of claim 1, performing a transition process including transitioning from the virtual environment to the ambient environment, including at least one of:

fading out a virtual image output of the virtual environment displayed on a display of the user electronic device and fading in a transition image output displayed on the display of the user electronic device;

or

fading out of audio output of the virtual environment output by the user electronic device and fading in of transition audio output by the user electronic device.

14. The method of claim 13, wherein fading out a virtual image output and fading in a transition image output includes replacing a virtual video output of the virtual world displayed on the display with a monochromatic display.

15. The method of claim 13, fading out a virtual image output and fading in a transition image output including replacing a virtual video output of the virtual world displayed on the display with a pass through image of the ambient environment captured by an imaging device of the user electronic device.

16. The method of claim 13, fading out a virtual image output and fading in a transition image output including:

transitioning the display from a non-transparent state to a transparent state while fading out the virtual image output displayed on the display such that the ambient environment is visible through the display upon completion of the fade out of the virtual image output.

17. An audio visual system, comprising:

a user electronic device generating a virtual environment, the user electronic device operating in and being movable within an ambient environment;

a tracking device in communication with the user electronic device, the tracking device tracking a position and movement of the user electronic device in the ambient environment; and

a processor configured to;

translate the detected movement of the user electronic device in the ambient environment into virtual movement in the virtual environment generated by the user electronic device;

detect a transition condition, including

detect an external input recognized as a request from an external party for transition of the user electronic device between the virtual environment and the ambient environment, or

compare a distance between the user electronic device and at least one boundary of a tracking area tracked by the tracking device within the ambient environment and generate a transition command when the distance between the user electronic device and the at least one boundary of the tracking area is less than or equal to a preset threshold distance; and

perform a transition between the virtual environment generated by the user electronic device and the ambient environment in response to the detected transition condition.

18. The system of claim 17, wherein the processor is further configured to:

transition out of the virtual environment generated by the user electronic device based on transition parameters included in a transition profile corresponding to the detected transition condition, including at least one of;

replacing virtual image output displayed on a display of the user electronic device in the virtual environment with transition image output displayed on the display based on the transition parameters included in the transition profile, the transition image output including at least one of a monochromatic display or a pass through image of the ambient environment captured by a camera of the user electronic device,

replacing virtual audio output generated by the user electronic device in the virtual environment with transition audio output, the transition audio output including at least one of a preset signal or tone or a pass through of ambient sound captured by a microphone of the user electronic device, or

generating at least one of an externally directed visual indicator, an externally directed audible indicator, or an externally directed physical indicator indicating to an external party that the transition process is complete.

19. The system of claim 17, the external input including at least one of a physical gesture implemented by an external party, an audio command generated by an external party, an approach of an external party, or an eye gaze of an external party.

20. The system of claim 17, wherein the processor is further configured to:

perform a transition from the virtual environment generated by the user electronic device to the ambient environment such that the transition out of the virtual environment is complete at or before the user electronic device reaches the boundary; and

perform a second transition from the ambient environment into the virtual environment when the tracking device detects that the user electronic device has crossed the boundary and has re-entered the tracking area.

21. A method, comprising:

generating, by an electronic device, a virtual environment, the electronic device operating in an ambient environment;

tracking, by a tracking device, a position of the electronic device in the ambient environment;

detecting a transition condition, including;

detecting an external input and recognizing the detected external input as a request from an external party to initiate a transition, or

detecting a distance between the electronic device and at least one boundary of a tracking area tracked by the tracking device that is less than or equal to a previously set threshold distance, including determining at least one of a direction, a speed, or an acceleration of movement of the electronic device based on a comparison of a current position to at least one previous position of the electronic device detected by the tracking device; and

performing a transition process to transition between the virtual environment generated by the electronic device and the ambient environment in response to the detected transition condition, including;

replacing virtual image output displayed on a display of the user electronic device in the virtual environment with transition image output displayed on the display, the transition image output including at least one of a monochromatic display, a pass through image of the ambient environment captured by a camera of the electronic device, or a fading out of the virtual image output of the virtual environment and a fading in of the transition image output, or

replacing virtual audio output generated by the electronic device in the virtual environment with transition audio output, the transition audio output including at least one of a preset signal or tone or a pass through of ambient sound captured by a microphone of the electronic device, or a fading out of the audio output of the virtual environment and a fading in of transition audio output, or

generating at least one of an externally directed visual indicator, an externally directed audible indicator, or an externally directed physical indicator indicating to an external party that the transition process is complete.