Autonomous drones for tactile feedback in immersive virtual reality

• US 9,836,117 B2
• Filed: 05/28/2015
• Issued: 12/05/2017
• Est. Priority Date: 05/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shapira, Lior
• Primary Examiner(s): Hilgendorf, Dale W
• Application Number:
  US14/724,360
• Publication Number:
  US 20160349835A1
• Time in Patent Office: 922 Days
• CPC Class Codes: B25J 9/1605 Simulation of manipulator lay-out, design, modelling of manipulator
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/16 for controlling, capturing or immobilising other vehicles
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A “Tactile Autonomous Drone” (TAD) (e.g., flying drones, mobile robots, etc.) supplies real-time tactile feedback to users immersed in virtual reality (VR) environments. TADs are not rendered into the VR environment, and are therefore not visible to users immersed in the VR environment. In various implementations, one or more TADs track users as they move through a real-world space while immersed in the VR environment. One or more TADs apply tracking information to autonomously position themselves, or one or more physical surfaces or objects carried by the TADs, in a way that enables physical contact between those surfaces or objects and one or more portions of the user'"'"'s body. Further, this positioning of surfaces or objects corresponds to some real-time virtual event, virtual object, virtual character, virtual avatar of another user, etc., in the VR environment to provide real-time tactile feedback to users immersed in the VR environment.

Claims

1. A system, comprising:

a processor; and

a memory comprising instructions that, when executed by the processor, cause the processor to perform a method comprising;

generating a real-time rendering of a virtual environment;

receiving real-time tracking information from one or more sensors, the real-time tracking information relating to real motions and positions of a user immersed in the real-time rendering of the virtual environment, the real-time tracking information further relating to motions and positions of an autonomous mobile drone in a real-world environment around the user;

responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to move relative to the user;

further responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to automatically deposit a real-world object in a position within the real-world environment that enables one or more user body parts to contact that object based on real user motions while the user interacts with the virtual environment;

responsive to depositing the real-world object by the autonomous mobile drone, rendering a virtual representation of the real-world object into a corresponding position in the virtual environment; and

wherein contact between the real-world object and the one or more user body parts generates a physically tactile sensation for the virtual representation of the real-world object.

2. The system of claim 1 further comprising directing one or more additional autonomous mobile drones to automatically move within the real-world environment around the user to maintain optimal positions to apply one or more integral sensors to track and model one or more specific user body parts.

3. The system of claim 1 wherein the movement of the autonomous mobile drone relative to the user is further synchronized to the real-time rendering of the virtual environment.

4. The system of claim 1 further comprising evaluating real-time positions and motions of the user and the autonomous mobile drone and automatically directing movement of the autonomous mobile drone to ensure that no unintended contact is made between the user and any portion of the autonomous mobile drone.

5. The system of claim 1 directing the autonomous mobile drone to automatically position itself relative to the user in a manner that causes one or more tactile virtual extensions of the autonomous mobile drone to contact one or more user body parts based on real user motions while the user interacts with the virtual environment.

6. The system of claim 5 further comprising causing the tactile virtual extension of the autonomous mobile drone to initiate a force-feedback upon contact between the tactile virtual extension and the one or more user body parts.

7. The system of claim 1 wherein a plurality of tactile virtual extensions are coupled to movable members of the autonomous mobile drone, each movable member capable of moving in multiple dimensions to accurately position one or more of the tactile virtual extensions relative to the user.

8. The system of claim 1 further comprising presenting the real-time rendering of the virtual environment to the user in an otherwise empty real-world environment around the user that allows the user full freedom of motion.

9. The system of claim 1 further comprising rendering one or more safety markers into the immersive virtual environment to alert the user as to real elements existing within the real-world environment around the user.

10. The system of claim 1 further comprising replacing the real-time rendering of a virtual environment with a live video of the real-world environment around the user based on a measured distance between the user and the autonomous mobile drone to ensure that no unintended contact is made between the user and the autonomous mobile drone.

11. The system of claim 1 wherein the autonomous mobile drone is capable of autonomous flight in the real-world environment around the user.

12. The system of claim 1 further comprising an audio output device coupled to the autonomous mobile drone, and further comprising causing the audio output device of the autonomous mobile drone to deliver sound from one or more points in the real-world space around the user, said sound being coordinated with and mapped to positions of corresponding virtual elements in the virtual environment.

13. A method, comprising:

generating a real-time rendering of a virtual environment;

displaying the real-time rendering of the virtual environment via a head-worn virtual reality display device;

receiving real-time tracking information from one or more sensors, the real-time tracking information relating to real motions and positions of a user immersed in the real-time rendering of the virtual environment, the real-time tracking information further relating to motions and positions of an autonomous mobile drone in a real-world environment around the user;

responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to move relative to the user;

further responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to automatically deposit a real-world object in a position within the real-world environment that enables one or more user body parts to contact that object based on real user motions while the user interacts with the virtual environment;

responsive to depositing the real-world object by the autonomous mobile drone, rendering a virtual representation of the real-world object into a corresponding position in the virtual environment; and

wherein contact between the real-world object and the one or more user body parts generates a physically tactile sensation for the virtual representation of the real-world object.

14. The method of claim 13 further comprising directing one or more additional autonomous mobile drones to automatically move within the real-world environment around the user to maintain optimal positions to apply one or more integral sensors to track and model one or more specific user body parts.

15. The method of claim 13 further comprising synchronizing the movement of the autonomous mobile drone relative to the user to the real-time rendering of the virtual environment.

16. The method of claim 13 further comprising evaluating real-time positions and motions of the user and the autonomous mobile drone and automatically directing movement of the autonomous mobile drone to ensure that no unintended contact is made between the user and any portion of the autonomous mobile drone.

17. The method of claim 13 further comprising directing the autonomous mobile drone to automatically position itself relative to the user in a manner that causes one or more tactile virtual extensions of the autonomous mobile drone to contact one or more user body parts based on real user motions while the user interacts with the virtual environment.

18. The method of claim 13 further comprising directing a plurality of tactile virtual extensions that are coupled to movable members of the autonomous mobile drone to move in one or more dimensions to accurately position one or more of the tactile virtual extensions relative to one or more body parts of the user.

19. The method of claim 13 further comprising rendering one or more safety markers into the immersive virtual environment to alert the user as to real elements existing within the real-world environment around the user.

20. A computer-readable storage device having computer executable instructions stored therein for rendering a real-time virtual environment and directing one or more autonomous mobile drones, said instructions causing a computing device to execute a method comprising:

generating a real-time rendering of a virtual environment within a user-worn virtual reality display device;

receiving real-time tracking information from one or more sensors, the real-time tracking information relating to real motions and positions of a user immersed in the real-time rendering of the virtual environment, the real-time tracking information further relating to motions and positions of one or more autonomous mobile drone in a real-world environment around the user;

responsive to receipt of the real-time tracking information, directing one or more of the autonomous mobile drones to move relative to the user;

further responsive to receipt of the real-time tracking information, directing one or more of the autonomous mobile drones to automatically deposit one or more real-world objects in positions within the real-world environment that enables one or more user body parts to contact those objects based on real user motions while the user interacts with the virtual environment;

responsive to depositing the one or more real-world objects by the one or more autonomous mobile drones, rendering corresponding virtual representations of the real-world objects into corresponding positions in the virtual environment; and

wherein contact between any of the real-world objects and the one or more user body parts generates a physically tactile sensation for the virtual representation of the corresponding real-world object.