Immersive interactive telepresence

• US 10,244,211 B2
• Filed: 02/29/2016
• Issued: 03/26/2019
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Smolyanskiy, Nikolai, Zhang, Zhengyou, Dendi, Vikram R., Hall, Michael
• Primary Examiner(s): Sosanya, Obafemi O
• Application Number:
  US15/056,983
• Publication Number:
  US 20170251181A1
• Time in Patent Office: 1,121 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01C 21/3484 Personalized, e.g. from learned user behaviour or user-defined profiles
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/005 by providing the operator with signals other than visual, e.g. acoustic, haptic
• CPC Class Codes: G08G 5/0026 located on the ground
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0039 Modification of a flight plan
• CPC Class Codes: G08G 5/006 in accordance with predefined flight zones, e.g. to avoid prohibited zones
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0082 for monitoring traffic from a ground station radar or analogous systems specially adapted for traffic control G01S13/91
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

In embodiments of immersive interactive telepresence, a system includes a vehicle that captures an experience of an environment in which the vehicle travels, and the experience includes audio and video of the environment. User interactive devices receive the audio and the video of the environment, and each of the user interactive devices represent the experience for one or more users who are remote from the environment. A trajectory planner is implemented to route the vehicle based on obstacle avoidance and user travel intent as the vehicle travels in the environment. The trajectory planner can route the vehicle to achieve a location objective in the environment without explicit direction input from a vehicle operator or from the users of the user interactive devices.

Claims

1. A system implemented for immersive interactive telepresence, the system comprising:

a vehicle configured to capture an experience of an environment in which the vehicle travels, the experience comprising at least audio and video of the environment, wherein the vehicle includes a camera system configured to capture the video as a wide angle view of the environment;

a first interface, at the vehicle, to route the audio and the video of the environment to user interactive devices that are remote from the environment, wherein the video is routed to the user interactive devices to cause each of the one or more users to be able to view a different portion of the environment based on where a user is looking within the wide angle view of the environment, wherein the environment is virtually represented, at each user interactive device, based on a view point or a head position of a user; and

a second interface, at the vehicle, to receive, from a trajectory planner, routing information to route the vehicle based on obstacle avoidance and user travel intent as the vehicle travels in the environment.

2. The system as recited in claim 1, wherein the trajectory planner is configured to route the vehicle in the environment without step-by-step travel instructions received from the one or more users of the user interactive devices.

3. The system as recited in claim 1, wherein the trajectory planner is configured to route the vehicle to achieve a location objective in the environment without explicit direction input from a vehicle operator or from the one or more users of the user interactive devices.

4. The system as recited m claim 1, wherein the trajectory planner is configured to:

determine the user travel intent to route the vehicle based on previous vehicle routing associated with the one or more users of the user interactive devices; and

alter the route of the vehicle in the environment based on the determined user travel intent.

5. The system as recited m claim 1, wherein the trajectory planner is configured to:

determine the user travel intent to route the vehicle based on a user preference provided prior to a start of the vehicle traveling in the environment; and

alter the route of the vehicle in the environment based on the determined user travel intent.

6. The system as recited in claim 1, wherein the vehicle is commissioned by the one or more users who are remote from the environment to capture the experience of an area of interest in the environment, the area of interest selected by the one or more users or the area of interest selected by the trajectory planner based on the user travel intent.

7. The system as recited in claim 6, wherein the vehicle is controlled based on at least one of:

the trajectory planner receivs travel instructions from the one or more users via the user interactive devices for user directed travel of the vehicle;

or

vehicle operator in the environment of the vehicle who directs the route of the vehicle.

8. The system as recited in claim 1, wherein:

the vehicle is a virtual tourist guide in the environment configured to capture the experience as a virtual tour for the one or more users of the user interactive devices; and

the trajectory planner is configured to said route the vehicle effective to facilitate the virtual tour.

9. The system as recited in claim 1, wherein the vehicle is an aerial drone configured to fly over a region of the environment to capture the experience of the region for the one or more users of the user interactive devices;

and

the system further comprising;

one or more additional aerial drones each configured to fly over a different region of the environment, the aerial drone configured to handoff the one or more users to one of the additional aerial drones that captures the experience of the different region of the environment for the one or more users of the user interactive devices.

10. The system as recited in claim 9, wherein the trajectory planner is configured to initiate the handoff of the one or more users to the additional aerial drone based on the different region of the environment selected by the one or more users, or the different region of the environment selected by the trajectory planner based on the user travel intent.

11. The system as recited in claim 1, wherein the trajectory planner is configured to said route the vehicle in a steady motion effective to minimize motion sickness of the one or more users of the user interactive devices.

19. The system of claim 1, wherein the first interface and the second interface reside within a wireless network communication interface.

12. A method for immersive interactive telepresence implemented by a vehicle, the method comprising:

capturing an experience of an environment in which the vehicle travels, the experience comprising at least audio and video of the environment;

capturing the video of the experience as a wide angle view of the environment;

communicating the audio and the video of the experience to user interactive devices that represent the experience for one or more users who are remote from the environment, wherein communicating the video to the user interactive devices causes each of the one or more users to be able to view a different portion of the environment based on where a user is looking within the wide angle view of the environment, wherein the environment is virtually represented, at each user interactive device, based on a view point or a head position of a user;

receiving routing information for the vehicle based on obstacle avoidance and user travel intent as the vehicle travels in the environment; and

instructing a travel control system of the vehicle how to travel in the environment based on the routing information.

13. The method as recited in claim 12, further comprising determining the routing information for the vehicle to achieve a location objective in the environment without explicit direction input from a vehicle operator or from the one or more users of the user interactive devices.

14. The method as recited in claim 12, further comprising:

determining the user travel intent to route the vehicle based on previous vehicle routing associated with the one or more users of the user interactive devices; and

altering the route of the vehicle in the environment based on the determined user travel intent.

15. The method as recited in claim 12, further comprising:

determining the user travel intent to route the vehicle based on a user preference provided prior to a start of the vehicle traveling in the environment; and

altering the route of the vehicle in the environment based on the determined user travel intent.

16. The method as recited in claim 12, further comprising:

flying over a region of the environment and said capturing the experience of the environment for the one or more users of the user interactive devices; and

handing off the one or more users to an additional vehicle that flies over a different region of the environment and captures the experience of the different region of the environment for the one or more users of the user interactive devices.

17. A method for immersive interactive telepresence implemented at a user interactive device, the method comprising:

communicating a user preference to virtually experience an environment that is remote from a user of the user interactive device;

receiving audio and video of an experience captured in the environment in which a vehicle travels a route based on obstacle avoidance and user travel intent as determined from the user preference, the vehicle including a camera system that captures the video as a wide angle view of the environment; and

representing the experience of the environment for the user with the user interactive device, the experience including displaying a portion of the environment based on where the user is looking within the wide angle view of the environment, wherein the environment is virtually represented, at the user interactive device, based on a view point or a head position of a user.

18. The method as recited in claim 17, wherein:

the experience of the environment is a virtual tour represented with the user interactive device for the user; and

the vehicle is a virtual tourist guide being routed in the environment effective to facilitate the virtual tour, the vehicle configured to alter the route in the environment based on the determined user travel intent.