Vehicle trajectory determination to stabilize vehicle-captured video

• US 10,271,021 B2
• Filed: 02/29/2016
• Issued: 04/23/2019
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Smolyanskiy, Nikolai, Zhang, Zhengyou, Dendi, Vikram R.
• Primary Examiner(s): Sosanya, Obafemi O
• Application Number:
  US15/056,678
• Publication Number:
  US 20170251179A1
• Time in Patent Office: 1,149 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: H04B 7/18506 Communications with or from aircraft, i.e. aeronautical mobile service
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A vehicle has a camera system that captures video while the vehicle moves. The vehicle records the captured video and/or wirelessly transmits the captured video to a remote user device for playback. When the vehicle is moving, a coarse waypoint is identified and a trajectory is determined from the current location of the vehicle to the coarse waypoint that reduces (e.g., minimizes) sudden changes in direction of movement of the vehicle, reduces (e.g., minimizes) sudden changes in speed of the vehicle, and/or reduces (e.g., minimizes) sudden changes in acceleration of the vehicle by reducing (e.g., minimizing) jerk or snap of the vehicle trajectory. One or more fine waypoints along the trajectory are selected and the vehicle moves to the coarse waypoint along the trajectory by passing through those fine waypoints, resulting in smooth movement of the device that reduces or eliminates motion sickness for users viewing the captured video.

Claims

1. A method to stabilize vehicle-captured video, the method comprising:

identifying a coarse waypoint for a vehicle;

determining a trajectory for the vehicle from a current location of the vehicle to the coarse waypoint, wherein the trajectory is a polynomial, wherein determining the trajectory comprises;

reducing sudden changes in direction of movement of the vehicle by minimizing a first derivative of the trajectory at a first time and minimizing a second derivative of the trajectory at a second, different, time;

reducing sudden changes in speed of the vehicle; and

reducing sudden changes in acceleration of the vehicle;

selecting one or more fine waypoints along the trajectory;

communicating commands to a movement control system to maneuver the vehicle from the current location to the coarse waypoint and passing through each of the one or more fine waypoints; and

capturing video via a camera system of the vehicle while the vehicle maneuvers from the current location to the coarse waypoint.

2. The method of claim 1, the vehicle comprising an unmanned aerial vehicle.

3. The method of claim 1, the captured video being stabilized due to the vehicle maneuvering from the current location to the coarse waypoint through the one or more fine waypoints.

4. The method of claim 1, the video comprising a first person view of the vehicle.

5. The method of claim 4, further comprising recording the video for later playback.

6. The method of claim 4, further comprising transmitting the video to a viewing device for playback by a user of the viewing device, the viewing device comprising virtual reality or augmented reality glasses, or a virtual reality or augmented reality headset.

7. The method of claim 4, further comprising the trajectory reducing sudden changes in direction of movement of the vehicle by reducing jerk of the vehicle.

8. The method of claim 7, wherein reducing sudden changes in direction of movement of the vehicle comprises minimizing a third derivative of the trajectory.

9. The method of claim 4, further comprising the trajectory reducing sudden changes in direction of movement of the vehicle by reducing snap of the vehicle.

10. The method of claim 9, wherein reducing sudden changes in direction of movement of the vehicle comprises minimizing a fourth derivative of the trajectory.

11. A device comprising:

a coarse waypoint determination module configured to identify a coarse waypoint for a vehicle;

a trajectory determination module configured to determine a trajectory for the vehicle from a current location of the vehicle to the coarse waypoint, wherein the trajectory is a polynomial, wherein to determine the trajectory the trajectory determination module is configured to;

minimize a first derivative of the trajectory to reduce sudden changes in direction of movement of the vehicle at a first time and minimizing a second derivative of the trajectory at a second, different, time;

reduce sudden changes in speed of the vehicle; and

reduce sudden changes in acceleration of the vehicle;

a fine waypoint selection module configured to select multiple fine waypoints along the trajectory;

a movement control module configured to maneuver the vehicle from the current location to the coarse waypoint by passing through each of the one or more fine waypoints; and

a camera system configured to capture video while the vehicle maneuvers from the current location to the coarse waypoint.

12. The device of claim 11, the movement control module being further configured to maneuver the vehicle by providing control signals to one or more motors of the vehicle.

13. The device of claim 11, the device comprising a drone.

14. The device of claim 11, the trajectory reducing sudden changes in direction of movement of the vehicle by reducing jerk of the camera system.

15. The device of claim 11, the trajectory reducing sudden changes in direction of movement of the vehicle by reducing snap of the camera system.

16. The device of claim 11, the video comprising a first person view of the vehicle, the video being stabilized due to the vehicle maneuvering from the current location to the coarse waypoint by passing through the one or more fine waypoints.

17. A device comprising:

one or more processors;

a computer-readable storage medium having stored thereon multiple instructions that, responsive to execution by the one or more processors, cause the one or more processors to perform acts comprising;

identifying a coarse waypoint for a vehicle;

determining a trajectory for the vehicle from a current location of the vehicle to the coarse waypoint, wherein the trajectory is a polynomial, by;

reducing sudden changes in direction of movement of the vehicle by minimizing a first derivative of the trajectory at a first time and minimizing a second derivative of the trajectory at a second, different, time;

reducing sudden changes in speed of the vehicle; and

reducing sudden changes in acceleration of the vehicle;

selecting multiple fine waypoints along the trajectory;

maneuvering the vehicle from the current location to the coarse waypoint and passing through each of the one or more fine waypoints; and

capturing video via a camera system of the vehicle while the vehicle maneuvers from the current location to the coarse waypoint.

18. The device of claim 17, the trajectory reducing sudden changes in direction of movement of the vehicle by reducing jerk of the camera system.

19. The device of claim 17, the trajectory reducing sudden changes in direction of movement of the vehicle by reducing snap of the camera system.

20. The device of claim 17, the video comprising a first person view of the vehicle, the video being stabilized due to the vehicle maneuvering from the current location to the coarse waypoint passing through the one or more fine waypoints.