Methods and systems for providing feedback based on information received from an aerial vehicle

• US 10,258,534 B1
• Filed: 02/14/2017
• Issued: 04/16/2019
• Est. Priority Date: 01/21/2016
• Status: Active Grant

• Current Assignee: Wing Aviation LLC (Alphabet Inc.)
• Original Assignee: Wing Aviation LLC (Alphabet Inc.)
• Inventors: Sills, Maxwell Andrew, Gordon, Robert Samuel, Wetherbee, Ian
• Primary Examiner(s): Dunn, Alex C
• Application Number:
  US15/432,416
• Time in Patent Office: 791 Days
• CPC Class Codes: A61H 3/06 Walking aids for blind persons replacing direct visual perception by another kind of perception A61F9/08
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/25 Fixed-wing aircraft VTOL aircraft B64U10/20
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/64 for parcel delivery or retrieval
• CPC Class Codes: B64U 2101/69 the UAVs provided with means for airdropping goods, e.g. deploying a parachute during descent
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: B64U 50/37 Charging when not in flight
• CPC Class Codes: B64U 70/80 Vertical take-off or landing, e.g. using rockets rotorcrafts B64U10/10; VTOL aircraft B64U10/20
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3652 Guidance using non-audiovisual output, e.g. tactile, haptic or electric stimuli
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan

Abstract

Described herein is a control system that facilitates assistance mode(s). In particular, the control system may determine a particular assistance mode associated with an account. This particular assistance mode may specify (i) operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information. The control system may transmit to the aerial vehicle an indication of the particular operations corresponding to the particular assistance mode and may then receive environment information for the location associated with the account. Based on the received environment information, the control system may apply the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.

Claims

1. A method comprising:

determining, by a control system as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receiving, by the control system from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determining, by the control system, a second location of an obstacle relative to the first location corresponding to the feedback system; and

applying, by the control system, a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

2. The method of claim 1, wherein the one or more feedback processes include one or more of the following forms of feedback:

(i) auditory feedback, (ii) vibrational feedback, and (iii) visual feedback.

3. The method of claim 1, wherein the feedback system comprises a wearable feedback system including one or more of the following components:

(i) at least one listening device, (ii) at least one vibration-generating device, (iii) at least one display, (iv) at least one projector, and (v) at least one light source.

4. The method of claim 1, further comprising:

based on the environment information, determining, by the control system, a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

5. The method of claim 4,

wherein determining the potential future contact comprises determining one or more of the following factors:

(i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.

6. The method of claim 1, further comprising:

based on the environment information, determining, by the control system, a potential future contact with the obstacle; and

in response to determining the potential future contact with the obstacle, the control system directing the aerial vehicle to move the obstacle from the second location to a third location that is selected based on the potential future contact being avoidable.

7. The method of claim 1, further comprising:

receiving, by the control system, input data that comprises a request for navigation to a particular destination; and

applying, by the control system, another feedback process to initiate, via the feedback system, other feedback indicative of at least one instruction specifying at least one movement towards the particular destination.

8. The method of claim 1, wherein the feedback system comprises a display device, and wherein applying the particular feedback process comprises:

directing the feedback system to display, on the display device, visual feedback information related to one or more of (i) the first location corresponding to the feedback system and (ii) the second location of the obstacle.

9. The method of claim 8, wherein the feedback system further comprises a video-capture device configured to provide video data representative of the first location corresponding to the feedback system, and wherein directing the feedback system to display, on the display device, the visual feedback information comprises:

directing the feedback system to display, on the display device, the visual feedback information as being overlaid on at least a portion of the provided video data.

10. The method of claim 8, wherein the visual feedback information comprises one or more of the following forms of information:

(i) the environment information, (ii) at least one visual symbol that is representative of at least one instruction specifying at least one movement towards a particular destination, (iii) at least one visual symbol that is representative of the second location of the obstacle relative to the first location corresponding to the feedback system, and (iv) at least one visual symbol that is representative of a point-of-interest associated with the first location corresponding to the feedback system.

11. The method of claim 1, wherein the aerial vehicle is a first aerial vehicle, wherein the obstacle is a first obstacle, wherein the environment information is first environment information, and wherein the particular feedback process is a first feedback process, the method further comprising:

receiving, by a control system from a second aerial vehicle, second environment information related to the first location;

based on the second environment information, determining, by the control system, a third location of a second obstacle relative to the first location corresponding to the feedback system; and

applying, by the control system, a second feedback process in accordance with the assistance mode to initiate, via the feedback system, further feedback that indicates the third location of the second obstacle relative to the first location corresponding to the feedback system.

12. The method of claim 1, wherein the account is a first account, wherein the feedback system is a first feedback system, wherein the environment information is first environment information, and wherein the feedback process is a first feedback process, the method further comprising:

receiving, by the control system from the aerial vehicle, second environment information related to a third location corresponding to a second feedback system associated with a second account;

based on the second environment information, determining, by the control system, a fourth location of a second obstacle relative to the third location corresponding to the second feedback system; and

applying, by the control system, a second feedback process to initiate, via the second feedback system, further feedback that indicates the fourth location of the second obstacle relative to the third location corresponding to the second feedback system.

13. The method of claim 1, wherein the control system is part of the aerial vehicle, and wherein the control system receiving environment information from the aerial vehicle comprises the control system receiving environment information from one or more sensors of the aerial vehicle.

14. The method of claim 1, wherein the control system is part of the feedback system.

15. A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a control system to perform functions comprising:

determining, as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receiving, from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determining a second location of an obstacle relative to the first location corresponding to the feedback system; and

applying a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

16. The non-transitory computer readable medium of claim 15, the functions further comprising:

based on the environment information, determining a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

17. The non-transitory computer readable medium of claim 16,

wherein determining the potential future contact comprises determining one or more of the following factors:

(i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.

18. A control system comprising:

one or more processors;

a non-transitory computer readable medium; and

program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to;

determine, as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receive, from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determine a second location of an obstacle relative to the first location corresponding to the feedback system; and

apply a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

19. The control system of claim 18, wherein the program instructions are further executable to:

based on the environment information, determine a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

20. The control system of claim 19,

wherein the program instructions being executable to determine the potential future contact comprises the program instructions being executable to determine one or more of the following factors:

(i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.