Methods and systems for providing feedback based on information received from an aerial vehicle

• US 9,594,372 B1
• Filed: 01/21/2016
• Issued: 03/14/2017
• Est. Priority Date: 01/21/2016
• Status: Active Grant

• Current Assignee: Wing Aviation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Wetherbee, Ian, Sills, Maxwell Andrew, Gordon, Robert Samuel
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US15/002,733
• Time in Patent Office: 418 Days
• US Class Current: 1/1
• CPC Class Codes: A61H 3/06 Walking aids for blind persons replacing direct visual perception by another kind of perception A61F9/08
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/25 Fixed-wing aircraft VTOL aircraft B64U10/20
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/64 for parcel delivery or retrieval
• CPC Class Codes: B64U 2101/69 the UAVs provided with means for airdropping goods, e.g. deploying a parachute during descent
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: B64U 50/37 Charging when not in flight
• CPC Class Codes: B64U 70/80 Vertical take-off or landing, e.g. using rockets rotorcrafts B64U10/10; VTOL aircraft B64U10/20
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3652 Guidance using non-audiovisual output, e.g. tactile, haptic or electric stimuli
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan

Abstract

Described herein is a control system that facilitates assistance mode(s). In particular, the control system may determine a particular assistance mode associated with an account. This particular assistance mode may specify (i) operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information. The control system may transmit to the aerial vehicle an indication of the particular operations corresponding to the particular assistance mode and may then receive environment information for the location associated with the account. Based on the received environment information, the control system may apply the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.

Claims

1. A method comprising:

determining, by a control system, a particular assistance mode associated with an account, wherein the particular assistance mode specifies (i) particular operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information;

transmitting, by the control system to the aerial vehicle, at least an indication of the particular operations corresponding to the particular assistance mode;

receiving, by the control system, environment information for the location associated with the account, wherein the received environment information is based on the particular operations corresponding to the particular assistance mode; and

based on the received environment information for the location associated with the account, applying, by the control system, the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.

2. The method of claim 1, further comprising:

receiving, by the control system, input data specifying the particular assistance mode, wherein determining the particular assistance mode is based at least in part on the received input data specifying the particular assistance mode.

3. The method of claim 2, wherein the input data comprises a request for navigation to a particular destination, and wherein determining the particular assistance mode based at least in part on the received input data comprises:

based on the request for navigation to the particular destination, selecting a navigation mode and arranging the navigation mode to specify a respective feedback process to provide feedback indicative of at least one instruction specifying at least one movement towards the particular destination.

4. The method of claim 2,

wherein the input data comprises a specification of one or more of the following criteria:

(i) a requested extent of feedback to be provided and (ii) a requested form of feedback to be provided, and

wherein determining the particular assistance mode based at least in part on the received input data comprises arranging the particular assistance mode in accordance with one or more of the specified criteria.

5. The method of claim 1, further comprising:

before determining the particular assistance mode, receiving, by the control system, preceding environment information corresponding to the location associated with the account, wherein determining the particular assistance mode is based at least in part on the preceding environment information.

6. The method of claim 5, wherein determining the particular assistance mode based at least in part on the preceding environment information comprises:

based on the preceding environment information, determining one or more of the following criteria;

(i) an extent of feedback to be provided and (ii) a form of feedback to be provided; and

arranging the particular assistance mode to operate in accordance with one or more of the determined criteria.

7. The method of claim 1, wherein the particular operations comprise one or more of the following operations:

(i) at least one particular flight procedure to carry out when at least one particular movement associated with the account is observed, (ii) at least particular environment portion for which to obtain sensor data when the at least one particular movement associated with the account is observed, and (iii) at least one particular sensor with which to obtain sensor data when the at least one particular movement associated with the account is observed.

8. The method of claim 1, wherein the feedback processes respectively include one or more of the following forms of feedback:

(i) respective auditory feedback, (ii) respective vibrational feedback, and (iii) respective visual feedback.

9. The method of claim 1, wherein the associated feedback system comprises a wearable feedback system including one or more of the following components:

(i) at least one listening device, (ii) at least one vibration-generating device, (iii) at least one display, (iv) at least one projector, and (v) at least one light source.

10. The method of claim 1, wherein applying the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system comprises:

based on the received environment information for the location associated with the account, determining a potential future contact with a particular obstacle; and

determining a respective feedback process to initiate feedback that indicates the potential future contact with the particular obstacle.

11. The method of claim 10,

wherein determining the potential future contact comprises determining one or more of the following factors:

(i) a distance between the particular obstacle and the location associated with the account and (ii) a time period until the potential future contact occurs, and

wherein determining a respective feedback process to initiate feedback that indicates the potential future contact with the particular obstacle comprises determining a respective feedback process to initiate feedback that indicates one or more of the determined factors.

12. The method of claim 1, further comprising:

based on the received environment information for the location associated with the account, determining, by the control system, a potential future contact with a particular obstacle; and

in response to determining the potential future contact with the particular obstacle, the control system directing the aerial vehicle to move the particular obstacle to a different location that is selected based on the potential future contact being avoidable.

13. The method of claim 1, wherein the associated feedback system comprises at least a computing device associated with the account, wherein initiating feedback in accordance with the particular assistance mode via the associated feedback system comprises initiate feedback in accordance with the particular assistance mode via the associated computing device, the method further comprising:

determining, by the control system, that the associated computing device is located within a threshold distance away from the location associated with the account, wherein applying the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated computing system is at least in response to determining that the associated computing device is located within the threshold distance away from the location associated with the account.

14. The method of claim 1, wherein the associated feedback system comprises a display device, and wherein applying the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system comprises:

based on the received environment information for the location associated with the account, determining visual feedback information and responsively directing the associated feedback system to display, on the display device, the determined visual feedback information.

15. The method of claim 14, wherein the associated feedback system further comprises a video-capture device configured to provide video data corresponding to the location associated with the account, and wherein directing the associated feedback system to display, on the display device, the determined visual feedback information comprises:

directing the associated feedback system to display, on the display device, the determined visual feedback information as being overlaid on at least a portion of the provided video data.

16. The method of claim 14, wherein the determined visual feedback information comprises one or more of the following forms of information:

(i) the received environment information, (ii) at least one visual symbol that is representative of at least one instruction specifying at least one movement towards a particular destination, (iii) at least one visual symbol that is representative of a particular obstacle in an environment corresponding to the location associated with the account, and (iv) at least one visual symbol that is representative of a particular point-of-interest in an environment corresponding to the location associated with the account.

17. The method of claim 1, wherein the aerial vehicle is a first aerial vehicle, and wherein the particular assistance mode further specifies particular operations for a second aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to the location associated with the account, the method further comprising:

transmitting, by the control system to the second aerial vehicle, at least the indication of the particular operations corresponding to the particular assistance mode; and

receiving, by the control system, other environment information for the location associated with the account, wherein the other received environment information is based on the particular operations corresponding to the particular assistance mode, and wherein applying the specified feedback processes is also based on the other received environment information.

18. The method of claim 1, wherein the account is a first account, wherein the particular assistance mode is a first assistance mode, and wherein the received environment information comprises first received environment information, the method further comprising:

determining, by the control system, a second assistance mode associated with a second account, wherein the second assistance mode specifies (i) particular operations for the aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the second account and (ii) feedback processes to provide feedback, via a feedback system associated with the second account, that corresponds to respective environment information;

transmitting, by the control system to the aerial vehicle, at least an indication of the particular operations corresponding to the second assistance mode;

receiving, by the control system, second environment information for the location associated with the second account, wherein the received second environment information is based on the particular operations corresponding to the second assistance mode; and

based on the received second environment information for the location associated with the second account, applying, by the control system, the specified feedback processes to initiate feedback in accordance with the second assistance mode via the feedback system associated with the second account.

19. A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a control system to perform functions comprising:

determining a particular assistance mode associated with an account, wherein the particular assistance mode specifies (i) particular operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information;

transmitting, to the aerial vehicle, at least an indication of the particular operations corresponding to the particular assistance mode;

receiving environment information for the location associated with the account, wherein the received environment information is based on the particular operations corresponding to the particular assistance mode; and

based on the received environment information for the location associated with the account, applying the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.

20. A control system comprising:

one or more processors;

a non-transitory computer readable medium; and

program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to;

determine a particular assistance mode associated with an account, wherein the particular assistance mode specifies (i) particular operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information;

transmit, to the aerial vehicle, at least an indication of the particular operations corresponding to the particular assistance mode;

receive environment information for the location associated with the account, wherein the received environment information is based on the particular operations corresponding to the particular assistance mode; and

based on the received environment information for the location associated with the account, apply the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.