Control of an aerial drone using recognized gestures

• US 10,168,700 B2
• Filed: 02/11/2016
• Issued: 01/01/2019
• Est. Priority Date: 02/11/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gordon, Michael S., Kozloski, James R., Kundu, Ashish, Malkin, Peter K., Pickover, Clifford A.
• Primary Examiner(s): Hutchinson, Alan D
• Application Number:
  US15/041,313
• Publication Number:
  US 20170235308A1
• Time in Patent Office: 1,055 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 50/11 using internal combustion piston engines
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target

Abstract

A method, system, and/or computer program product controls movement and adjusts operations of an aerial drone. A drone camera observes an aerial maneuver physical gesture by a user. The aerial drone then performs an aerial maneuver that correlates to the aerial maneuver physical gesture. The drone camera observes the user performing a physical action. One or more processors associate the physical action with a particular type of activity. A drone on-board computer adjusts an operation of an aerial drone based on the particular type of activity.

Claims

1. A method of controlling movement and adjusting operations of an aerial drone, the method comprising:

observing, by a drone camera mounted on an aerial drone, an aerial maneuver physical gesture by a user;

performing, by the aerial drone, an aerial maneuver that correlates to the aerial maneuver physical gesture in response to the drone camera observing the aerial maneuver physical gesture by the user;

observing, by the drone camera, the user performing a physical action;

associating, by one or more processors, the physical action with a particular type of recreational activity;

adjusting, by a drone on-board computer, operations of the aerial drone based on the particular type of recreational activity;

placing, by the drone on-board computer, the aerial drone in manual mode, wherein the aerial drone is manually controlled by the user when in the manual mode;

detecting, by the drone camera, a facial expression of the user;

comparing, by the one or more processors, the facial expression of the user to a database of facial expressions;

determining, by the one or more processors, that the facial expression of the user is a facial expression of fear; and

in response to determining that the facial expression of the user is the facial expression of fear, switching, by the drone on-board computer, the aerial drone from manual mode to autonomous mode, wherein the aerial drone operates autonomously while in autonomous mode.

2. The method of claim 1, further comprising:

analyzing, by one or more processors, a video feed from the drone camera, wherein said analyzing identifies a type of physical environment occupied by the user; and

determining, by one or more processors, the particular type of recreational activity based on the type of physical environment occupied by the user.

3. The method of claim 1, further comprising:

detecting, by the drone camera, a predefined anomalous physical action by the user; and

in response to detecting the predefined anomalous physical action by the user, adjusting, by the drone on-board computer, a type of monitoring performed by the drone camera.

4. The method of claim 1, further comprising:

detecting, by the drone camera, a predefined anomalous physical action by the user; and

in response to detecting the predefined anomalous physical action by the user, transmitting, by the drone on-board computer, a video feed from the drone camera to an emergency responder.

5. The method of claim 1, further comprising:

detecting, by the drone camera, a physical hazard at a location towards which the user is approaching; and

in response to detecting the physical hazard at the location towards which the user is approaching, transmitting, by a transmitter on the aerial drone, an alert to the user, wherein the alert warns the user of the physical hazard at the location towards which the user is approaching.

6. The method of claim 1, wherein the aerial drone is part of a drone swarm of multiple drones, and wherein the method further comprises:

detecting, by drone cameras mounted on the drone swarm, a first physical gesture of the user, wherein the first physical gesture identifies a drone subset of the drone swarm;

detecting, by drone cameras mounted on the drone subset, a second physical gesture of the user, wherein the second physical gesture is specific for an action to be performed by the drone subset; and

performing, by the drone subset, the action in response to said detecting the second physical gesture of the user.

7. The method of claim 1, further comprising:

detecting, by the drone camera, an authorization physical gesture by the user, wherein the authorization physical gesture is a predefined physical gesture that authorizes the user to take control of the aerial drone, and wherein the authorization physical gesture is a predefined sequence of hand and arm movements made by the user moving his hand and arm through the air; and

in response to the drone camera detecting the authorization physical gesture by the user, transferring, by the drone on-board computer on the aerial drone, control of the aerial drone to the user.

8. The method of claim 1, further comprising:

identifying, by one or more sensors, an aerial obstacle to the aerial drone; and

adjusting, by the drone on-board computer, a flight path of the aerial drone in order to avoid the aerial obstacle.

9. The method of claim 1, wherein said adjusting the operations of the aerial drone comprises:

switching, by the drone on-board computer, the aerial drone from a first mode to a second mode, wherein the first mode causes the aerial drone to perform a first set of actions, and the second mode causes the aerial drone to perform a second set of actions that is different from the first set of actions.

10. The method of claim 1, wherein lift for the aerial drone is initially being provided by an internal combustion engine on the aerial drone, and wherein the method further comprises:

observing, by the drone camera, a noise abatement gesture by the user, wherein the noise abatement gesture indicates a desire of the user for the aerial drone to operate at a reduced noise level; and

in response to observing the noise abatement gesture, powering off, by the drone on-board computer, the internal combustion engine and powering on, by the drone on-board computer, an electric motor on the aerial drone, wherein the electric motor provides the lift for the aerial drone.

11. The method of claim 1, further comprising:

detecting, by the drone camera, a set of lip movements of the user;

comparing, by the one or more processors, the set of lip movements of the user to a database of lip movements;

determining, by the one or more processors, that the set of lip movements of the user is a command for the aerial drone;

converting, by one or more processors, the command for the aerial drone into a set of executable instructions for the aerial drone; and

executing, by the drone on-board computer, the set of executable instructions in order to perform said adjusting operations of the aerial drone.

12. The method of claim 1, further comprising:

detecting, by the drone camera, a facial expression of the user;

comparing, by the one or more processors, the facial expression of the user to a database of facial expressions;

determining, by the one or more processors, that the facial expression of the user is a facial expression of pain; and

in response to determining that the facial expression of the user is the facial expression of pain, transmitting, from the aerial drone, a call for emergency medical help to an emergency health care provider.

13. The method of claim 1, further comprising:

observing, by the drone camera, a concealment aerial maneuver physical gesture by the user; and

in response to observing the concealment aerial maneuver physical gesture by the user, performing, by the aerial drone, a concealment aerial maneuver that moves the aerial drone to a location in which the aerial drone is concealed from the user.

14. A computer program product to control movement and to adjust operations of an aerial drone, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code readable and executable by a processor to perform a method comprising:

observing, by a drone camera mounted on an aerial drone, an aerial maneuver physical gesture by a user;

performing, by the aerial drone, an aerial maneuver that correlates to the aerial maneuver physical gesture in response to the drone camera observing the aerial maneuver physical gesture by the user;

observing, by the drone camera, the user performing a physical action;

associating the physical action with a particular type of recreational activity;

adjusting, by a drone on-board computer, operations of the aerial drone based on the particular type of recreational activity;

placing, by the drone on-board computer, the aerial drone in manual mode, wherein the aerial drone is manually controlled by the user when in the manual mode;

detecting, by the drone camera, a facial expression of the user;

comparing the facial expression of the user to a database of facial expressions;

determining that the facial expression of the user is a facial expression of fear; and

in response to determining that the facial expression of the user is the facial expression of fear, switching, by the drone on-board computer, the aerial drone from manual mode to autonomous mode, wherein the aerial drone operates autonomously while in autonomous mode.

15. The computer program product of claim 14, wherein the method further comprises:

analyzing a video feed from the drone camera, wherein said analyzing identifies a type of physical environment occupied by the user; and

determining the particular type of recreational activity based on the type of physical environment occupied by the user.

16. The computer program product of claim 14, wherein the method further comprises:

detecting, by the drone camera, a predefined anomalous physical action by the user; and

in response to detecting the predefined anomalous physical action by the user, adjusting, by the drone on-board computer, a type of monitoring performed by the drone camera.

17. The computer program product of claim 14, wherein the method further comprises:

detecting, by the drone camera, a predefined anomalous physical action by the user; and

in response to detecting the predefined anomalous physical action by the user, transmitting, by the drone on-board computer, a video feed from the drone camera to an emergency responder.

18. The computer program product of claim 14, wherein the program code is provided and executed as a service in a cloud environment.

19. A computer system comprising:

a processor, a computer readable memory, and a non-transitory computer readable storage medium;

first program instructions to observe, by a drone camera mounted on an aerial drone, an aerial maneuver physical gesture by a user;

second program instructions to perform, by the aerial drone, an aerial maneuver that correlates to the aerial maneuver physical gesture in response to the drone camera observing the aerial maneuver physical gesture by the user;

third program instructions to observe, by the drone camera, the user performing a physical action;

fourth program instructions to associate the physical action with a particular type of recreational activity;

fifth program instructions to adjust, by a drone on-board computer, an operation of the aerial drone based on the particular type of recreational activity;

sixth program instructions to place, by the drone on-board computer, the aerial drone in manual mode, wherein the aerial drone is manually controlled by the user when in the manual mode;

seventh program instructions to detect, by the drone camera, a facial expression of the user;

eighth program instructions to compare the facial expression of the user to a database of facial expressions;

ninth program instructions to determine that the facial expression of the user is a facial expression of fear; and

tenth program instructions to, in response to determining that the facial expression of the user is the facial expression of fear, switch, by the drone on-board computer, the aerial drone from manual mode to autonomous mode, wherein the aerial drone operates autonomously while in autonomous mode; and

wherein

the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, and tenth program instructions are stored on the non-transitory computer readable storage medium for execution by one or more processors via the computer readable memory.