System and method for controller-free user drone interaction

• US 10,409,276 B2
• Filed: 07/20/2017
• Issued: 09/10/2019
• Est. Priority Date: 12/21/2016
• Status: Active Grant

• Current Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Wang, Mengqiu, Lu, Jia, Zhang, Tong, Liu, Lixin
• Primary Examiner(s): To, Tuan C
• Application Number:
  US15/655,024
• Publication Number:
  US 20180173220A1
• Time in Patent Office: 782 Days
• CPC Class Codes: B64C 27/006 Safety devices
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 70/00 Launching, take-off or landing arrangements
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

System and method for controlling an aerial system, without physical interaction with a separate remote device, based on sensed user expressions. User expressions may include thought, voice, facial expressions, and/gestures. User expressions may be sensed by sensors associated with the aerial device or a remote device.

Claims

1. An aerial system, comprising:

a body;

a lift mechanism coupled to the body;

an orientation sensor coupled to the body for sensing an orientation of the body;

an optical system controllably mounted to the body by an actuation system; and

a processing system coupled to the lift mechanism, the optical system, and the actuation system, the processing system configured to;

detect a release event in which the aerial system is released from being physically held by a user;

enter a hover idle state in which the aerial system searches for a target person;

recognize the target person;

track the target person and scan for user expressions;

identify a hand of the target person;

detect a first predefined user expression including an open palm gesture of the hand of the target person and responsively track a movement of the hand in the open palm gesture and execute a first corresponding event including operating the lift mechanism to fly the aerial system in the direction of the tracked movement of the hand in the open palm gesture while maintaining the target person within a field of view of the optical system;

detect a second predefined user expression including a “

V”

sign gesture by the hand of the target person and responsively execute a second corresponding event including operating the optical system to take a picture of the target person;

enter the hover idle state upon completing the second corresponding event;

detect a standby event including a change in an orientation vector of the body upon sensing a change in an orientation sensor signal received from the orientation sensor; and

responsively operate the lift mechanism in a standby mode.

2. An aerial system, as set forth in claim 1, further comprising additional sensor(s) associated with the aerial system configured to sense user features and to send data related to the sensed user features to the processing system, the processing system being configured to scan for user expressions as a function of the data related to the sensed user features.

3. An aerial system, as set forth in claim 2, wherein the additional sensors are coupled to the body.

4. An aerial system, as set forth in claim 2, wherein the additional sensors are located at a remote device, the remote device configured to send the data related to the sensed user features to the aerial system for processing by the processing system.

5. An aerial system, as set forth in claim 2, wherein the processing system utilizes facial recognition and/or gesture recognition to identify user expressions as a function of the data.

6. An aerial system, as set forth in claim 5, wherein the processing system is configured to manage a release and hover process, the processing system being configured to control the aerial system to one of:

(1) rotate and scan for the target person and (2) wait for the target person to enter a file of vision of the optical system.

7. An aerial system, as set forth in claim 6, wherein the processing system scans for user expressions after the target person has been identified.

8. An aerial system, as set forth in claim 7, wherein the target person is identified by comparing data received from the additional sensor(s) with pre-registered data.

9. An aerial system, as set forth in claim 8, wherein the processing system in further configured to identify specific user expressions and to perform an associated task.

10. An aerial system, as set forth in claim 9, wherein the user expressions are gestures, the processing system being configured to identify one or more of the following gestures:

thumb up, thumb down, open palm, fist, and victory gesture.

11. An aerial system, as set forth in claim 1, wherein the processing system is configured to identify and track movement of the open palm gesture relative to a face of the target person.

12. An aerial system, as set forth in claim 11, wherein the processing system is configured to move the aerial system as a function of the relative movement of the open palm gesture relative to the face of the target person.

13. An aerial system, as set forth in claim 8, wherein the associated task includes one of the following:

take a picture or snapshot, initiate shooting video, initiate auto-follow, starting auto-capturing routine, terminate routine, land, and auto-capture.

14. An aerial system, as set forth in claim 2, wherein the additional sensor(s) for sensing user features includes one or more camera(s) of the optical system and/or a microphone, wherein the user expressions are facial expressions and/or gestures and/or voice commands.

15. A method for controlling an aerial system including a body, a lift mechanism coupled to the body, an orientation sensor coupled to the body for sensing an orientation of the body, an optical system, and a processing system, the method comprising the processing system performing the steps of:

detecting a release event in which the aerial system is released from being physically held by a user;

operating the lift mechanism to enter a hover idle state in which the aerial system searches for a target person;

recognizing the target person;

tracking the target person and scanning for user expressions;

identifying a hand of the target person;

detecting a first predefined user expression including an open palm gesture of the hand of the target person and responsively tracking a movement of the hand in the open palm gesture and executing a first corresponding event including operating the lift mechanism to fly the aerial system in a direction of the tracked movement of the hand in the open palm gesture while maintaining the target person within a field of view of the optical system;

detecting a second predefined user expression including a “

V”

sign gesture by the hand of the target person and responsively execute a corresponding second event including operating the optical system to take a picture of the target person;

entering the hover idle state upon completing the second corresponding event;

detecting a standby event including a change in an orientation vector of the body upon sensing a change in an orientation sensor signal received from the orientation sensor; and

responsively operating the lift mechanism in a standby mode.

16. A method, as set forth in claim 15, including the steps of:

sensing user features, using one or more additional sensors and sending data related to the sensed user features to the processing system, and

scanning for user expressions as a function of the data related to the sensed user features.

17. A method, as set forth in claim 16, wherein the additional sensors are coupled to the body.

18. A method, as set forth in claim 16, wherein the additional sensors are located at a remote device, the remote device configured to send the data related to the sensed user features to the aerial system for processing by the processing system.

19. A method, as set forth in claim 16, including the step of utilizing, by the processing system, facial recognition and/or gesture recognition to identify user expressions as a function of the data.

20. A method, as set forth in claim 19, including the steps of managing, by the processing system, release and hover process and controlling the aerial system to one of:

(1) rotate and scan for the target person and (2) wait for the target person to enter a file of vision of the optical system.

21. A method, as set forth in claim 20, including the step of scanning, by the processing system, for user expressions after the target person has been identified.

22. A method, as set forth in claim 21, wherein the target person is identified by comparing data received from the additional sensor(s) with pre-registered data.

23. A method, as set forth in claim 22, including the step of identifying, by the processing system, specific user expressions and to perform an associated task.

24. A method, as set forth in claim 23, wherein the user expressions are gestures, the processing system being configured to identify one or more of the following gestures:

thumb up, thumb down, open palm, fist, and victory gesture.

25. A method, as set forth in claim 15, including the steps of identifying and tracking movement of the open palm gesture relative to a face of the target person.

26. A method, as set forth in claim 25, including the step of moving the aerial system as a function of the relative movement of the open palm gesture relative to the face of the target person.

27. A method, as set forth in claim 23, wherein the associated task includes one of the following:

take a picture or snapshot, initiate shooting video, initiate auto-follow, starting auto-capturing routine, terminate routine, land, and auto-capture.

28. A method, as set forth in claim 16, wherein the additional sensor(s) for sensing user features includes one or more camera(s) of the optical system and/or a microphone, wherein the user expressions are facial expressions and/or gestures and/or voice commands.