Motion and image-based control system

• US 10,768,639 B1
• Filed: 06/30/2017
• Issued: 09/08/2020
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Meisenholder, David, Horowitz, Steven
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US15/640,061
• Time in Patent Office: 1,166 Days
• Field of Search: 382103, 382154, 382190, 382284, 382285, 382294, 706 48, 345419, 345420
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/23 for providing telephone services
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G05D 1/12 Target-seeking control
• CPC Class Codes: G05D 1/223 Command input arrangements on the remote controller, e.g. joysticks or touch screens
• CPC Class Codes: G05D 1/652 Take-off delivering or retrieving payloads G05D1/667
• CPC Class Codes: G05D 1/686 Maintaining a relative position with respect to moving targets, e.g. following animals or humans for pointing payloads towards targets G05D1/689; involving controlling the position or course of two or more vehicles G05D1/69
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command

Abstract

Systems, devices, media, and methods are presented for detecting and interpreting motion of a device and a remote object to control operations of the device. The systems and methods identify a sensor input within a drone. The sensor input indicates movement of the drone within a three dimensional space. The systems and methods determine one or more movement attributes from the sensor input and, in response to the one or more movement attributes, selects one or more maneuvers corresponding to at least one movement attribute. The system and methods then execute the one or more maneuvers by controlling one or more drone control components to move the drone within the three dimensional space.

Claims

1. A method, comprising:

identifying a sensor input within a drone, the sensor input indicating movement of the drone within a three dimensional space, the drone comprising one or more sensors and one or more drone control components;

upon receiving the sensor input indicating that the drone moved within the three dimensional space, determining that one or more movement attributes from the sensor input are associated with the drone having been physically displaced by a user, the determining comprising determining that a combination of movement component values, representing movement of the drone within the three-dimensional space, received within the sensor input, matches a set of movement attributes associated with the drone having been physically displaced by the user;

in response to the determining that the one or more movement attributes are associated with the drone having been physically displaced by the user, selecting one or more maneuvers corresponding to at least one movement attribute of the one or more movement attributes; and

executing the one or more maneuvers by controlling the one or more drone control components to move the drone within the three dimensional space.

2. The method of claim 1, wherein the drone comprises an image capture device and the method further comprises:

determining a field of view of the image capture device;

determining at least one point of focus of the image capture device within the field of view; and

executing the one or more maneuvers with respect to at least one of the field of view and the at least one point of focus of the image capture device.

3. The method of claim 1, wherein the drone is physically displaced by the user based on an initial toss, roll or spin action performed by the user, wherein the set of movement attributes that matches the combination of movement component values includes values associated with the initial toss, roll or spin action, further comprising:

determining a reference point within the three dimensional space; and

executing the one or more maneuvers in relation to the reference point.

4. The method of claim 3, wherein the drone comprises an image capture device and the method further comprises:

determining a field of view of the image capture device;

controlling the one or more drone control components to position the reference point within the field of view of the image capture device; and

executing the one or more maneuvers in relation to the reference point while maintaining at least a portion of the reference point within the field of view of the image capture device.

5. The method of claim 1, wherein the drone comprises an image capture device and the method further comprises:

determining a field of view of the image capture device;

identifying an area of interest within the field of view, the area of interest including at least one object of interest;

tracking the at least one object of interest; and

executing the one or more maneuvers in relation to the at least one object of interest.

6. The method of claim 5, wherein the object of interest is a person at least partially depicted within the field of view of the image capture device and the method further comprises:

determining a characteristic of the person depicted within the field of view;

identifying a position corresponding to the characteristic;

determining a current position of the drone relative to the person depicted within the field of view; and

controlling the one or more drone control component to change the current position of the drone to the position corresponding to the characteristic.

7. The method of claim 6, wherein changing the current position of the drone to the position corresponding to the characteristic further comprises:

determining at least one maneuver associated with one or more of the characteristic and the position corresponding to the characteristic; and

controlling the one or more drone control component to execute the at least one maneuver while changing the current position of the drone to the position corresponding to the characteristic.

8. The method of claim 1, wherein the drone comprises an image capture device, the one or more maneuvers is a first maneuver, and the method further comprises:

identifying two or more objects of interest within a field of view of the image capture device;

executing the first maneuver with respect to a first object of interest of the two or more objects of interest;

selecting a second object of interest of the two or more objects of interest;

selecting a second maneuver corresponding to the second object of interest; and

executing the second maneuver with respect to the second object of interest, the second maneuver executed by controlling the one or more drone control components to move the drone within the three dimensional space.

9. The method of claim 1, wherein the drone comprises an image capture device, the one or more maneuvers is a first maneuver, and the method further comprises:

identifying the user within a field of view of the image capture device in response to determining that the one or more movement attributes from the sensor input are associated with the drone having been physically displaced by the user;

detecting, based on one or more images captured by the image capture device, a gesture performed by the user; and

executing a second maneuver corresponding to the detected gesture.

10. The method of claim 1, wherein the drone comprises an image capture device and the method further comprises:

identifying a first object of interest and a second object of interest within a field of view of the image capture device;

receiving a communication message from a client device remote from the drone, the communication message comprising an attribute identifying the user;

determining the first object of interest corresponds to the user identified in the communication message; and

executing the one or more maneuver relative to the first object of interest maintaining the first object of interest within the field of view of the image capture device while executing the one or more maneuver.

11. A system comprising:

one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying a sensor input within a drone, the sensor input indicating movement of the drone within a three dimensional space, the drone comprising one or more sensors and one or more drone control components;

upon receiving the sensor input indicating that the drone has moved within the three dimensional space, determining that one or more movement attributes from the sensor input are associated with the drone having been physically displaced by a user, the determining comprising determining that a combination of movement component values, representing movement of the drone within the three-dimensional space, received within the sensor input, matches a set of movement attributes associated with the drone having been physically displaced by the user;

in response to the determining that the one or more movement attributes are associated with the drone having been physically displaced by the user, selecting one or more maneuvers corresponding to at least one movement attribute of the one or more movement attributes; and

executing the one or more maneuvers by controlling the one or more drone control components to move the drone within the three dimensional space.

12. The system of claim 11, wherein the drone comprises an image capture device and the operations further comprise:

determining a field of view of the image capture device;

determining at least one point of focus of the image capture device within the field of view; and

executing the one or more maneuvers with respect to at least one of the field of view and the at least one point of focus of the image capture device.

13. The system of claim 11, wherein the drone is physically displaced by the user based on an initial toss, roll or spin action performed by the user, and wherein the operations further comprise:

determining a reference point within the three dimensional space;

determining a field of view of an image capture device coupled to the drone;

controlling the one or more drone control components to position the reference point within the field of view of the image capture device; and

executing the one or more maneuvers in relation to the reference point while maintaining at least a portion of the reference point within the field of view of the image capture device.

14. The system of claim 11, wherein the drone comprises an image capture device and the operations further comprise:

determining a field of view of the image capture device;

identifying the user within the field of view in response to determining that the one or more movement attributes from the sensor input are associated with the drone having been physically displaced by the user;

detecting, based on one or more images captured by the image capture device, a gesture performed by the user; and

executing the one or more maneuvers corresponding to the gesture.

15. The system of claim 14, wherein the object of interest is a person at least partially depicted within the field of view of the image capture device and the operations further comprise:

determining a characteristic of the person depicted within the field of view;

identifying a position corresponding to the characteristic;

determining a current position of the drone relative to the person depicted within the field of view; and

controlling the one or more drone control component to change the current position of the drone to the position corresponding to the characteristic.

16. The system of claim 15, wherein changing the current position of the drone to the position corresponding to the characteristic further comprise:

determining at least one maneuver associated with one or more of the characteristic and the position corresponding to the characteristic; and

controlling the one or more drone control component to execute the at least one maneuver while changing the current position of the drone to the position corresponding to the characteristic.

17. The system of claim 11, wherein the drone comprises an image capture device, the one or more maneuvers is a first maneuver, and the operations further comprise:

identifying two or more objects of interest within a field of view of the image capture device;

executing the first maneuver with respect to a first object of interest of the two or more objects of interest;

selecting a second object of interest of the two or more objects of interest;

selecting a second maneuver corresponding to the second object of interest; and

executing the second maneuver with respect to the second object of interest, the second maneuver executed by controlling the one or more drone control components to move the drone within the three dimensional space.

18. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

identifying a sensor input within a drone, the sensor input indicating movement of the drone within a three dimensional space, the drone comprising one or more sensors and one or more drone control components;

upon receiving the sensor input indicating that the drone has moved within the three dimensional space, determining that one or more movement attributes from the sensor input are associated with the drone having been physically displaced by a user, the determining comprising determining that a combination of movement component values, representing movement of the drone within the three-dimensional space, received within the sensor input, matches a set of movement attributes associated with the drone having been physically displaced by the user;

in response to the determining that the one or more movement attributes are associated with the drone having been physically displaced by the user, selecting one or more maneuvers corresponding to at least one movement attribute of the one or more movement attributes; and

executing the one or more maneuvers by controlling the one or more drone control components to move the drone within the three dimensional space.

19. The non-transitory processor-readable storage medium of claim 18, wherein the drone comprises an image capture device and the operations further comprise:

determining a field of view of the image capture device;

determining at least one point of focus of the image capture device within the field of view; and

executing the one or more maneuvers with respect to at least one of the field of view and the at least one point of focus of the image capture device.

20. The non-transitory processor-readable storage medium of claim 18, wherein the operations further comprise:

determining a reference point within the three dimensional space;

determining a field of view of an image capture device coupled to the drone;

controlling the one or more drone control components to position the reference point within the field of view of the image capture device; and

executing the one or more maneuvers in relation to the reference point while maintaining at least a portion of the reference point within the field of view of the image capture device.