Methods and systems for collaborative control of a remote vehicle

• US 10,416,666 B2
• Filed: 02/23/2018
• Issued: 09/17/2019
• Est. Priority Date: 03/26/2014
• Status: Active Grant

• Current Assignee: Unanimous A.I., Inc.
• Original Assignee: Unanimous A I Incorporated
• Inventors: Rosenberg, Louis B.
• Primary Examiner(s): Hailu, Tadesse
• Application Number:
  US15/904,239
• Publication Number:
  US 20180181117A1
• Time in Patent Office: 571 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06

Abstract

Systems and methods are for enabling a group of individuals, each using an individual computing device, to collaboratively control a vehicle in real-time as a unified intelligence. The collaboration system comprises a plurality of computing devices, each of the devices being used by an individual user, each of the computing devices enabling its user to contribute to the emerging real-time group-wise intent. A collaboration server is disclosed that moderates the closed-loop system, enabling convergence upon a unified group intent. Control commands are repeatedly sent to the vehicle based on the determined group intent.

Claims

1. A real-time collaborative vehicle control system comprising:

a plurality of computing devices each comprising a communications infrastructure coupled to each of a processor, a memory, and a collaborative intent application stored on each memory and configured to run on each processor, and a user interface coupled to a display and configured to receive input from a user;

a collaborative server in networked communication with the plurality of computing devices, the collaboration server including a processor and a memory;

a vehicle controller configured to receive commands from the collaborative server; and

a vehicle operatively controlled by the vehicle controller, wherein the control system is configured to perform the following steps, wherein computing device steps are performed substantially simultaneously on each of the plurality of computing devices;

display, on each of the computing devices, vehicle control information, and a collaboratively controlled graphical indication having a coordinate location in relation to the vehicle control information;

receive, on each of the computing devices, repeatedly in real-time, user input of a user intent vector through the user interface, the user intent vector having a direction in relation to the user control information and a magnitude,

send, repeatedly in real-time, the user intent vector to the collaboration server,

responsively determine, repeatedly in real-time by the collaboration server, a group intent vector from the plurality of received user intent vectors, and

determine, repeatedly in real-time by the collaboration server, an updated coordinate location based on the group intent vector;

determine, repeatedly in real-time by the collaboration server, at least one vehicle command in response to the updated coordinate location;

send, repeatedly in real-time by the collaboration server, the at least one a vehicle command to the vehicle controller, whereby the vehicle operates in response to the at least one vehicle command;

send, repeatedly in real-time by the collaboration server, the updated coordinate location to the plurality of computing devices; and

update, repeatedly in real-time by the computing devices, the location of the collaboratively controlled graphical indication using the received updated coordinate location, whereby a closed-loop system is formed between the collaboration server and each collaborative intent application.

2. The real-time collaborative vehicle control system of claim 1, wherein the vehicle is a flying drone vehicle.

3. The real-time collaborative vehicle control system of claim 2, wherein the at least one control axis includes at least one of a control axis indicating forward/backward motion of the vehicle and a control axis indicating leftward/rightward motion of the vehicle.

4. The real-time collaborative vehicle control system of claim 2, wherein the collaboratively controlled graphical indication is influenced by a simulated spring force, said simulated spring force configured to pull the displayed collaboratively controlled graphical indication towards an origin position on the at least one control axis.

5. The real-time collaborative vehicle control system of claim 1, wherein the vehicle control information includes at least one control axis, wherein a location of the graphical indication relative to the at least one control axis indicates the at least one command sent to the vehicle.

6. The real-time collaborative vehicle control system of claim 5, wherein the first control axis and the second control axis intersect at an origin location.

7. The real-time collaborative vehicle control system of claim 5, wherein the at least one control axis includes a first control axis oriented vertically and a second control axis oriented horizontally.

8. The real-time collaborative vehicle control system of claim 1, the system further comprising a camera coupled to the vehicle and operatively controlled by the vehicle controller.

9. The real-time collaborative vehicle control system of claim 8, wherein the camera is configured to stream live video to the plurality of computing devices.

10. The real-time collaborative vehicle control system of claim 8, wherein each of the plurality of computing devices displays the live video on a same display as the collaboratively controlled graphical indication.

11. The real-time collaborative vehicle control system of claim 1, wherein the at least one vehicle command includes at least one command for camera operation.

12. The real-time collaborative vehicle control system of claim 1, wherein the at least one command for camera operation includes at least one of forward/backward tilt and leftward/rightward pan.

13. The real-time collaborative vehicle control system of claim 1, wherein the system is configured to display a graphical indication of the user input.

14. The real-time collaborative vehicle control system of claim 13, wherein the graphical indication is a U-shaped magnet icon.

15. The real-time collaborative vehicle control system of claim 14, wherein the system is configured to change at least one of the size, orientation, and location of the magnet icon based on the user input.

16. The real-time collaborative vehicle control system of claim 1, wherein the collaboratively controlled graphical indication is displayed on a control area corresponding to a plan area of operation of the vehicle.

17. The real-time collaborative vehicle control system of claim 16, wherein the collaboratively controlled graphical indication represents a location of the vehicle in the control area.

18. The real-time collaborative vehicle control system of claim 16, the control area further including a display of plan coordinate axes.

19. The real-time collaborative vehicle control system of claim 1, wherein the collaboratively controlled graphical indication is an icon indicating the orientation of the vehicle.

20. The real-time collaborative vehicle control system of claim 1, wherein the collaboratively controlled graphical indication has at least one of a simulated mass value and a simulated damping value, the at least one of the simulated mass value and the simulated damping value selected to slow a displayed collaboratively controlled motion of the collaboratively controlled graphical indication to better match dynamics of the vehicle.