INTELLIGENTLY ASSISTED IOT ENDPOINT DEVICE

• US 20180232250A1
• Filed: 02/16/2017
• Published: 08/16/2018
• Est. Priority Date: 02/16/2017
• Status: Active Grant

• Current Assignee: San Jacinto Digital Services
• Original Assignee: Sanctum Solutions Inc.
• Inventors: Stephens, Noel Shepard
• Granted Patent: US 10,140,147 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/4411 Configuring for operating with peripheral devices; Loading of device drivers
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 84/18 Self-organising networks, e.g. ad-hoc networks or sensor networks
• CPC Class Codes: H04W 88/04 adapted for relaying to or from another terminal or user
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Intelligently assisted IoT endpoint devices are disclosed. For example, an endpoint device determined to have an active network connection redirects input/output data from a physical sensor to a network interface. First redirected data of the input/output data is received, over a network by a virtual sensor of a virtualized endpoint service then output to a virtualized endpoint engine that resulting in conversion into first converted data, which is sent to an endpoint control service. The virtualized endpoint engine receives a first command from the endpoint control service then sends the first command to a virtual interactive element controller which converts the first command into a second command compatible with an interactive element of the endpoint device which performs a task after receiving the second command. The virtual sensor receives second redirected data collected by the physical sensor different from first redirected data as a result of the task.

Claims

1. A system comprising:

a plurality of endpoint devices connected to a network, the plurality of endpoint devices including a first endpoint device and a second endpoint device, wherein the first endpoint device includes a first plurality of physical sensors including a first physical sensor and a second physical sensor and the first endpoint device includes a first plurality of interactive elements including a first interactive element and a second interactive element;

a first endpoint processor executing on the first endpoint device to;

determine that the first endpoint device has an active network connection;

responsive to determining that the first endpoint device has an active network connection, redirect input/output data from the first plurality of sensors to a first network interface;

one or more processors, wherein the one or more processors are located over the network from the plurality of endpoint devices;

a first virtualized endpoint service associated with the first endpoint device executing on the one or more processors including;

a first plurality of virtual sensors each respectively associated with a physical sensor of the first plurality of physical sensors including a first virtual sensor associated with the first physical sensor, and a second virtual sensor associated with the second physical sensor;

a first plurality of virtual interactive element controllers each respectively associated with an interactive element of the first plurality of interactive elements, including a first virtual interactive element controller associated with the first interactive element and a second virtual interactive element controller associated with the second interactive element;

a first virtualized endpoint engine;

the first virtualized endpoint service executing to;

receive, over the network by the first virtual sensor, first redirected data collected by the first physical sensor, wherein the first redirected data is part of the input/output data redirected by the first endpoint device;

output, by the first virtual sensor, the first redirected data to the first virtualized endpoint engine;

convert, by the first virtualized endpoint engine, the first redirected data into a first converted data compatible with an endpoint control service;

send, by the first virtualized endpoint engine, the first converted data to the endpoint control service;

receive, by the first virtualized endpoint engine, from the endpoint control service, a first command;

send, by the first virtualized endpoint engine, the first command to the first virtual interactive element controller;

convert, by the first virtual interactive element controller, the first command into a second command compatible with the first interactive element;

send the second command, over the network, causing the first interactive element to perform a first task; and

receive, over the network by the first virtual sensor, second redirected data collected by the first physical sensor, wherein the second redirected data is different from the first redirected data as a result of performing the first task.

2. The system of claim 1, wherein the first task results in a physical change in one of the first endpoint device and a local environment surrounding the first endpoint device.

3. The system of claim 2, wherein the first interactive element includes at least one of, a propeller, a wheel, a tread track, a motor, a jet, a magnetic levitation system, a gyroscope stabilizer, a braking system, and a steering system.

4. The system of claim 2, wherein the first interactive element controls at least one of the first physical sensor and the second physical sensor and the first task changes at least one of an orientation, a focus, and a mode of the first physical sensor.

5. The system of claim 1, wherein the second redirected data is an updated version of the first redirected data.

6. The system of claim 1, wherein the first physical sensor is one of a camera, a microphone, an accelerometer, an antenna, a battery life sensor, a compass, a Global Positioning System sensor, a gyroscope, and a radio.

7. The system of claim 1, wherein at least one of the first endpoint device and the second endpoint device is connected to the network with a wireless connection.

8. The system of claim 7, wherein the first endpoint device is connected to the network via a first connection to the second endpoint device, and the second endpoint device relays information between the first connection and the network.

9. The system of claim 8, wherein the first endpoint device connects to the second endpoint device responsive to losing a direct connection to the network.

10. The system of claim 9, wherein the first endpoint device communicates with the second endpoint device via short distance wireless communications.

11. The system of claim 1, wherein the first virtualized endpoint service is exclusively associated with one endpoint device of the plurality of endpoint devices.

12. The system of claim 1, wherein the first virtualized endpoint service executes in an isolated guest in a cloud environment.

13. The system of claim 12, wherein the cloud environment is one of a public cloud environment and a private cloud environment.

14. The system of claim 12, wherein the isolated guest is one of a virtual machine and a container.

15. The system of claim 1, wherein the endpoint control service executes on a plurality of isolated guests.

16. The system of claim 15, wherein a quantity of isolated guests in the plurality of isolated guests scales based on the computing requirements for interpreting a plurality of converted data from a plurality of virtualized endpoint services and an issuing of a plurality of commands.

17. The system of claim 1, wherein communications between the first virtualized endpoint service and the first endpoint device are secured by encryption and obfuscation.

18. The system of claim 1, wherein the first endpoint device is at least one of a robotic vehicle, a drone, and a robotic device.

19. The system of claim 1, wherein the first endpoint device forwards sensor data from the first physical sensor and the second physical sensor to the first virtualized endpoint service for storage.

20. The system of claim 19, wherein the first endpoint device deletes any sensor data on the first endpoint device upon confirmation of transmission to the first virtualized endpoint service.

21. The system of claim 1, wherein the first virtualized endpoint service is communicatively coupled with an interface for a user to control the first endpoint device.

22. The system of claim 1, wherein the first endpoint device includes safety components that control the first endpoint device in response to a loss of network connectivity.

23. The system of claim 1, wherein a second virtualized endpoint service controls the second endpoint device.

24. The system of claim 23, wherein the first virtualized endpoint service includes a first application programming interface and the second virtualized endpoint service includes a second application programming interface.

25. The system of claim 24, wherein a user interface controls the first endpoint device with the first application programming interface and controls the second endpoint device with the second application programming interface.

26. The system of claim 1, wherein the first endpoint processor determines that the first endpoint device lacks the active network connection.

27. The system of claim 26, wherein the first endpoint processor determines that the first endpoint device lacks the active network connection due to at least one of a latency of transmissions to the network and a wireless signal strength.

28. The system of claim 26, wherein responsive to determining that the first endpoint device lacks the active network connection, the first endpoint processor stops redirecting the input/output data from the first plurality of sensors to the first network interface.

29. The system of claim 28, wherein responsive to a stoppage of redirecting the input/output data, the first endpoint device:

processes a non-redirected data into a third command compatible with the second interactive element; and

sends the third command to the second interactive element causing the second interactive element to perform a second task.

30. The system of claim 28, wherein responsive to a stoppage of redirecting the input/output data, the first endpoint device continues to perform the first task.

31. The system of claim 28, wherein responsive to a stoppage of redirecting the input/output data, the first endpoint device enters a safety mode until the active network is re-established, wherein the safety mode is configured to at least one of prevent and mitigate physical damage to the first endpoint device.

32. The system of claim 28, wherein the first endpoint device temporarily stores the input/output data from the first plurality of sensors in a local buffer.

33. The system of claim 1, wherein the first plurality of virtual sensors includes a respective virtual sensor associated with each physical sensor of the first plurality of physical sensors whose input/output data is redirected to the first network interface.

34. The system of claim 33, wherein the first virtual sensor appears to be the first physical sensor to the first virtualized endpoint engine.

35. The system of claim 1, wherein the first plurality of virtual interactive element controllers includes a respective virtual interactive element controller associated with each interactive element of the first plurality of interactive elements.

36. A method comprising:

determining that an endpoint device has an active network connection;

responsive to determining that the endpoint device has an active network connection, redirecting input/output data from a plurality of sensors of the endpoint device including a first physical sensor to a network interface;

receiving, over a network by a first virtual sensor of a virtualized endpoint service associated with the first physical sensor, first redirected data collected by the first physical sensor, wherein the first redirected data is part of the input/output data redirected by the first endpoint device;

outputting, by the first virtual sensor, the first redirected data to a first virtualized endpoint engine;

converting, by the virtualized endpoint engine, the first redirected data into a first converted data compatible with an endpoint control service;

sending, by the virtualized endpoint engine, the first converted data to the endpoint control service;

receiving, by the virtualized endpoint engine from the endpoint control service, a first command;

sending, by the virtualized endpoint engine, the first command to a first virtual interactive element controller associated with a first interactive element of the endpoint device;

converting, by the virtual interactive element controller, the first command into a second command compatible with the first interactive element;

sending the second command, over the network, causing the first interactive element to perform a first task; and

receive, over the network by the first virtual sensor, second redirected data collected by the first physical sensor, wherein the second redirected data is different from the first redirected data as a result of performing the first task.

37. A computer-readable non-transitory storage medium storing executable instructions, which when executed by a computer system, cause the computer system to:

determine that an endpoint device has an active network connection;

responsive to determining that the endpoint device has an active network connection, redirect input/output data from a plurality of sensors of the endpoint device including a first physical sensor to a network interface;

receive, over a network by a first virtual sensor of a virtualized endpoint service associated with the first physical sensor, first redirected data collected by the first physical sensor, wherein the first redirected data is part of the input/output data redirected by the first endpoint device;

output, by the first virtual sensor, the first redirected data to a first virtualized endpoint engine;

convert, by the virtualized endpoint engine, the first redirected data into a first converted data compatible with an endpoint control service;

sending, by the virtualized endpoint engine, the first converted data to the endpoint control service;

receive, by the virtualized endpoint engine from the endpoint control service, a first command;

send, by the virtualized endpoint engine, the first command to a first virtual interactive element controller associated with a first interactive element of the endpoint device;

convert, by the virtual interactive element controller, the first command into a second command compatible with the first interactive element;

send the second command, over the network, causing the first interactive element to perform a first task; and

receive, over the network by the first virtual sensor, second redirected data collected by the first physical sensor, wherein the second redirected data is different from the first redirected data as a result of performing the first task.