Enhanced configuration and control of robots

• US 9,713,871 B2
• Filed: 08/11/2015
• Issued: 07/25/2017
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hill, David M., Evertt, Jeffrey J., Jones, Alan M., Roesler, Richard C., Jean, Andrew William, Charbonneau, Emiko V.
• Primary Examiner(s): PATTON, SPENCER D
• Application Number:
  US14/824,019
• Publication Number:
  US 20160311115A1
• Time in Patent Office: 714 Days
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/06 Communication with another machine

Abstract

Concepts and technologies are described herein for providing enhanced configuration and control of robots. Configurations disclosed herein augment a mobile computing device, such as a robot, with resources for understanding and navigation of an environment surrounding the computing device. The resources can include sensors of a separate computing device, which may be in the form of a head-mounted display. Data produced by the resources can be used to generate instructions for the mobile computing device. The sensors of the separate computing device can also detect a change in an environment or a conflict in the actions of the mobile computing device, and dynamically modify the generated instructions. By the use of the techniques disclosed herein, a simple, low-cost robot can understand and navigate through a complex environment and appropriately interact with obstacles and other objects.

Claims

1. A computer-implemented method, comprising:

receiving a signal from one or more sensors attached to a head-mounted device, wherein the one or more sensors are configured to detect a location of a computing device and locations of one or more boundaries that define an environment within which the computing device is located;

generating data identifying geometric parameters of the environment based, at least in part, on the signal;

generating location data identifying the location of the computing device relative to the environment based, at least in part, on the signal;

generating a set of instructions for the computing device to perform one or more tasks based, at least in part, on the location data identifying the location of the computing device and the data identifying the geometric parameters of the environment; and

communicating the set of instructions to the computing device thereby causing the computing device to perform the one or more tasks.

2. The method of claim 1, wherein the computing device has one or more devices configured to move the computing device based, at least in part, on the set of instructions, and wherein the method further comprises:

determining, at the head-mounted device, that a movement of the computing device conflicts with the one or more boundaries that define the environment;

generating data indicating a modification to the set of instructions to mitigate the conflict; and

communicating the data indicating the modification to the set of instructions to the computing device thereby causing the computing device to move based, at least in part, on the data indicating the modification to the set of instructions.

3. The method of claim 2, wherein the conflict is detected by an analysis of the signal received from the one or more sensors attached to the head-mounted device.

4. The method of claim 1, wherein the data identifying the geometric parameters defines one or more objects located in the environment, wherein the computing device has one or more devices configured to move the computing device based, at least in part, on the set of instructions, and wherein the method further comprises:

determining, at the head-mounted device, that a movement of the computing device conflicts with the one or more objects located in the environment;

generating data indicating a modification to the set of instructions to mitigate the conflict; and

communicating the data indicating the modification to the set of instructions to the computing device thereby causing the computing device to move based, at least in part, on the data indicating the modification to the set of instructions.

5. The method of claim 1, wherein the head-mounted device comprises a display and wherein the one or more sensors comprise at least one image camera, wherein the signal comprises image data, and wherein the location data identifying the location of the computing device is generated by:

analyzing the image data to generate data indicating a first distance between the computing device and the head-mounted device;

analyzing the image data to generate data indicating a second distance between the computing device and at least one boundary defined by the geometric parameters of the environment; and

generating the location data based, at least in part, on the first distance and the second distance.

6. The method of claim 5, wherein the computing device comprises at least two visual indicators separated at a predetermined distance, and wherein the first distance is determined based, at least in part, by the predetermined distance separating the at least two visual indicators.

7. The method of claim 1, wherein the set of instructions defines a path for the computing device to navigate within the environment and to avoid a collision with at least one object in the environment.

8. The method of claim 1, wherein the set of instructions define an interaction with at least one object within the environment.

9. The method of claim 8, wherein the interaction comprises moving the at least one object.

10. The method of claim 8, wherein the at least one object comprises an input device, and wherein the interaction includes an entry of data at the input device.

11. The method of claim 1, wherein the one or more boundaries comprise one or more walls of a room.

12. A head-mounted device, comprising:

a processor;

one or more sensors configured to detect a location of a computing device and locations of one or more boundaries that define an environment surrounding the computing device; and

a memory having a first set of computer-executable instructions stored thereupon which, when executed by the processor, cause the head-mounted device to;

receive a signal from the one or more sensors,

generate data identifying geometric parameters of the environment based, at least in part, on the signal,

generate location data identifying the location of the computing device relative to the environment based, at least in part, on the signal,

generate a second set of instructions for the computing device to perform one or more tasks based, at least in part, on the location data identifying the location of the computing device and the data identifying the geometric parameters of the environment, and

communicate the second set of instructions to the computing device thereby causing the computing device to perform the one or more tasks.

13. The head-mounted device of claim 12, wherein the computing device has one or more devices configured to move the computing device based, at least in part, on the second set of instructions, and wherein the first set of computer-executable instructions further cause the head-mounted device to:

determine that a movement of the computing device conflicts with the one or more boundaries of the environment;

generate a modification to the second set of instructions to mitigate the conflict; and

communicate the modification to the second set of instructions to the computing device thereby causing the computing device to move based, at least in part, on the modification to the second set of instructions.

14. The head-mounted device of claim 12, wherein the data identifying the geometric parameters defines one or more objects located in the environment, wherein the computing device has one or more devices configured to move the computing device based, at least in part, on the second set of instructions, and wherein the first set of computer-executable instructions further cause the head-mounted device to:

determine a presence of a changed circumstance of the one or more objects;

generate a modification to the second set of instructions to address the changed circumstance; and

communicate the modification to the second set of instructions to the computing device thereby causing the computing device to address the changed circumstance based, at least in part, on the modification to the second set of instructions.

15. The head-mounted device of claim 12, wherein the one or more sensors comprise at least one image camera, wherein the signal comprises image data, and wherein the location data identifying the location of the computing device is generated by:

analyzing the image data to generate data indicating a first distance between the computing device and the head-mounted device;

analyzing the image data to generate data indicating a second distance between the computing device and at least one boundary defined by the geometric parameters of the environment; and

generating the location data based, at least in part, on the first distance and the second distance.

16. The head-mounted device of claim 12, wherein the one or more boundaries comprise one or more walls of a room.

17. A computer storage medium having a first set of computer-executable instructions stored thereupon which, when executed by a head-mounted device, cause the head-mounted device to:

receive a signal from one or more sensors attached to the head-mounted device, wherein the one or more sensors are configured to detect a location of a computing device and locations of one or more boundaries that define an environment surrounding the computing device;

generate data identifying geometric parameters of the environment based, at least in part, on the signal;

generate location data identifying the location of the computing device relative to the environment based, at least in part, on the signal;

generate a second set of instructions for the computing device to perform one or more tasks based, at least in part, on the location data identifying the location of the computing device and the data identifying the geometric parameters of the environment; and

communicate the second set of instructions to the computing device thereby causing the computing device to perform the one or more tasks.

18. The computer storage medium of claim 17, wherein the second set of instructions defines a path for the computing device to navigate within the environment and to avoid a collision with at least one object in the environment.

19. The computer storage medium of claim 17, wherein the second set of instructions define an interaction with at least one object within the environment.

20. The computer storage medium of claim 19, wherein the interaction comprises moving the at least one object.