Mixed environment display of robotic actions

• US 10,099,382 B2
• Filed: 08/11/2015
• Issued: 10/16/2018
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hill, David M., Jones, Alan M., Evertt, Jeffrey J., Roesler, Richard C., Charbonneau, Emiko V., Jean, Andrew William
• Primary Examiner(s): Patton, Spencer D
• Application Number:
  US14/824,031
• Publication Number:
  US 20160311116A1
• Time in Patent Office: 1,162 Days
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/06 Communication with another machine

Abstract

Concepts and technologies are described herein for providing a mixed environment display of robotic actions. In some configurations, techniques disclosed herein can execute a set of instructions or run a simulation of the set of instructions to generate model data defining actions of a robot based on a set of instructions. Using the model data, one or more computing devices may generate graphical data comprising a graphical representation of the robot performing tasks or actions based on an execution of the instructions. The graphical data can be in form of an animation showing a robots actions, which may include movement of the robot and the robot'"'"'s interaction with other objects. Graphical elements showing the status of the robot or graphical representations of the instructions may be included in the graphical data. The graphical data may be displayed on an interface or communicated to one or more computers for further processing.

Claims

1. A computer-implemented method, comprising:

obtaining instructions configured for a computing device to perform one or more tasks;

generating, based at least in part on the instructions, model data defining a plurality of actions to be performed by the computing device to complete the one or more tasks;

generating graphical data comprising a graphical representation of the computing device performing the plurality of actions, wherein the graphical representation is based at least in part on the model data;

generating text that describes execution of an individual action of the plurality of actions to be performed by the computing device;

causing the graphical data to be displayed on an interface to simulate the computing device performing the plurality of actions, wherein the interface includes an opaque display comprising a transparent section allowing a real-world view of the computing device in a surrounding environment; and

causing the text that describes the execution of the individual action to be displayed on the interface prior to the computing device performing the individual action, the text displayed in association with a portion of the graphical data that correspond to the individual action.

2. The computer-implemented method of claim 1, wherein the instructions define interactions between the computing device and a real-world object, and wherein the method further comprises generating location data indicating a location of the real-world object, wherein the model data is further based, at least in part, on the location data.

3. The computer-implemented method of claim 1, further comprising determining a path for the computing device based, at least in part, on the instructions, wherein the graphical representation corresponds to the path.

4. The computer-implemented method of claim 3, wherein the path is displayed as an overlay over the real-world view of the surrounding environment.

5. The computer-implemented method of claim 1, wherein the graphical representation of the computing device is displayed with the real-world view of the surrounding environment.

6. A first computing device, comprising:

a processor;

a memory having a first set of computer-executable instructions stored thereupon which, when executed by the processor, cause the first computing device to

obtain a second set of computer-executable instructions configured for a second computing device to perform one or more tasks;

obtain location data describing a location of the second computing device;

generate, based at least in part on the second set of computer-executable instructions and the location data, model data defining a plurality of actions to be performed by the second computing device to complete the one or more tasks;

generate graphical data comprising a graphical representation of the second computing device performing the plurality of actions, wherein the graphical representation is based at least in part on the model data;

generate text that describes execution of an individual action of the plurality of actions to be performed by the second computing device;

cause the graphical data to be displayed on an interface associated with the first computing device to simulate the second computing device performing the plurality of actions, wherein the interface associated with the first computing device includes an opaque display comprising a transparent section allowing a real-world view of the second computing device in a surrounding environment; and

cause the text that describes the execution of the individual action to be displayed on the interface prior to the second computing device performing the individual action, the text displayed in association with a portion of the graphical data that correspond to the individual action.

7. The first computing device of claim 6, wherein the second set of computer-executable instructions define interactions between the second computing device and one or more objects, wherein the location data further defines a location of individual ones of the one or more objects, and wherein the graphical data illustrates the interactions between the second computing device and the one or more objects.

8. The first computing device of claim 6, wherein the first set of computer-executable instructions cause the first computing device to:

generate path data modeling a path of the second computing device based, at least in part, on an analysis of the second set of computer-executable instructions, wherein the graphical representation corresponds to the path.

9. The first computing device of claim 8, wherein the graphical representation is displayed as an overlay over the real-world view of the surrounding environment.

10. The first computing device of claim 6, wherein the graphical representation of the second computing device is displayed with the real-world view of the surrounding environment.

11. A computer storage medium having a first set of computer-executable instructions stored thereupon which, when executed by a first computing device, cause the first computing device to:

obtain a second set of computer-executable instructions configured for a second computing device to perform one or more tasks;

generate, based at least in part on the second set of computer-executable instructions, model data defining a plurality of actions to be performed by the second computing device to complete the one or more tasks;

generate, based at least in part on the model data, graphical data comprising a graphical representation of the second computing device performing the plurality of actions;

generate text that describes execution of an individual action of the plurality of actions to be performed by the second computing device;

cause the graphical data to be displayed on an interface associated with the first computing device to simulate the second computing device performing the plurality of actions, wherein the interface associated with the first computing device includes an opaque display comprising a transparent section allowing a real-world view of the second computing device in a surrounding environment; and

cause the text that describes the execution of the individual action to be displayed on the interface prior to the second computing device performing the individual action, the text displayed in association with a portion of the graphical data that correspond to the individual action.

12. The computer storage medium of claim 11, wherein the second set of computer-executable instructions define interactions between the second computing device and one or more objects, and wherein the graphical data illustrates the interactions between the second computing device and the one or more objects.

13. The computer storage medium of claim 11, wherein the first set of computer-executable instructions cause the first computing device to:

cause path data modeling a path of the second computing device to be generated, wherein the graphical data corresponds to the path.

14. The computer storage medium of claim 13, wherein the graphical data further comprises a graphical representation of one or more real-world objects that are arranged based, at least in part, on a position of the one or more real-world objects in the surrounding environment.