Tracking objects during processes

• US 9,911,351 B2
• Filed: 02/27/2014
• Issued: 03/06/2018
• Est. Priority Date: 02/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: White, Christopher, Tsao, Anson, Molyneaux, David
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Wang, Carol
• Application Number:
  US14/192,337
• Publication Number:
  US 20150243013A1
• Time in Patent Office: 1,468 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

Embodiments are disclosed that relate to tracking one or more objects during a process that utilizes the objects. For example, one embodiment provides a method for monitoring performance of a process involving one or more objects, wherein the method includes receiving a set of rules defining one or more portions of the process and receiving object identification information regarding the one or more objects. The method further includes, for a selected portion of the process, receiving image information of a physical scene, identifying from the image information and the object identification information an operation performed with an identified object in the physical scene, and taking an action based upon whether the operation satisfies a rule of the set of rules associated with the selected portion of the process.

Claims

1. On a computing device, a method for monitoring performance of a process by a user involving a first object and a second object, the method comprising:

sending a request to a remote server for instructional information for the process;

receiving from the remote server the instructional information, the instructional information including object identification information comprising one or more of three-dimensional object data and an identification function for each of the first object and the second object, and also comprising a set of rules defining one or more portions of the process;

for a selected portion of the process,

receiving image information of a physical scene from a depth camera,

identifying from the image information and the one or more of the three-dimensional object data and the identification function for each of the first object and the second object an operation performed by the user, the operation comprising an interaction of the first object with the second object in the physical scene,

identifying whether one or more of the first object and the second object is a correct object for the operation; and

if one or more of the first object and the second object is an incorrect object for the operation, outputting an alert informing the user of the correct object for the operation;

if the operation does not satisfy a rule associated with the selected portion of the process, outputting an instruction to a display device to instruct how to perform an expected operation to satisfy the rule; and

if the operation does satisfy the rule, advancing to a subsequent portion of the process.

2. The method of claim 1, wherein the object identification information includes three-dimensional object data usable to identify the first object and the second object in a plurality of different orientations from the image information.

3. The method of claim 1, wherein the rule specifies one or more of an expected object, an expected first position of the expected object, an expected second position of the expected object, an expected path between the first and second positions, and movement of the expected object between the first and second positions in at least one of six degrees of freedom.

4. The method of claim 3, further comprising determining that the operation satisfies the rule if the one or more of the expected object, the expected first position of the expected object, the expected second position of the expected object, and the expected path between the first and second positions are met by the operation.

5. The method of claim 3, further comprising determining that the operation does not satisfy the rule if one or more of the identified object, a beginning position of the identified object, a path of the identified object during the operation, and an ending position of the identified object does not match one or more conditions related to one or more of the expected object, the expected first position, the expected path, and the expected second position of the rule.

6. The method of claim 5, wherein the rule further specifies a given time threshold in which the expected object is to be moved from the first position to the second position, and further comprising determining that the operation does not satisfy the rule if the identified object is not moved from the beginning position to the ending position by the given time threshold.

7. The method of claim 1, wherein taking an action based upon whether the operation satisfies the rule comprises, if the operation does satisfy the rule, for a subsequent selected portion of the process:

receiving image information of the physical scene;

identifying from the image information and the object identification information a subsequent operation performed with the interaction of the first object with the second identified object and a different identified object in the physical scene; and

taking an action based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

8. The method of claim 1, wherein the instruction comprises one or more of a video instruction, an auditory instruction, and an augmented reality instruction.

9. The method of claim 1, wherein taking an action based upon whether the operation satisfies the rule comprises storing information indicating whether the operation satisfied the rule.

10. The method of claim 1, wherein receiving image information further comprises receiving image information from a visible light camera.

11. The method of claim 1, further comprising tracking movement of the identified object with the received image information in order to identify the operation.

12. On a computing device, a method for monitoring performance of a process involving a first object and a second object, the method comprising:

receiving a set of rules defining one or more actions to be performed by a user on the first object and the second object during the process, the first object and the second object being different than the user;

receiving object identification information regarding the first object and the second object;

for a selected portion of the process,

receiving image information of a physical scene from a depth camera;

identifying from the image information and the object identification information an operation performed by the user on the first object and the second object in the physical scene, the operation comprising an interaction of the first object with the second object;

identifying whether one or more of the first object and the second object is a correct object for the operation;

if one or more of the first object and the second object is an incorrect object for the operation, outputting an alert informing the user of the correct object for the operation;

if the operation does not satisfy a rule, outputting an instruction to a display device to instruct how to perform an expected operation to satisfy the rule; and

if the operation does satisfy the rule, advancing to a subsequent selected portion of the process.

13. The method of claim 12, wherein advancing to the subsequent selected portion of the process comprises:

receiving image information of the physical scene;

identifying from the image information and the object identification information a subsequent operation performed with a different identified object of the first object and the second object in the physical scene; and

outputting an instruction or advancing to a next selected portion of the process based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

14. The method of claim 12, wherein outputting the instruction comprises outputting one or more of an audio instruction and a visual instruction.

15. The method of claim 14, wherein outputting the visual instruction comprises outputting an augmented reality view of the imaged physical scene to a display device for display, the augmented reality view including a virtual object overlaid on a representation of the physical scene.

16. A hardware storage device holding instructions executable by a logic device to:

send a request to a remote server for instructional information for a process involving a first object and a second object;

receive from the remote server the instructional information, the instructional information including object identification information comprising one or more of three-dimensional object data and an identification function for one or more objects used in the process, and also comprising

a set of rules defining one or more actions to be performed by a user on one or more of the first object and the second object during a process, the first and the second objects being different than the user;

receive depth image information of a physical scene from a depth camera;

receive object identification information comprising three-dimensional object data usable to identify the first object and the second object in a plurality of different orientations from the depth image information;

for a selected portion of the process,

identify from the depth image information and the object identification information an operation performed by the user involving the interaction of the first object with the second object in the physical scene;

identify whether one or more of the first object and the second object is a correct object for the operation;

if one or more of the first object and the second object is an incorrect object for the operation, outputting an alert informing the user of the correct object for the operation;

if the operation does not satisfy a rule, output an instruction to a display device to instruct how to perform an expected operation to satisfy the rule; and

if the operation does satisfy the rule, advance to a subsequent selected portion of the process.

17. The storage device of claim 16, wherein the instruction comprises one or more of an audio instruction and a visual instruction.

18. The storage device of claim 17, wherein the visual instruction comprises an augmented reality view of the physical scene including a virtual object overlaid on a representation of the physical scene.

19. The storage device of claim 16, wherein the process is a first process, and wherein the instructions are further executable to:

observe a second process performed on one or more additional identified objects; and

define a second set of rules for the second process based on the observation.