TRACKING OBJECTS DURING PROCESSES

• US 20150243013A1
• Filed: 02/27/2014
• Published: 08/27/2015
• Est. Priority Date: 02/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: White, Christopher, Tsao, Anson, Molyneaux, David
• Granted Patent: US 9,911,351 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

Embodiments are disclosed that relate to tracking one or more objects during a process that utilizes the objects. For example, one embodiment provides a method for monitoring performance of a process involving one or more objects, wherein the method includes receiving a set of rules defining one or more portions of the process and receiving object identification information regarding the one or more objects. The method further includes, for a selected portion of the process, receiving image information of a physical scene, identifying from the image information and the object identification information an operation performed with an identified object in the physical scene, and taking an action based upon whether the operation satisfies a rule of the set of rules associated with the selected portion of the process.

Claims

1. On a computing device, a method for monitoring performance of a process utilizing one or more objects, the method comprising:

receiving a set of rules defining one or more portions of the process;

receiving object identification information regarding the one or more objects;

for a selected portion of the process,

receiving image information of a physical scene;

identifying from the image information and the object identification information an operation performed with an identified object in the physical scene; and

taking an action based upon whether the operation satisfies a rule of the set of rules associated with the selected portion of the process.

2. The method of claim 1, wherein the object identification information includes three-dimensional object data usable to identify the one or more objects in a plurality of different orientations from the image information.

3. The method of claim 1, wherein the rule specifies one or more of an expected object, an expected first position of the expected object, an expected second position of the expected object, and an expected path between the first and second positions, wherein movement of the expected object between the first and second positions includes movement in at least one of six degrees of freedom.

4. The method of claim 3, further comprising determining that the operation satisfies the rule if the one or more of the expected object, the expected first position of the expected object, the expected second position of the expected object, and the expected path between the first and second positions are met by the operation.

5. The method of claim 3, further comprising determining that the operation does not satisfy the rule if one or more of the identified object, a beginning position of the identified object, a path of the identified object during the operation, and an ending position of the identified object does not match one or more conditions related to one or more of the expected object, the expected first position, the expected path, and the expected second position of the rule.

6. The method of claim 5, wherein the rule further specifies a given time threshold in which the expected object is to be moved from the first position to the second position, and further comprising determining that the operation does not satisfy the rule if the identified object is not moved from the beginning position to the ending position by the given time threshold.

7. The method of claim 1, wherein taking an action based upon whether the operation satisfies the rule comprises, if the operation does satisfy the rule, for a subsequent selected portion of the process:

receiving image information of the physical scene;

identifying from the image information and the object identification information a subsequent operation performed with one or more of the identified object and a different identified object in the physical scene; and

taking an action based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

8. The method of claim 1, wherein taking an action based upon whether the operation satisfies the rule comprises, if the operation does not satisfy the rule, outputting an instruction to instruct how to satisfy the rule.

9. The method of claim 8, wherein the instruction comprises one or more of a video instruction, an auditory instruction, and an augmented reality instruction.

10. The method of claim 1, wherein taking an action based upon whether the operation satisfies the rule comprises storing information indicating whether the operation satisfied the rule.

11. The method of claim 1, wherein receiving image information comprises receiving image information from one or more of a depth camera and a visible light camera.

12. The method of claim 1, further comprising tracking movement of the identified object with the received image information in order to identify the operation.

13. On a computing device, a method for monitoring performance of a process, the method comprising:

receiving a set of rules defining one or more actions to be performed on one or more objects during the process;

receiving object identification information regarding the one or more objects;

for a selected portion of the process,

receiving image information of a physical scene;

identifying from the image information and the object identification information an operation performed with an identified object in the physical scene;

if the operation does not satisfy the rule, outputting an instruction to instruct how to perform an expected operation to satisfy the rule; and

if the operation does satisfy the rule, advancing to a subsequent selected portion of the process.

14. The method of claim 13, wherein advancing to the subsequent selected portion of the process comprises:

receiving image information of the physical scene;

identifying from the image information and the object identification information a subsequent operation performed with one or more of the identified object and a different identified object in the physical scene; and

outputting an instruction or advancing to a next selected portion of the process based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

15. The method of claim 13, wherein outputting the instruction comprises outputting one or more of an audio instruction and a visual instruction.

16. The method of claim 15, wherein outputting the visual instruction comprises outputting an augmented reality view of the imaged physical scene to a display device for display, the augmented reality view including a virtual object overlaid on a representation of the physical scene.

17. A hardware storage device holding instructions executable by a logic device to:

receive a set of rules defining one or more actions to be performed on one or more objects during a process;

receive depth image information of a physical scene;

receive object identification information comprising three-dimensional object data usable to identify the one or more objects in a plurality of different orientations from the depth image information;

for a selected portion of the process,

identify from the depth image information and the object identification information an operation performed with an identified object in the physical scene;

if the operation does not satisfy the rule, output an instruction to instruct how to perform an expected operation to satisfy the rule; and

if the operation does satisfy the rule, advance to a subsequent selected portion of the process.

18. The storage device of claim 17, wherein the instruction comprises one or more of an audio instruction and a visual instruction.

19. The storage device of claim 18, wherein the visual instruction comprises an augmented reality view of the physical scene including a virtual object overlaid on a representation of the physical scene

20. The storage device of claim 17, wherein the process is a first process, and wherein the instructions are further executable to:

observe a second process performed on one or more additional identified objects; and

define a second set of rules for the second process based on the observation.