Computer-aided tracking of physical entities

• US 10,606,814 B2
• Filed: 02/17/2017
• Issued: 03/31/2020
• Est. Priority Date: 01/18/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mital, Vijay, Colle, Olivier, Flutre, Arnaud Christian, Lee, Anthony Wah, Sarkar, Jaideep, Benmouffok, Moahemmed Amine
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/436,689
• Publication Number:
  US 20180203883A1
• Time in Patent Office: 1,138 Days
• Field of Search: 707803, 707700, 707796
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/24565 Triggers; Constraints
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The tracking of physical entities within a space such that computer-implemented computations (or ambient computing) may be performed on the features and/or activity of physical entities within that physical space. A space-time data structure corresponding to the physical space is set up. For at least one physical entity within that physical space, physical feature(s) of the physical entity is sensed at different instances in time. That sensed feature is stored in an entity data structure that is associated with the space-time data structure. This may be done for multiple physical entities within the location using multiple entity data structures that are each associated with the space-time data structure. A computing system may navigate such a space-time data structure to perform computation on physical entities, physical features and physical activity within that physical space.

Claims

1. A computing system comprising:

one or more processors;

one or more computer-readable media having thereon computer-executable instructions that are structured such that, when executed by the one or more processors, cause the computing system to perform a method for tracking physical entities within time and space, the method comprising;

creating a space-time data structure, the space-time data structure corresponding to a physical space;

storing the space-time data structure in a data store;

tracking one or more physical entities within the physical space, the tracking comprising one or more sensors detecting signals from the one or more physical entities within the physical space at each of a plurality of times;

based on patterns in the detected sensor signals, identifying the one or more physical entities and one or more features of each of the one or more entities at each of the plurality of times; and

modifying the space-time data structure to include the identified one or more physical entities and one or more features and, for each of the one or more features, an associated signal segment, the associated signal segment being a portion of the detected signals that was used to identify the each of the one or more features, each of the identified one or more physical entities and one or more features associated in the space-time data structure with a corresponding time, the space-time data structure being queryable and enabled to trigger one or more actions upon a particular state being achieved.

2. The computing system in accordance with claim 1, the modifying of the space-time data structure comprising the following for at least a particular physical entity that is at least temporarily within the physical space:

sensing the particular physical entity at a first time;

at least approximately representing the first time within an entity data structure that corresponds to the particular physical entity and that is computing-associated with the space-time data structure;

sensing at least one physical feature of the particular physical entity in which the particular physical entity exists at a second time, the second time being after the first time;

representing the sensed at least one physical feature of the particular physical entity at the second time within the entity data structure; and

representing an at least approximation of the second time in the entity data structure in a manner computer-associated with the sensed at least one physical feature of the particular physical entity at the second time.

3. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising an at least estimated absence of the physical entity from the location.

4. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising at least an estimated presence of the physical entity within the location.

5. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising at least an estimated position of the physical entity within the location.

6. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising at least an estimated orientation of the physical entity within the location.

7. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising at least an estimated pose of the physical entity within the location.

8. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity including a confidence level for the sensed physical feature.

9. The computing system in accordance with claim 2, the sensed at least one physical feature of the particular physical entity comprising a sensed plurality of physical features of the particular physical entity, and including a confidence level for at least one of the plurality of physical features.

10. The computing system in accordance with claim 2, the method further comprising the following:

based on the sensing of the particular physical entity at the first time, sensing at least one physical feature of the particular physical entity in which the particular physical entity exists at the first time.

11. The computing system in accordance with claim 10, the sensed at least one physical feature of the particular physical entity in which the particular physical entity exists at the first time comprising at least estimated presence of the physical entity in the location.

12. The computing system in accordance with claim 10, the sensed at least one physical feature of the particular physical entity in which the particular physical entity exists at the first time comprising at least first recent appearance of the physical entity in the location.

13. The computing system in accordance with claim 10, further comprising:

representing the sensed at least one physical feature of the particular physical entity in which the particular physical entity existed at the first time in the entity data structure in a manner computer-associated with the at least approximation of the first time.

14. The computing system in accordance with claim 10, further comprising:

sensing a feature change in the particular physical entity based on a comparison of the sensed feature of the particular physical entity at the first and second times.

15. The computing system in accordance with claim 14, further comprising:

representing the sensed feature change in the entity data structure.

16. The computing system in accordance with claim 2, the particular physical entity being a person.

17. The computing system in accordance with claim 2, the particular physical entity being a first physical entity, the entity data structure being a first entity data structure, the method further comprising:

sensing a second physical entity in the location;

sensing at least one physical feature of the second physical entity in which the second physical entity exists at the time the second physical entity was sensed;

representing the sensed at least one physical feature of the particular physical entity at the second time within a second entity data structure that is also computing-associated with the space-time data structure.

18. The computing system in accordance with claim 17, further comprising:

sensing a physical relationship between the first physical entity and the second physical entity at the first time based on a comparison of the sensed features of the first physical entity and the second physical entity.

19. A method for tracking physical entities within a location:

creating a space-time data structure, the space-time data structure corresponding to a physical space;

storing the space-time data structure in a data store;

tracking one or more physical entities within the physical space, the tracking comprising one or more sensors detecting signals from the one or more physical entities within the physical space at each of a plurality of times;

based on patterns in the detected sensor signals, identifying the one or more physical entities and one or more features of each of the one or more entities at each of the plurality of times; and

modifying the space-time data structure to include the identified one or more physical entities and one or more features and, for each of the one or more features, an associated signal segment, the associated signal segment being a portion of the detected signals that was used to identify the each of the one or more features, each of the identified one or more physical entities and one or more features associated in the space-time data structure with a corresponding time, the space-time data structure being queryable and able to trigger one or more actions upon a particular state being achieved.

20. A computer program product comprising one or more computer-readable storage media that have thereon computer-executable instructions that are structured such that, when executed by one or more processors of a computing system, cause the computing system to perform a method for tracking physical entities within a location, the method comprising:

creating a space-time data structure, the space-time data structure corresponding to a physical space;

storing the space-time data structure in a data store;

tracking one or more physical entities within the physical space, the tracking comprising one or more sensors detecting signals from the one or more physical entities within the physical space at each of a plurality of times;

based on patterns in the detected sensor signals, identifying the one or more physical entities and one or more features of each of the one or more entities at each of the plurality of times; and

modifying the space-time data structure to include the identified one or more physical entities and one or more features and, for each of the one or more features, an associated signal segment, the associated signal segment being a portion of the detected signals that was used to identify the each of the one or more features, each of the identified one or more physical entities and one or more features associated in the space-time data structure with a corresponding time, the space-time data structure being queryable and able to trigger one or more actions upon a particular state being achieved.