Automatic narration of signal segment

• US 10,679,669 B2
• Filed: 02/17/2017
• Issued: 06/09/2020
• Est. Priority Date: 01/18/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mital, Vijay
• Primary Examiner(s): Harvey, David E
• Application Number:
  US15/436,655
• Publication Number:
  US 20180204596A1
• Time in Patent Office: 1,208 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals

Abstract

Automatic generation of a narration of what is happening in a signal segment (live or recorded). The signal segment that is to be narrated is accessed from a physical graph. In the physical graph, the signal segment evidences state of physical entities, and thus has a semantic understanding of what is depicted in the signal segment. The system then automatically determines how the physical entities are acting within the signal segment based on that semantic understanding, and builds a narration of the activities based on the determined actions. The system may determine what is interesting for narration based on a wide variety of criteria. The system could use machine learning to determine what will be interesting to narrate.

Claims

1. A computing system comprising:

one or more processors;

one or more computer-readable media having thereon computer-executable instructions that are structured such that, when executed by the one or more processors, cause the computing system to perform a method for automatically generating a narration of what has happened in a physical space, the method comprising;

accessing a sensed feature store, the sensed feature store comprising a plurality of signal segments, each signal segment comprising signals detected by sensors from one or more physical entities located within the physical space, each signal segment associated with a time when the signals of the physical entities were detected, the sensed feature store also storing a plurality of sensed features associated with the one or more physical entities, each sensed feature also associated with a time and location within the physical space;

accessing a signal segment from the sensed feature store based on a particular one or more physical entities and a particular one or more sensed features of the particular physical entities;

determining a semantic understanding of what is depicted in the signal segment based on the particular one or more physical entities, the particular one or more sensed features of the particular physical entities, and the associated times and locations in which the signal segment evidences state of the particular one or more of the physical entities, determining the semantic understanding including determining actions that happen repeatedly in the signal segment, determining what has not changed within the signal segment, and determining actions that occur continuously within the signal segment;

determining how the particular one or more physical entities within the signal segment are acting in the signal segment using the determined semantic understanding of the signal segment and at least one feature and associated times and locations within the physical space; and

generating a narration of the actions in the signal segment using the determined semantic understanding, the particular one or more sensed features, and determined actions of the particular one or more physical entities.

2. The computing system in accordance with claim 1, the signal segment comprising a video signal segment.

3. The computing system in accordance with claim 2, the generated narration of the one or more physical entities comprising an audio narration.

4. The computing system in accordance with claim 2, the generated narration of the one or more physical entities also comprising a diagrammatic narration.

5. The computing system in accordance with claim 1, the signal segment comprising an audio signal segment.

6. The computing system in accordance with claim 5, the generated narration comprising a video narration.

7. The computing system in accordance with claim 5, the generated narration comprising a diagrammatic narration.

8. The computing system in accordance with claim 1, the generated narration showing an intended recipient what to do with respect to a detected environment of the intended recipient.

9. The computing system in accordance with claim 1, the determining of how the one or more physical entities within the signal segment are acting is performed through analysis of saliency.

10. The computing system in accordance with claim 9, the analysis of saliency taking into consideration whether one or more determined actions of one or more physical entities occur repeatedly.

11. The computing system in accordance with claim 9, the analysis of saliency taking into consideration one or more sensed features of one or more physical entities that have not changed.

12. The computing system in accordance with claim 9, the analysis of saliency taking into consideration determined actions of one or more physical entities that occur continuously.

13. The computing system in accordance with claim 9, the analysis of saliency using machine learning.

14. The computing system in accordance with claim 9, the analysis of saliency including an interpretation of a user instruction.

15. The computing system in accordance with claim 9, the analysis of saliency including an analysis of a portion or portions of the signal segment that have been provided to one or more users.

16. The computing system in accordance with claim 1, the signal segment comprising a live signal segment.

17. The computing system in accordance with claim 1, the signal segment comprising a recorded signal segment.

18. The computing system in accordance with claim 1, the method being initiated by a detection that a human being is to be trained by narration of work represented in the signal segment.

19. A method for automatically generating a narration of what has happened in a physical space, the method comprising:

accessing a sensed feature store, the sensed feature store comprising a plurality of signal segments, each signal segment comprising signals detected by sensors from one or more physical entities located within the physical space, each signal segment associated with a time when the signals of the physical entities were detected, the sensed feature store also storing a plurality of sensed features associated with the one or more physical entities, each sensed feature also associated with a time and location within the physical space;

accessing a signal segment from the sensed feature store based on a particular one or more physical entities and a particular one or more sensed features of the particular physical entities;

determining a semantic understanding of what is depicted in the signal segment based on the particular one or more physical entities, the particular one or more sensed features of the particular physical entities, and the associated times and locations in which the signal segment evidences state of the particular one or more of the physical entities, determining the semantic understanding including determining actions that happen repeatedly in the signal segment, determining what has not changed within the signal segment, and determining actions that occur continuously within the signal segment;

determining how the particular one or more physical entities within the signal segment are acting in the signal segment using the determined semantic understanding of the signal segment and at least one feature and associated times and locations within the physical space; and

generating a narration of the actions in the signal segment using the determined semantic understanding, the particular one or more sensed features, and determined actions of the particular one or more physical entities.

20. A computer program product comprising one or more computer-readable storage devices having thereon computer-executable instructions that are structured such that, when executed by the one or more processors, cause the computing system to perform a method for automatically generating a narration of what has happened in a physical space, the method comprising:

accessing a sensed feature store, the sensed feature store comprising a plurality of signal segments, each signal segment comprising signals detected by sensors from one or more physical entities located within the physical space, each signal segment associated with a time when the signals of the physical entities were detected, the sensed feature store also storing a plurality of sensed features associated with the one or more physical entities, each sensed feature also associated with a time and location within the physical space;

accessing a signal segment from the sensed feature store based on a particular one or more physical entities and a particular one or more sensed features of the particular physical entities;

determining a semantic understanding of what is depicted in the signal segment based on the particular one or more physical entities, the particular one or more sensed features of the particular physical entities, and the associated times and locations in which the signal segment evidences state of the particular one or more of the physical entities, determining the semantic understanding including determining actions that happen repeatedly in the signal segment, determining what has not changed within the signal segment, and determining actions that occur continuously within the signal segment;

determining how the particular one or more physical entities within the signal segment are acting in the signal segment using the determined semantic understanding of the signal segment and at least one feature and associated times and locations within the physical space; and

generating a narration of the actions in the signal segment using the determined semantic understanding, the particular one or more sensed features, and determined actions of the particular one or more physical entities.