Alignment of video and textual sequences for metadata analysis

• US 10,558,761 B2
• Filed: 07/05/2018
• Issued: 02/11/2020
• Est. Priority Date: 07/05/2018
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Li, Boyang, Sigal, Leonid, Dogan, Pelin
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US16/028,183
• Publication Number:
  US 20200012725A1
• Time in Patent Office: 586 Days
• Field of Search: 386241, 386230, 386244, 704 2, 704235, 704254, 704 3, 709219, 709206, 709224, 715716, 715865, 725 34, 725 37, 725 61, 725105, 725136, 3482221, 348515, 348700, 358 117, 37524028, 705 144, 705 1449
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/2276 with probabilistic networks, e.g. hidden Markov models

Abstract

Systems, methods and computer program products related to aligning heterogeneous sequential data. A first sequential data stream and a second sequential data stream are received. An action related to aligning the first sequential data stream and the second sequential data stream is determined using an alignment neural network. The alignment neural network includes a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data relating to a previously determined action by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream.

Claims

1. A method of aligning heterogeneous sequential data, comprising:

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising;

a fully connected layer that receives as input;

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

2. The method of claim 1, wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

3. The method of claim 1, wherein the first sequential data stream comprises video data and wherein the second sequential data stream comprises text data.

4. The method of claim 1, wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

5. The method of claim 1, further comprising storing data related to the determined first alignment action.

6. The method of claim 1, wherein the first alignment action comprises at least one of:

a pop action related to the first sequential data stream, a pop action related to the second sequential data stream, a match action, or a match-retain action.

7. The method of claim 6, wherein the first alignment action comprises the match action, and the method further comprises storing data related to the match action.

8. The method of claim 1, further comprising:

receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream.

9. The method of claim 8, wherein the determined first alignment action is a parameterized action.

10. A computer program product for aligning heterogeneous sequential data, the computer program product comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to perform an operation, the operation comprising;

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising;

a fully connected layer that receives as input;

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

11. The computer program product of claim 10 wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

12. The computer program product of claim 10, wherein the first sequential data stream comprises video data and wherein the second sequential data stream comprises text data.

13. The computer program product of claim 10, wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

14. The computer program product of claim 10, wherein the first alignment action comprises at least one of:

a pop action related to the first sequential data stream, a pop action related to the second sequential data stream, a match action, or a match-retain action.

15. The computer program product of claim 10, the operation further comprising:

receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream, and wherein the determined first alignment action is a parameterized action.

16. A system, comprising:

a processor; and

a memory containing a program that, when executed on the processor, performs an operation, the operation comprising;

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising;

a fully connected layer that receives as input;

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine a first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

17. The system of claim 16, wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

18. The system of claim 16, wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

19. The system of claim 16, wherein the first alignment action comprises a match action, and the operation further comprises storing data related to the match action.

20. The system of claim 16, the operation further comprising:

receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream, and wherein the determined first alignment action is a parameterized action.