SCENE ANNOTATION USING MACHINE LEARNING

• US 20200134316A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Krishnamurthy, Sudha, Adams, Justice, Jati, Arindam, Omote, Masanori, Zheng, Jian
• Granted Patent: US 11,636,673 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/65 Clustering; Classification
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A system enhances existing audio-visual content with audio describing the setting of the visual content. A scene annotation module classifies scene elements from an image frame received from a host system and generates a caption describing the scene elements. A text to speech synthesis module may then convert the caption to synthesized speech data describing the scene elements within the image frame

Claims

1. A system for enhancing the accessibility of Audio Visual content, the system comprising:

a scene annotation module configured to classify scene elements from an image frame received from a host system and generate a caption describing the scene elements.

2. The system of claim 1 wherein the scene annotation module includes a first neural network configured to generate a feature vector from the image frame and a second neural network configured to generate a caption describing elements within the image frame from the feature vector.

3. The system of claim 2 wherein the caption describing elements within the image frame is a sentence predicted by the second neural network.

4. The system of claim 1 further comprising a text to speech synthesis module coupled to the scene annotation module, wherein the text to speech synthesis module is configured to convert the caption to synthesized speech data describing the scene elements within the image frame.

5. The system of claim 1, further comprising a controller coupled to the host system and the scene annotation module, wherein the controller is configured to activate the scene annotation module in response to an input from a user and synchronize the output of the scene annotation module with one or more other neural network modules.

6. The system of claim 5 wherein the one or more other neural network modules includes an acoustic effect annotation module configured classify primary acoustic effects occurring within an audio segment wherein the audio segment is synchronized to occur during presentation of the image frame.

7. The system of claim 1, wherein the image frame data is video game frame data.

8. A method for enhancing the accessibility of Audio Visual content, comprising:

classifying scene elements from an image frame received from a host system with a scene annotation module and generating a caption describing the scene elements with the scene annotation module.

9. The method of claim 8 wherein the scene annotation module includes a first neural network configured to generate a feature vector from the image frame and a second neural network configured to generate a caption describing elements within the image frame from the feature vector.

10. The method of claim 9 wherein the caption describing elements within the image frame is a sentence predicted by the second neural network.

11. The method of claim 8, further comprising converting the caption to synthesized speech data describing the scene elements within the image frame with a speech synthesis module coupled to the scene annotation module.

12. The method of claim 8, further comprising activating the scene annotation module in response to an input from a user and synchronizing the output of the scene annotation module with one or more other neural network modules with a controller coupled to the host system and the scene annotation module.

13. The method of claim 12 wherein the one or more other neural network modules includes an acoustic effect annotation module configured classify primary acoustic effects occurring within an audio segment wherein the audio segment is synchronized to occur during presentation of the image frame.

14. The method of claim of claim 8, wherein the image frame data is video game frame data.

15. A non-transitory computer-readable medium having computer readable instructions embodied therein, the computer-readable instructions being configured, when executed to implement a method for enhancing the accessibility of Audio Visual content, the method comprising:

classifying scene elements from an image frame received from a host system with a scene annotation module and generating a caption describing the scene elements with the scene annotation module.