Textual annotation of acoustic effects

• US 11,375,293 B2
• Filed: 10/31/2018
• Issued: 06/28/2022
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Kumar, Naveen, Adams, Justice, Jati, Arindam, Omote, Masanori
• Primary Examiner(s): Tekle, Daniel T
• Application Number:
  US16/177,232
• Publication Number:
  US 20200137463A1
• Time in Patent Office: 1,336 Days
• Field of Search: 386239
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/007 using both tactile and audible presentation of the information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4666 using neural networks, e.g. processing the feedback provided by the user
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

Accommodation for color or visual impairments may be implemented by selective color substitution. A color accommodation module receives an image frame from a host system and generates a color-adapted version of the image frame. The color accommodation module may include a rule based filter that substitutes one or more colors within the image frame with one or more corresponding alternative colors.

Claims

1. A system for enhancing the accessibility of Audio Visual content, the system comprising:

an acoustic effects annotation module configured classify primary audio events occurring within an audio segment to generate one or more tags describing the primary audio events occurring within the audio segment, wherein the acoustic effects annotation module includes two neural networks that together are configured to classify the primary acoustic effects occurring within the audio segment wherein a first neural network is trained with unsupervised learning techniques to predict the primary acoustic effects occurring within the audio segment and a second neural network trained with supervised learning techniques to classify the primary acoustic effects predicted by the first neural network and wherein the acoustic effect annotation module includes an audio tagging function that combines the one or more tags with video frame information wherein the tags appear on the corresponding video images.

2. The system of claim 1 wherein the one or more primary audio events include a top three most important sounds within the audio segment.

3. The system of claim 1 wherein the audio segment is a clip of video game audio having multiple sounds associated with multiple sources.

4. The system of claim 1, further comprising a controller coupled to the acoustic effects annotation module, wherein the controller is configured to provide the one or more tags to a host system for display on a display screen and synchronize the output of the acoustic effects annotation module with one or more other neural network modules.

5. The system of claim 4 wherein the one or more other neural network modules includes a Graphical Style Modification module configured to apply a style adapted from a reference image frame to a source image frame wherein the source image frame is synchronized to appear during the audio segment.

6. The system of claim 1, further comprising a controller coupled to the host system and the action description module, wherein the controller is configured to synchronize presentation of text corresponding to the one or more tags with display of a sequence of image frames associated with the audio segment.

7. A method for enhancing accessibility of Audio Visual content, comprising:

classifying primary acoustic effects occurring within an audio segment to generate one or more tags describing the primary acoustic effects occurring within the audio segment with an acoustic effects annotation module, wherein the acoustic effects annotation module includes two neural networks that together are configured to classify the primary acoustic effects occurring within the audio segment, wherein the first neural network is trained unsupervised learning techniques to predict the primary acoustic effects occurring within the audio segment and a second neural network trained with supervised learning techniques to classify the primary acoustic effects predicted by the first neural network and wherein the acoustic effect annotation module includes an audio tagging function that combines the one or more tags with video frame information wherein the tags appear on the corresponding video images.

8. The method of claim 7 wherein the one or more primary audio events include a top three most important sounds within the audio segment.

9. The method of claim 7 wherein the audio segment is a clip of video game audio having multiple sounds associated with multiple sources.

10. The method of claim 7, further comprising providing the one or more tags to a host system for display on a display screen and synchronizing the output of the audio description module with one or more other neural network modules with a controller coupled to the audio description module.

11. The method of claim 10 wherein the one or more other neural network modules includes a Graphical Style Modification module configured to apply a style adapted from a reference image frame to a source image frame wherein the source image frame is synchronized to appear during the audio segment.

12. The method of claim 7, further comprising a controller coupled to the host system and the action description module, wherein the controller is configured to synchronize presentation of text corresponding to the one or more tags with display of a sequence of image frames associated with the audio segment.

13. A non-transitory computer-readable medium having computer readable instructions embodied therein, the instructions being configured upon execution to implement a method for enhancing accessibility of Audio Visual content, the method comprising

classifying primary audio events occurring within an audio segment to generate one or more tags describing the primary audio events occurring within the audio segment with an audio description module, wherein the acoustic effects annotation module includes two neural networks that together are configured to classify the primary acoustic effects occurring within the audio segment, wherein a first neural network is trained unsupervised learning techniques to predict the primary acoustic effects occurring within the audio segment and a second neural network trained with supervised learning techniques to classify the primary acoustic effects predicted by the first neural network and wherein the acoustic effect annotation module includes an audio tagging function that combines the one or more tags with video frame information wherein the tags appear on the corresponding video images.