Specifying a point of origin of a sound for audio effects using displayed visual information from a motion picture

• US 6,829,017 B2
• Filed: 02/01/2001
• Issued: 12/07/2004
• Est. Priority Date: 02/01/2001
• Status: Active Grant

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Phillips, Michael E.
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US09/775,113
• Publication Number:
  US 20020103553A1
• Time in Patent Office: 1,405 Days
• Field of Search: H04/N.560, H04/N.974, 348/736, 348/738, 348/578, 348/579, 348/580, 348/581, 348/583, 348/462, 348/481, 348/483, 348/485, 386/54, 386/39, 386/96, 386/98, 386/99, 386/104-106, 381/26-28, 381/58, 381/61, 700/94, 352/12, 352/15
• US Class Current: 348/738
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence

Abstract

Displaying visual information from a motion picture in a visual field within a designated extent of a related aural field supports editing of a spatial audio effect for the motion picture. The extent of a related aural field also is displayed. Information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field is received for each of a number of frames of a portion of the motion picture. This information may be received from a pointing device that indicates a point in the displayed extent of the aural field, or from a tracker that indicates a position of an object in the displayed visual information, or from a three-dimensional model of an object that indicates a position of an object in the displayed visual field. Using the specified point of origin and the relationship of the visual and aural fields, parameters of the spatial audio effect may be determined, from which a soundtrack may be generated. Information describing the specified point of origin may be stored. The frames for which points of origin are specified may be key frames that specify parameters of a function defining how the point of origin changes from frame to frame in the portion of the motion picture. The relationship between a visual field and an aural field may be different for each of the plurality of frames in the motion picture. This relationship may be specified by displaying the visual information from the motion picture and an indication of the extent of the aural field to a user, who in turn, through an input device, may indicate changes to the extent of the aural field with respect to the visual information.

Claims

1. A process for defining a spatial audio effect for a motion picture, comprising:

receiving information defining a relationship between a visual field and an aural field;

displaying visual information from the motion picture in the visual field and an indication of an extent of the aural field according to the relationship between the visual field and the aural field; and

receiving information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture.

2. The process of claim 1, further comprising:

determining parameters for the spatial audio effect according to the specified point of origin.

3. The process of claim 1, further comprising:

storing information describing the specified points of origin for the number of frames.

4. The process of claim 3, wherein the information stored comprises:

an indication of the visual field;

an indication of the audio field;

an indication of the relationship between the audio field and the video field; and

parameters specifying the points of origin for the number of frames according to the relationship between the audio field and the video field.

5. The process of claim 1, wherein the number of frames are key frames specifying parameters of a function defining how the point of origin changes from frame to frame in the portion of the motion picture.

6. The process of claim 1, wherein the spatial audio effect is a one-dimensional effect.

7. The process of claim 6, wherein the spatial audio effect is panning.

8. The process of claim 1, wherein the spatial audio effect is a two-dimensional effect.

9. The process of claim 1, wherein the spatial audio effect is a three-dimensional effect.

10. The process of claim 9, wherein the spatial audio effect is a spatialization effect.

11. The process of claim 9, wherein the spatial audio effect is a surround sound effect.

12. The process of claim 1, wherein the visual field is defined by a shape and size of an image from a sequence of still images.

13. The process of claim 1, wherein the visual field is defined by a shape and size of a rendered image of a three-dimensional model.

14. The process of claim 1, wherein the aural field is rectangular.

15. The process of claim 1, wherein the aural field is elliptical.

16. The process of claim 1, wherein the aural field is a polygon.

17. The process of claim 1, wherein the aural field is a circle.

18. The process of claim 1, wherein the aural field is larger than the image.

19. The process of claim 1, wherein the displayed visual information from the motion picture comprises an image from a sequence of still images.

20. The process of claim 1, wherein the displayed visual information from the motion picture comprises a rendered image of a three-dimensional model.

21. The process of claim 1, wherein receiving information specifying a point of origin comprises:

receiving information from a pointing device that indicates a point in the displayed extent of the aural field.

22. The process of claim 1, wherein receiving information specifying a point of origin comprises:

receiving information from a tracker that indicates a position of an object in the displayed visual information.

23. The process of claim 1, wherein receiving information specifying a point of origin comprises:

receiving information from a three-dimensional model of an object that indicates a position of an object in the displayed visual field.

24. The process of claim 1, wherein receiving information defining a relationship between a visual field and an aural field includes receiving such information for each of a plurality of frames of the motion picture, and wherein such information may be different for each of the plurality of frames.

25. The process of claim 1, wherein receiving information defining the relationship between the visual field and the aural field comprises:

displaying the visual information from the motion picture;

displaying an indication of the extent of the aural field; and

receiving input from an input device indicative of changes to the extent of the aural field with respect to the visual information.

26. A graphical user interface for allowing an editor to define a spatial audio effect for a motion picture, comprising:

means for displaying visual information from the motion picture in a visual field and for displaying an indication of an extent of an aural field according to a relationship between the visual field and the aural field; and

means for receiving information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture.

27. A graphical user interface for allowing an editor to define a spatial audio effect for a motion picture, comprising:

an display output processing section having an input for receiving visual information from the motion picture, and data describing a visual field and an aural field and a relationship between the visual field an the aural field and an output for providing display data for display, including an indication of an extent of an aural field according to a relationship between the visual field and the aural field; and

an input device processing section having an input for receiving information from an input device specifying a position of the input device, and an output for providing a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture.

28. A computer program product, comprising:

a computer readable medium;

computer program instructions stored on the computer readable medium that, when executed by a computer instruct the computer to perform a process for defining a spatial audio effect for a motion picture, comprising;

receiving information defining a relationship between a visual field and an aural field;

displaying visual information from the motion picture in the visual field and an indication of an extent of the aural field according to the relationship between the visual field and the aural field; and

receiving information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture.

29. A digital information product, comprising:

a computer readable medium;

information stored on the computer readable medium that, when interpreted by a computer, indicates metadata defining a spatial audio effect for a motion picture, comprising;

an indication of a visual field associated with the motion picture;

an indication of an audio field;

an indication of a relationship between the audio field and the video field; and

parameters specifying the points of origin of a sound used in the spatial audio effect for each of a number of frames of a portion of the motion picture.

30. A process for creating a soundtrack with at least one spatial audio effect for a motion picture, comprising:

performing editing operations on one or more audio tracks of an edited motion picture to add a spatial audio effect, including specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture;

generating metadata specifying the point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture; and

generating the soundtrack using the generated metadata and sound sources.

31. A system for creating a soundtrack with at least one spatial audio effect for a motion picture, comprising:

means for performing editing operations on one or more audio tracks of an edited motion picture to add a spatial audio effect, including specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture;

means for generating metadata specifying the point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture; and

means for generating the soundtrack using the generated metadata and sound sources.

32. A computer program product, comprising:

a computer readable medium;

computer program instructions stored on the computer readable medium that, when executed by a computer instruct the computer to perform a process for creating a soundtrack with at least one spatial audio effect for a motion picture, comprising;

performing editing operations on one or more audio tracks of an edited motion picture to add a spatial audio effect, including specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture;

generating metadata specifying the point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture; and

generating the soundtrack using the generated metadata and sound sources.

33. A system for creating a soundtrack with at least one spatial audio effect for a motion picture, comprising:

a user interface module having an input for receiving editing instructions for performing editing operations on at least one audio track of an edited motion picture to add a spatial audio effect, the editing instructions including specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture;

a metadata output module having an input for receiving the editing instructions and an output for providing metadata specifying the point of origin of a sound used in the spatial audio effect with respect to the visual field for each of a number of frames of a portion of the motion picture; and

a soundtrack generation module having an input for receiving the metadata and an input for receiving sound sources and an output for providing the soundtrack using the generated metadata and sound sources.