Audio alteration techniques

• US 9,930,310 B2
• Filed: 09/09/2009
• Issued: 03/27/2018
• Est. Priority Date: 09/09/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lindahl, Aram, Chiu, Kelvin
• Primary Examiner(s): SOSANYA, OBAFEMI OLUDAYO
• Application Number:
  US12/556,380
• Publication Number:
  US 20110058056A1
• Time in Patent Office: 3,121 Days
• Field of Search: 37524016, 382254
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 9/8063 using time division multiplex of the PCM audio and PCM video signals

Abstract

A method of altering audio output from an electronic device based on image data is provided. In one embodiment, the method includes acquiring image data and determining one or more characteristics of the image data. Such characteristics may include sharpness, brightness, motion, magnification, zoom setting, and so forth, as well as variation in any of the preceding characteristics. The method may also include producing audio output, wherein at least one characteristic of the audio output is determined based on one or more of the image data characteristics. Various audio output characteristics that may be varied based on the video data characteristics may include, for instance, pitch, reverberation, tempo, volume, filter frequency response, added sound effects, or the like. Additional methods, devices, and manufactures are also disclosed.

Claims

1. A method comprising:

receiving, at an electronic device, a plurality of video data frames representing an object;

receiving, at the electronic device, an audio signal associated with the plurality of video data frames, wherein the audio signal includes a first audio property and a second audio property;

processing, by the electronic device, the video data frames, wherein processing the video data frames includes encoding the video data frames via a video encoder of the electronic device to provide encoding data, wherein the encoding data includes a motion vector calculated by the video encoder for the object, and wherein the motion vector includes a horizontal directionality and a vertical directionality; and

altering, by the electronic device, the audio signal associated with the plurality of video data frames, wherein altering the audio signal includes;

adjusting the first audio property based on the horizontal directionality of the motion vector, and

adjusting the second audio property based on the vertical directionality of the motion vector.

2. The method of claim 1, wherein the audio signal further includes a third audio property that varies based on a total amount of motion in the received plurality of video data frames.

3. The method of claim 1, wherein adjusting the first audio property or the second audio property includes varying at least one of a volume, a pitch, or a tempo of the audio signal.

4. The method of claim 1, further comprising outputting the audio signal from the electronic device, wherein outputting the audio signal includes at least one of playing the audio signal to a user via a speaker or outputting electronic data encoding the audio signal to an additional electronic device.

5. The method of claim 1, further comprising outputting video generated from the plurality of video data frames to a user via a display of the electronic device.

6. The method of claim 1, wherein receiving the plurality of video data frames includes acquiring the plurality of video data frames via an image capture device of the electronic device.

7. An electronic device comprising:

an image capture device disposed in a housing;

an image signal processor disposed in the housing;

a video encoder disposed in the housing; and

audio processing logic disposed in the housing;

wherein the electronic device is configured to acquire a plurality of video data frames representing an object via the image capture device, to acquire an audio signal associated with the plurality of video data frames and having a first audio property and a second audio property, to encode the video data frames such that the encoding data includes a motion vector having a horizontal directionality and a vertical directionality calculated by the video encoder for motion of the object, to adjust the first audio property via the audio processing logic based on the horizontal directionality of the motion vector, and to adjust the second audio property via the audio processing logic based on the vertical directionality of the motion vector.

8. The electronic device of claim 7, wherein the electronic device is configured to determine at least one of sharpness, brightness, or color statistics for the plurality of video data frames, and to adjust at least one of the first audio property or the second audio property via the audio processing logic based on the at least one of sharpness, brightness, or color statistics for the plurality of video data frames.

9. The electronic device of claim 8, wherein adjusting at least one of the first audio property or the second audio property includes at least one of variation of a pitch of the audio signal based on the directionality of the motion vector or variation of a volume of the audio signal based on the directionality of the motion vector.

10. The electronic device of claim 8, wherein adjusting the first audio property or the second audio property includes changing a tempo of music played from the electronic device, and wherein changing the tempo is based on the motion of the object.

11. The electronic device of claim 7, wherein the electronic device is configured to output the adjusted audio signal to a user during encoding of the acquired video data frames.

12. The electronic device of claim 7, wherein the video encoder includes an H.264 video encoder.

13. A method comprising:

receiving, at an electronic device, image data representing an object;

receiving, at the electronic device, an audio signal associated with the image data, wherein the audio signal includes a first audio property and a second audio property;

analyzing the image data via image processing logic of the electronic device; and

altering a characteristic of the audio signal based on the analyzed image data;

wherein analyzing the image data includes determining motion of the object represented in the image data and calculating a motion vector having a horizontal directionality and a vertical directionality for the object, and wherein altering the characteristic of the audio signal includes altering the first audio property based on the vertical directionality of the motion vector or altering the second audio property based on the horizontal directionality of the motion vector.

14. The method of claim 13, wherein receiving the image data at the electronic device includes receiving a plurality of static images.

15. The method of claim 14, comprising sequentially displaying the plurality of static images and outputting audio to a user therewith, wherein the output audio varies based on the analysis of the plurality of static images via the image processing logic.

16. The method of claim 13, wherein analyzing the image data includes determining at least one of sharpness, brightness, or color statistics, and wherein altering the characteristic of the audio signal includes altering at least one of the first audio property or the second audio property of the audio signal based on the at least one of sharpness, brightness, or color statistics.

17. The method of claim 16, wherein determining at least one of sharpness, brightness, or color statistics includes determining at least one of sharpness, brightness, or color statistics for one or more of a plurality of still images or a sequence of frames in video data.

18. A manufacture comprising:

one or more non-transitory, computer-readable storage media having application instructions encoded thereon for execution by a processing system, the application instructions comprising;

instructions for acquiring image data representing an object;

instructions for acquiring an audio signal associated with the image data, wherein the audio signal includes a first audio property, a second audio property, and a third audio property;

instructions for receiving an image attribute from image processing logic configured to encode the acquired image data;

instructions for altering the audio signal based on a motion vector calculated for motion of the object represented in the image data, wherein the motion vector includes a horizontal directionality and a vertical directionality, and wherein altering the audio signal includes adjusting the first audio property based on the horizontal directionality of the motion vector and adjusting the second audio property based on the vertical directionality of the motion vector; and

instructions for generating an additional audio effect that alters the audio signal, wherein altering the audio signal includes adjusting the third audio property in response to a physical motion of an electronic device caused by a user input, and wherein adjusting the third audio property is based on a direction of the physical motion.

19. The manufacture of claim 18, wherein the instructions for altering the audio signal includes at least one of instructions for creating the additional audio effect via a synthesizer or instructions for altering a playback characteristic of a stored audio file.

20. The manufacture of claim 18, wherein the application instructions include instructions enabling the user to input preferences with respect to the generation of the additional audio effect.

21. The manufacture of claim 18, wherein the application instructions include:

instructions for displaying a virtual scratch record turntable to the user via a display of the electronic device; and

instructions for generating the additional audio effect based on a direction of rotational motion of the electronic device caused by the user input during display of the virtual scratch record turntable, wherein the additional audio effect includes increasing the third audio property when the direction of rotational motion is clockwise and decreasing the third audio property when the direction of rotational motion is counter-clockwise.