Audiovisual capture and sharing framework with coordinated user-selectable audio and video effects filters
First Claim
1. An audiovisual processing method comprising:
- using a portable computing device to capture corresponding audio and video streams from microphone and camera interfaces thereof and to store, at least temporarily, the captured audio and video streams in storage;
on the portable computing device, selecting a first effect profile from a plurality of predefined effect profiles, the predefined effect profiles selective for respective and coordinated pairings of audio and visual filters to be applied to the audio and video streams, respectively;
processing the audio and video streams in respective and coordinated audio and video pipelines, wherein the coordination is based at least in part on application of the coordinated audio and visual filters to the respective audio and video streams, wherein the processing the audio and video streams includes;
in the audio pipeline, segmenting the audio stream into plural segments and mapping individual ones of the segments to respective subphrase portions of a phrase template for a target song; and
in the video pipeline, segmenting the video stream and mapping segments thereof in correspondence with the audio segmentation and mapping;
audiovisually rendering the audio and video streams, with the coordinated audio and visual filters applied, to a display of the portable computing device; and
thereafterstoring, transmitting or posting the rendered audiovisual content.
2 Assignments
0 Petitions
Accused Products
Abstract
Coordinated audio and video filter pairs are applied to enhance artistic and emotional content of audiovisual performances. Such filter pairs, when applied in audio and video processing pipelines of an audiovisual application hosted on a portable computing device (such as a mobile phone or media player, a computing pad or tablet, a game controller or a personal digital assistant or book reader) can allow user selection of effects that enhance both audio and video coordinated therewith. Coordinated audio and video are captured, filtered and rendered at the portable computing device using camera and microphone interfaces, using digital signal processing software executable on a processor and using storage, speaker and display devices of, or interoperable with, the device. By providing audiovisual capture and personalization on an intimate handheld device, social interactions and postings of a type made popular by modern social networking platforms can now be extended to audiovisual content.
25 Citations
48 Claims
-
1. An audiovisual processing method comprising:
-
using a portable computing device to capture corresponding audio and video streams from microphone and camera interfaces thereof and to store, at least temporarily, the captured audio and video streams in storage; on the portable computing device, selecting a first effect profile from a plurality of predefined effect profiles, the predefined effect profiles selective for respective and coordinated pairings of audio and visual filters to be applied to the audio and video streams, respectively; processing the audio and video streams in respective and coordinated audio and video pipelines, wherein the coordination is based at least in part on application of the coordinated audio and visual filters to the respective audio and video streams, wherein the processing the audio and video streams includes; in the audio pipeline, segmenting the audio stream into plural segments and mapping individual ones of the segments to respective subphrase portions of a phrase template for a target song; and in the video pipeline, segmenting the video stream and mapping segments thereof in correspondence with the audio segmentation and mapping; audiovisually rendering the audio and video streams, with the coordinated audio and visual filters applied, to a display of the portable computing device; and
thereafterstoring, transmitting or posting the rendered audiovisual content. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A computer program product encoded in a non-transitory computer-readable medium, the computer program product including instructions executable on a processor of a portable computing device to cause the portable computing device to perform steps including:
-
capturing corresponding audio and video streams from microphone and camera interfaces thereof and to store, at least temporarily, the captured audio and video streams in storage; selecting a first effect profile from a plurality of predefined effect profiles, the predefined effect profiles selective for respective and coordinated pairings of audio and visual filters to be applied to the audio and video streams, respectively; processing the audio and video streams in respective and coordinated audio and video pipelines, wherein the coordination is based at least in part on application of the coordinated audio and visual filters to the respective audio and video streams, wherein the processing includes; in the audio pipeline, segmenting the audio stream into plural segments and mapping individual ones of the segments to respective subphrase portions of a phrase template for a target song; and in the video pipeline, segmenting the video stream and mapping segments thereof in correspondence with the audio segmentation and mapping; audiovisually rendering the audio and video streams, with the coordinated audio and visual filters applied, to a display of the portable computing device; and
thereafterstoring, transmitting or posting the rendered audiovisual content.
-
-
20. A system comprising a portable computing device programmed with instructions executable on a processor thereof to cause the portable computing device to perform steps including:
-
capturing corresponding audio and video streams from microphone and camera interfaces thereof and to store, at least temporarily, the captured audio and video streams in storage; selecting a first effect profile from a plurality of predefined effect profiles, the predefined effect profiles selective for respective and coordinated pairings of audio and visual filters to be applied to the audio and video streams, respectively; processing the audio and video streams in respective and coordinated audio and video pipelines, wherein the coordination is based at least in part on application of the coordinated audio and visual filters to the respective audio and video streams, wherein the processing includes; in the audio pipeline, segmenting the audio stream into plural segments and mapping individual ones of the segments to respective subphrase portions of a phrase template for a target song; and in the video pipeline, segmenting the video stream and mapping segments thereof in correspondence with the audio segmentation and mapping; audiovisually rendering the audio and video streams, with the coordinated audio and visual filters applied, to a display of the portable computing device; and
thereafterstoring, transmitting or posting the rendered audiovisual content.
-
-
21. A computational method for transforming audiovisual content, the method comprising:
-
accessing the encoding of audiovisual content from computer readable storage, wherein the audiovisual content includes coordinated audio and video streams; and processing the audio and video streams in respective and coordinated audio and video pipelines, wherein coordination of the respective audio and video pipelines includes; using, in the processing by the video pipeline, temporally localizable features extracted in the audio pipeline, wherein at least some of the temporally localizable features delimit audio-side epochs, and reordering respective portions of both the audio and video streams in accordance with the epochs. - View Dependent Claims (22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35)
-
-
36. A computational method for transforming audiovisual content, the method comprising:
-
accessing the encoding of audiovisual content from computer readable storage, wherein the audiovisual content includes coordinated audio and video streams; and processing the audio and video streams in respective and coordinated audio and video pipelines, wherein coordination of the respective audio and video pipelines includes using, in the processing by the audio pipeline, temporally localizable features extracted in the video pipeline; in the audio pipeline, segmenting the audio stream into plural segments, temporally aligning successive ones of the segments with respective pulses of a rhythmic skeleton for a target song, temporally stretching at least some of the temporally aligned segments, and temporally compressing at least some other ones of the temporally aligned segments, in the video pipeline, segmenting the video stream and temporally aligning, stretching and compressing respective segments thereof in correspondence with the audio segmentation aligning, stretching and compressing. - View Dependent Claims (37, 38)
-
-
39. A method comprising:
-
capturing coordinated audio and video streams at a portable computing device, wherein the captured audio stream includes vocals; processing the audio and video streams in respective and coordinated audio and video pipelines, wherein coordination of the respective audio and video pipelines includes either or both of (i) using one or more features extracted in the audio pipeline in the processing by the video pipeline and (ii) using one or more features extracted in the video pipeline in the processing by the audio pipeline; automatically generating, in the audio pipeline, a musical accompaniment for the vocals based on a selection of chords that are harmonies of melody pitches detected in the vocals and further based on a statistical model of chord progressions; and on the portable computing device, audiovisually rendering output of the coordinated audio and video pipelines. - View Dependent Claims (40)
-
-
41. A computer program product encoded in a non-transitory computer-readable medium, the computer program product including instructions executable on a processor of a portable computing device to cause the portable computing device to perform the steps including:
-
capturing coordinated audio and video streams, wherein the captured audio stream includes vocals; processing the audio and video streams in respective and coordinated audio and video pipelines, wherein coordination of the respective audio and video pipelines includes either or both of (i) using one or more features extracted in the audio pipeline in the processing by the video pipeline and (ii) using one or more features extracted in the video pipeline in the processing by the audio pipeline; automatically generating, in the audio pipeline, a musical accompaniment for the vocals based on a selection of chords that are harmonies of melody pitches detected in the vocals and further based on a statistical model of chord progressions; and audiovisually rendering output of the coordinated audio and video pipelines.
-
-
42. A method comprising:
-
capturing coordinated audio and video streams using a portable computing device, the captured audio stream including vocals; selecting from amongst predefined pairs of coordinated audio and visual effects to be applied in audio and video pipelines of the portable computing device, automatically generating musical accompaniment for the vocals based on a selection of chords that are harmonies of melody pitches detected in the vocals and further based on a statistical model of chord progressions; and audiovisually rendering the audio and video streams with the coordinated audio and visual effects applied. - View Dependent Claims (44, 45, 48)
-
-
43. The method of 42,
wherein the audiovisual rendering includes the automatically generated musical accompaniment.
-
46. The method of 42, wherein the audio effects include one or more of:
-
a reverb or echo effect; a chorus or harmonize effect; a band-limited filter evocative of telephone, citizens'"'"' band (CB) radio or vintage audio; and an overlay track.
-
-
47. The method of 42, wherein the video effects include one or more of:
-
a chromatic shift; a brightness or contrast change; a sharpening or blurring; and a video gating or overlay.
-
Specification