VIDEO SYNTHESIS USING VIDEO VOLUMES

• US 20130117780A1
• Filed: 10/01/2012
• Published: 05/09/2013
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Jay Yagnik, Rahul Sukthankar
• Inventors: SUKTHANKAR, RAHUL, YAGNIK, JAY
• Granted Patent: US 9,087,242 B2
• Time in Patent Office: Days
• US Class Current: 725/32
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A volume identification system identifies a set of unlabeled spatio-temporal volumes within each of a set of videos, each volume representing a distinct object or action. The volume identification system further determines, for each of the videos, a set of volume-level features characterizing the volume as a whole. In one embodiment, the features are based on a codebook and describe the temporal and spatial relationships of different codebook entries of the volume. The volume identification system uses the volume-level features, in conjunction with existing labels assigned to the videos as a whole, to label with high confidence some subset of the identified volumes, e.g., by employing consistency learning or training and application of weak volume classifiers.

• • The labeled volumes may be used for a number of applications, such as training strong volume classifiers, improving video search (including locating individual volumes), and creating composite videos based on identified volumes.

Claims

1. A computer-implemented method of creating a composite video, the method comprising:

accessing a database storing volumes from a plurality of digital videos, each volume being a spatio-temporal segment of one of the videos and having a start time and an end time within the video, wherein the spatial extent of each volume is less than an entire frame of the video;

providing, to a client device, indicia for a plurality of the volumes;

receiving, from the client device, selections of a subset of the indicia, thereby selecting a corresponding subset of the plurality of the volumes;

retrieving, from the database, the volumes of the subset;

compositing the retrieved volumes to form a composite video.

2. The computer-implemented method of claim 1, wherein receiving the selections comprises:

receiving temporal ordering information ordering the volumes of the subset in a temporal order; and

receiving spatial arrangement information arranging the volumes of the subset in a spatial arrangement.

3. The computer-implemented method of claim 2, wherein compositing the retrieved volumes comprises:

ordering the volumes of the subset according to the temporal ordering information; and

arranging the volumes of the subset according to the spatial arrangement information;

wherein compositing the retrieved volumes to form the composite video comprises compositing the retrieved volumes according to the ordering and the arranging.

4. The computer-implemented method of claim 1, wherein the indicia for a volume comprise a visual thumbnail image representing the volume.

5. The computer-implemented method of claim 1, wherein the indicia for an identified volume comprise a visual indicator of a temporal duration of the volume.

6. The computer-implemented method of claim 1, wherein the indicia for an identified volume comprise a visual indicator of a temporal portion of one of the videos that is occupied by the volume.

7. The computer-implemented method of claim 1, wherein the indicia for an identified volume comprise a user interface element associated with the identified volume, the method further comprising:

responsive to a selection of the user interface element, providing a user interface for editing a new composite video that includes the identified volume, the user interface comprising;

a video region representing an area corresponding to a total spatial region of frames of the new composite video; and

a volume region contained by the video region and representing a spatial portion of the frames occupied by the identified volume.

8. The computer-implemented method of claim 7, wherein the user interface further comprises a mechanism for specifying a background of the video region for a portion of the video region not occupied by the volume region.

9. The computer-implemented method of claim 7, wherein the user interface further comprises a mechanism to change a temporal portion of the new composite video that corresponds to the identified volume.

10. The computer-implemented method of claim 9, wherein the mechanism comprises a volume timeline corresponding to the identified volume, such that the volume timeline being dragged changes a temporal portion of the new composite video that corresponds to the identified volume.

11. The computer-implemented method of claim 7, wherein the user interface further comprises a mechanism to change a spatial portion of the new composite video that is occupied by the identified volume.

12. The computer-implemented method of claim 7, wherein the user interface further comprises editing tools for performing at least one of scaling, rotating, and image processing the identified volume.

13. The computer-implemented method of claim 7, further comprising making the new composite video available to users of a video hosting service.

14. A computer-implemented method, comprising:

transmitting a search query over a computer network to a video sharing service;

receiving indicia of a plurality of volumes of digital videos, the volumes being spatio-temporal segments of the digital videos and matching the search query, at least one of the indicia comprising a user interface element associated with a corresponding one of the volumes;

responsive to a user selection of the user interface element, displaying a user interface for specifying a new composite video, the user interface comprising;

a video region representing an area corresponding to a total spatial region of frames of a new composite video, and

within the video region, a visual representation of the volume corresponding to the selected user interface element;

monitoring user actions that specify properties of the volume within the new composite video; and

sending a request to the video sharing service to save the new composite video comprising the specified properties to a video repository of the video sharing service.

15. A non-transitory computer-readable storage medium storing executable computer program instructions comprising:

instructions for accessing a database storing volumes from a plurality of digital videos, each volume being a spatio-temporal segment of one of the videos and having a start time and an end time within the video, wherein the spatial extent of each volume is less than an entire frame of the video;

instructions for providing, to a client device, indicia for a plurality of the volumes;

instructions for receiving, from the client device, selections of a subset of the indicia, thereby selecting a corresponding subset of the plurality of the volumes;

instructions for retrieving, from the database, the volumes of the subset; and

instructions for compositing the retrieved volumes to form a composite video.

16. The computer-readable storage medium of claim 15, wherein the instructions for receiving the selections comprise:

instructions for receiving temporal ordering information ordering the volumes of the subset in a temporal order; and

instructions for receiving spatial arrangement information arranging the volumes of the subset in a spatial arrangement.

17. The computer-readable storage medium of claim 16, wherein the instructions for compositing the retrieved volumes comprise:

instructions for ordering the volumes of the subset according to the temporal ordering information; and

instructions for arranging the volumes of the subset according to the spatial arrangement information;

wherein compositing the retrieved volumes to form the composite video comprises compositing the retrieved volumes according to the ordering and the arranging.

18. The computer-readable storage medium of claim 15, wherein the indicia for an identified volume comprise a visual indicator of a temporal portion of one of the videos that is occupied by the volume.

19. A computer-implemented method comprising:

accessing a database storing volumes from a plurality of digital videos, each volume being a spatio-temporal segment of one of the digital videos and having a start time and an end time within the video, wherein the spatial extent of each volume is less than an entire frame of the video, and wherein each volume is associated with a label descriptive of the volume;

receiving, from a client device, a search query;

responsive to receiving the search query, identifying a plurality of the volumes having a label matching the search query; and

providing indicia of the identified volumes to the client device.

20. The computer-implemented method of claim 19, further comprising:

receiving, from the client device, selections of a subset of the indicia, thereby selecting a corresponding subset of the plurality of the volumes;

retrieving, from the database, the volumes of the subset;

compositing the retrieved volumes to form a composite video; and

saving the composite video in a video repository.

21. The computer-implemented method of claim 20, wherein receiving the selections comprises receiving layering information specifying that a first one of the volumes is located in front of a second one of the volumes, and wherein compositing the retrieved volumes comprises displaying the first one of the volumes in front of the second one of the volumes.