Centralized database for 3-D and other information in videos

• US 9,542,975 B2
• Filed: 10/25/2010
• Issued: 01/10/2017
• Est. Priority Date: 10/25/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven, Stamate, Vlad
• Primary Examiner(s): Burke, Jeff A
• Assistant Examiner(s): MINCEY, JERMAINE A
• Application Number:
  US12/911,688
• Publication Number:
  US 20120102023A1
• Time in Patent Office: 2,269 Days
• US Class Current: 1/1
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/036 Insert-editing

Abstract

Methods and systems for a centralized database for 3-D and other information in videos are presented. A centralized database contains video metadata such as camera, lighting, sound, object, depth, and annotation data that may be queried for and used in the editing of videos, including the addition and removal of objects and sounds. The metadata stored in the centralized database may be open to the public and admit contributor metadata.

Claims

1. A method comprising:

accessing a first version of video data captured by a camera to use in post capture processing by a video player, the video data comprising a video identifier;

querying a server, with the video identifier by the video player, for video metadata stored within a centralized database on the server, the video metadata comprising video lighting location metadata with coordinates of a light source with respect to the camera that captured the video data, and wherein the video metadata comprises time data indicating time information corresponding to the coordinates of the light source during the capture of the video data;

receiving the video metadata by the video player from the server;

editing the first version of the video data with the received video metadata by the video player to produce a second version of the video data, wherein editing the first version of the video data to produce the second version of the video data includes;

determining, based on the time data and the coordinates of the light source, a change in the light source during the capture of the video data; and

producing the second version of the video data based on the first version of the video data, the second version of the video data produced by adjusting light provided by the light source for the second version of the video data for a time period during the capture of the video data, the light being adjusted using the determined change in the light source, wherein adjusting the light provided by the light source for the second version of the video data includes;

adjusting the light for each object of a plurality of objects in the second version of the video data, wherein the light for the object is adjusted based on object segmentation data and depth map data included in the video metadata for the object, and wherein the light for the object is adjusted by;

creating a set of reference images of the object;

rotating the light source for the light provided for each reference image in the set of reference images;

generating a composite image of the object based on the set of reference images after rotating the light source, wherein the composite image includes information about a location of the light source for the reference image after rotating the light source and includes information about a reflectance of the object;

wherein the light for the object in the second version of the video data is adjusted based on the composite image generated for the object, and wherein the composite image of the object indicates an animated progression of a movement of the light source with respect to the object in the image; and

outputting the second version of the video data to a display by the video player while accessing the first version of the video data.

2. The method as in claim 1 wherein the video metadata comprises lighting data, sound data, camera data, the object segmentation data, the depth map data, composition data, and annotation data.

3. The method as in claim 1 wherein editing the first version of the video data includes:

analyzing the object segmentation data, the depth map data, and annotation data included in the video metadata to identify one or more objects to remove from the first version of the video data to produce the second version of the video data after the capture of the first version of the video data by the camera; and

removing the one or more objects from the first version of the video data to produce the second version of the video data after the capture of the first version of the video data by the camera; and

wherein the video lighting location metadata is applied to determine a background behind the removed one or more objects in the second version of the video data, the background being consistent with a background identified based on the video lighting location metadata throughout the capture of the first version of the video data.

4. The method as in claim 1 wherein editing the first version of the video data includes:

adding one or more objects to the first version of the video data to produce the second version of the video data, wherein the one or more objects is added after the capture of the first version of the video data by the camera; and

wherein the video lighting location metadata is applied to the added one or more objects to cause the added one or more objects to be lighted by the light source.

5. The method as in claim 1 wherein the change in the light source includes a change in lighting provided by the light source, and wherein the second version of the video data is produced upon assessing that the change in the lighting provided by the light source satisfies a lighting condition.

6. The method as in claim 1 wherein editing the first version of the video data includes:

analyzing camera data and location data to combine the first version of the video data captured by two or more cameras for a scene; and

creating a pair of stereoscopic images to present a three dimensional image of the scene in the second version of the video data.

7. The method as in claim 1 wherein editing the first version of the video data includes:

removing a sound from the first version of the video data to produce the second version of the video data; and

wherein the video metadata is applied so that the removed sound is consistent with the sound data.

8. The method as in claim 3 wherein editing the first version of the video data includes:

identifying, using the object segmentation data, the depth map data, and the annotation data, the one or more objects to be removed.

9. The method as in claim 1 wherein the video metadata is stored at a granularity of a single frame in the centralized database on the server.

10. The method as in claim 2 wherein the sound data comprises sound location metadata including coordinates of a sound source, and editing of the first version of the video data incorporates the sound location metadata into the second version of the video data.

11. The method as in claim 1 wherein the change in lighting provided by the light source is from a first time period to a second time period, the method further comprising:

assessing whether the change in the lighting provided by the light source from the first time period to the second time period satisfies the lighting condition, the lighting condition being a lighting quality condition;

wherein adjusting the light provided by the light source occurs for the first time period during the capture of the video data, and wherein the light is adjusted based on light provided by the light source for the second time period.

12. The method as in claim 1, wherein the light for the object is adjusted by rotating the light source to a different position for each of a set of reference images that are stationary in the video data that is captured.

13. The method as in claim 1, wherein the composite image is generated using a polynomial texture map, and wherein the composite image is a parametric photograph.

23. The method as in claim 1 wherein the video data is a planar representation of visual information captured by the camera.

24. The method as in claim 1 wherein the video metadata comprises camera movement in relation to one or more objects in the video data, the method further comprising:

editing the first version of the video data with the camera movement to produce the second version of the video data that relights the second version of the video data consistent with the camera movement.

14. A system comprising:

a video player that performs operations comprising;

accessing a first version of video data, the first version of the video data comprising a video identifier;

querying a server with the video identifier for video metadata stored within a centralized database on the server, the video metadata comprising video lighting location metadata with coordinates of a light source with respect to a camera that captured the video data, and wherein the video metadata comprises time data indicating time information corresponding to the coordinates of the light source during capture of the video data;

receiving the video metadata from the server;

editing the first version of the video data with the received video metadata to produce a second version of the video data, wherein editing the first version of the video data to produce the second version of the video data includes;

determining, based on the time data and the coordinates of the light source, a change in the light source during the capture of the first version of the video data; and

producing the second version of the video data based on the first version of the video data, the second version of the video data produced by adjusting light provided by the light source for the second version of the video data for a time period during the capture of the video data, the light being adjusted using the determined change in the light source, wherein adjusting the light provided by the light source for the second version of the video data includes;

adjusting the light for each object of a plurality of objects in the second version of the video data, wherein the light for the object is adjusted based on object segmentation data and depth map data included in the video metadata for the object, and wherein the light for the object is adjusted by;

creating a set of reference images of the object;

rotating the light source for the light provided for each reference image in the set of reference images;

generating a composite image of the object based on the set of reference images after rotating the light source, wherein the composite image includes information about a location of the light source for the reference image after rotating the light source and includes information about a reflectance of the object;

wherein the light for the object in the second version of the video data is adjusted based on the composite image generated for the object, and wherein the composite image of the object indicates an animated progression of a movement of the light source with respect to the object in the image; and

outputting the second version of the video data for display; and

a server that performs operations comprising;

storing the video metadata.

15. The system as in claim 14, wherein the composite image is generated using a polynomial texture map, and wherein the composite image is a parametric photograph.

16. The system as in claim 14, wherein the light for the object is adjusted by rotating the light source to a different position for each of a set of reference images that are stationary in the video data that is captured.

17. The system as in claim 16 wherein the change in the light source includes a change in lighting provided by the light source, and wherein the second version of the video data is produced upon assessing that the change in the lighting provided by the light source satisfies a lighting condition.

18. The system as in claim 14 wherein editing the first version of the video data includes:

analyzing camera data and location data to combine the first version of the video data captured by two or more cameras for a scene; and

creating a pair of stereoscopic images to present a three dimensional image of the scene in the second version of the video data.

19. The system as in claim 14 wherein the captured video data is a movie, and wherein editing the first version of the video data includes:

removing a sound from the first version of the video data to produce the second version of the video data; and

wherein the operations performed by the server further include applying the video metadata so that the removed sound is consistent with the sound data.

20. A method comprising:

electronically receiving, by a computing resource, a first metadata contribution from a first contributor for a video, the first metadata contribution identifying information about an environment of the video as captured, the first metadata contribution including information enabling a video player to update the video before being displayed on a display, a first version of the video comprising a video identifier, the video player;

querying the computing resource with the video identifier for the first metadata contribution, the first metadata contribution comprising video lighting location metadata with coordinates of a light source with respect to a camera that captured the video, and wherein the first metadata contribution comprises time data indicating time information corresponding to the coordinates of the light source during the capture of the video,

receiving the first metadata contribution from the computing resource,

editing the first version of the video with the received first metadata contribution to produce a second version of the video, wherein editing the first version of the video to produce the second version of the video includes;

determining, based on the time data and the coordinates of the light source as located during the capture of the first version of the video, and camera movement in relation to an object in the video, a change in the light source during the capture of the video,

producing the second version of the video based on the first version of the video, the second version of the video produced by adjusting light provided by the light source for the second version of the video for a time period during the capture of the video, the light being adjusted using the determined change in the light source, wherein adjusting the light provided by the light source for the second version of the video includes;

adjusting the light for each object of a plurality of objects in the second version of the video, wherein the light for the object is adjusted based on object segmentation data and depth map data included in the received first metadata for the object, and wherein the light for the object is adjusted by;

creating a set of reference images of the object;

rotating the light source for the light provided for each reference image in the set of reference images;

generating a composite image of the object based on the set of reference images after rotating the light source, wherein the composite image includes information about a location of the light source for the reference image after rotating the light source and includes information about a reflectance of the object;

wherein the light for the object in the second version of the video is adjusted based on the composite image generated for the object, and wherein the composite image of the object indicates an animated progression of a movement of the light source with respect to the object in the image; and

outputting the second version of the video for display;

storing, by the computing resource, the first metadata contribution in a centralized video metadata database;

receiving, by the computing resource, from a second contributor a second metadata contribution for a video, the second metadata contribution being received after the first metadata contribution;

determining that the second metadata contribution is more accurate than the first metadata contribution in describing the coordinates of the light source and the camera movement of the video as captured;

editing, by the video player, the first version of the video with the received second metadata contribution to produce a third version of the video that relights the second version of the video consistent with the coordinates of the light source as located during the capture of the first version of the video and camera movement in relation to the object in the video; and

replacing at least a portion of the first metadata contribution stored on the centralized video metadata database with the second metadata contribution.

21. The method as in claim 20 wherein the first contributor and the second contributor are Internet users.

22. The method as in claim 20 wherein metadata stored in the centralized video metadata database comprises camera data, lighting data, sound data, the object segmentation data, the depth map data, and annotation data.