COOPERATIVE PHOTOGRAPHY

• US 20140285634A1
• Filed: 03/18/2014
• Published: 09/25/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B.
• Granted Patent: US 9,554,123 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

Imagery from two or more users'"'"' different smartphones is streamed to a cloud processor, enabling creation of 3D model information about a scene being imaged. From this model, arbitrary views and streams can be synthesized. In one arrangement, a user of such a system is at a sports arena, and her view of the sporting event is blocked when another spectator rises to his feet in front of her. Nonetheless, the imagery presented on her headworn display continues uninterrupted—the blocked imagery from that viewpoint being seamlessly re-created based on imagery contributed by other system users in the arena. A great variety of other features and arrangements are also detailed.

Claims

1. A method comprising:

from a first device at a first location, receiving a stream of captured imagery depicting a scene, a view of the scene from the first location being momentarily interrupted by an occlusion between the device and the scene; and

providing a user of the first device a view of the scene, without said interrupting occlusion, from the viewpoint of said first location, said providing employing a hardware processor to synthesize imagery to replace the synthesized imagery, using one or more streams of captured imagery received from one or more other devices at other locations.

2. A method comprising:

receiving image data captured by first and second smartphone cameras at first and second locations, the received image data depicting a scene including subjects positioned on a horizontal surface;

defining a 3D cuboid volume, one side of which comprises said horizontal surface; and

synthesizing a model of the subjects within said cuboid volume;

wherein the method includes spatially filtering image data captured by the first and second smartphone cameras depicting subjects outside said volume.

3. The method of claim 2 in which the defining comprises identifying two opposing corners on a face of the volume, on the horizontal surface, and determining a height of the volume.

4. A method comprising:

receiving a first stream of scene-related data including imagery captured by a first camera-equipped system, the first stream depicting a group of one or more persons from a first location, the first stream having auxiliary data including time information associated therewith;

receiving a second stream of scene-related data including imagery captured by a second camera-equipped system, the second stream depicting said group from a second location, the second stream having auxiliary data including time information associated therewith;

processing the first and second streams of data to produce a first image product, the first image product having a view of said group that is different than a view from the first location and different than a view from the second location; and

sending the produced first image product to the first system.

5. The method of claim 4 that includes sending said first image product to the first system while still receiving the first stream of data from the first system.

6. The method of claim 4 in which the stream of scene-related data captured by the first camera-equipped system depicts the scene at instants in time different than the stream of scene-related data captured by the second camera-equipped system.

7. The method of claim 4 in which the auxiliary data associated with the first and second stream also includes location information.

8. The method of claim 4 in which the first image product comprises a video image product.

9. The method of claim 4 in which the processing comprises processing the streams of imagery to deliberately introduce one or more non-photorealistic effects in the first image product.

10. The method of claim 4 in which said processing includes defining a model using the first and second streams of data, the model including a virtual structure, and imagery draped on said virtual structure.

11. The method of claim 10 in which the virtual structure comprises a mesh.

12. The method of claim 10 in which the virtual structure comprises a point cloud.

13. The method of claim 10 in which the virtual structure includes plural portions, wherein the method includes updating a first portion of the virtual structure—

but not a second portion, when there is a movement in said group.

14. The method of claim 10 in which the virtual structure includes plural portions, wherein the method includes updating said structure to include an additional portion based on a movement in said group.

15. The method of claim 10 wherein the method includes updating the imagery draped on the virtual structure, but not updating the virtual structure, when there is a certain movement in said group.

16. The method of claim 10 in which the method includes refreshing the virtual structure based on changes in said streams of data, and refreshing the draped imagery based on said changes, wherein the refreshing of the virtual structure occurs less frequently than the refreshing of the draped imagery.

17. A method comprising:

receiving a first stream of scene-related data including imagery captured by a first camera-equipped system, the first stream depicting a group of one or more persons from a first location, the first stream having auxiliary data including time information associated therewith;

receiving a second stream of scene-related data including imagery captured by a second camera-equipped system, the second stream depicting said group from a second location, the second stream having auxiliary data including time information associated therewith;

processing the first and second streams of imagery to produce a first image product, the first image product having a view of said group that is different than the a view from the first location and different than a view from the second location; and

sending the produced first image product to a third system, while still receiving the first stream of imagery from the first system.

18. A method comprising:

receiving first video imagery of a scene, captured using a sensor in a headworn apparatus of a user, the captured imagery having a first viewpoint of the scene;

processing said first video imagery of the scene in conjunction with second video imagery of the scene captured by a second sensor, said second sensor having a second viewpoint of the scene that is different than the first viewpoint, said processing yielding imagery of the scene as if viewed from a third viewpoint that is different than the first and second viewpoints; and

providing said produced imagery for presentation to said user using said headworn apparatus.

19. The method of claim 18 that includes providing said produced imagery to said user while still receiving said first video.

20. A method comprising:

receiving first data, the received first data including scene image data captured by a first camera;

receiving second data, the received second data including scene image data captured by a second camera;

based at least in part on the received first data, generating first projected disk data corresponding to positions and energies of plural projected first disks, the first data taking into account a point spread function for the first camera;

based at least in part on the received second data, generating second projected disk data corresponding to positions and energies of plural projected second disks, the projected second disk data taking into account a point spread function for the second camera;

generating a picture cell of output information by summing energy contributions from one or more projected first disks with energy contributions from one or more projected second disks.

21. The method of claim 20 that further includes receiving scene depth map information, and wherein said generating of first and second projected disk data is also based at least in part on said scene depth map information.