Cooperative photography

• US 9,554,123 B2
• Filed: 03/18/2014
• Issued: 01/24/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Zhou, Zhihan
• Application Number:
  US14/218,660
• Publication Number:
  US 20140285634A1
• Time in Patent Office: 1,043 Days
• Field of Search: 348/47
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

Imagery from two or more users'"'"' different smartphones is streamed to a cloud processor, enabling creation of 3D model information about a scene being imaged. From this model, arbitrary views and streams can be synthesized. In one arrangement, a user of such a system is at a sports arena, and her view of the sporting event is blocked when another spectator rises to his feet in front of her. Nonetheless, the imagery presented on her headworn display continues uninterrupted—the blocked imagery from that viewpoint being seamlessly re-created based on imagery contributed by other system users in the arena. A great variety of other features and arrangements are also detailed.

Claims

1. An imaging method for generating output information that depicts a scene from a virtual viewpoint, the method comprising the acts:

receiving first data, the received first data representing image data captured of the scene by a first camera system from a first viewpoint;

receiving second data, the received second data representing image data captured of the scene by a second camera system from a second viewpoint;

based at least in part on the received first data, generating first projected disk data corresponding to positions and energies of plural projected first disks, the first projected disk data taking into account a point spread function for the first camera system, each of the plural projected first disks having a different spatial orientation, but all of said plural projected first disks facing the first viewpoint;

based at least in part on the received second data, generating second projected disk data corresponding to positions and energies of plural projected second disks, the second projected disk data taking into account a point spread function for the second camera system, each of the plural projected second disks having a different spatial orientation, but all of said plural projected second disks facing the second viewpoint;

defining an array of image cells comprising the output information, each image cell having a boundary and defining an area, each of the image cells corresponding to a region of the scene; and

for each of said cells, defining a value thereof by integrating across its area;

(a) contributions from one or more first projected disks that fall within its boundary, and (b) contributions from one or more second projected disks that fall within its boundary;

wherein the contribution from at least one of said projected disks that falls within the boundary of a cell is reduced in accordance with a reflectivity function, that is dependent on a spatial orientation of the projected disk relative to said virtual viewpoint.

2. The method of claim 1 that further includes receiving scene depth map information, and wherein said generating of first and second projected disk data is also based at least in part on said scene depth map information.

3. The method of claim 1 wherein the first and second data each includes a temporal sequence of image data frames, and the method includes generating a temporal sequence of output data, said output data including a first set of disk elements that are shingled and depicted as facing the first camera system, said output data further including a second set of disk elements that are shingled and depicted as facing the second camera system.

4. The method of claim 1 in which, at one moment, disks in the first set of disk elements are strobed more brightly, or weighted more heavily, than the second set of disk elements, and at another moment, disks in the second set of disk elements are strobed more brightly, or weighted more heavily, than the first set of disk elements, in producing the output information.

5. The method of claim 3 wherein the temporal sequence of image data frames in the first data correspond to a first clock, and the temporal sequence of image data frames in the second data correspond to a second clock, and wherein the first and second clocks are different—

so that said frames in the first sequence are captured at different instants in time than said frames in the second sequence.

6. The method of claim 3 in which certain of said disk elements are asymmetrical.

7. The method of claim 3 in which certain of said disk elements are not circular.

8. The method of claim 3 in which the projected first disks are different in their shape than the projected second disks.

9. The method of claim 1 in which each of the first disks is characterized by an intensity that falls-off with distance from its center.

10. The method of claim 1 in which the first camera system shares information related to its captured scene image data with the second camera system, and the second camera system employs the shared information to make corresponding adjustments to its second data.

11. The method of claim 10 in which the second camera system adjusts brightness of one or more of the projected second disks, based on said information shared from the first camera.

12. The method of claim 10 in which the second camera system shares information related to its captured scene image data with the first camera system, and the first camera system employs the information shared from the second camera system to make corresponding adjustments to its first data.

13. The method of claim 3 in which a processing system receives said first and second data from the first and second camera systems, and generates image data representing plural new projected disks, based on said plural first and second projected disk data.

14. The method of claim 13 in which, at one moment, disks in the first set of disk elements are strobed more brightly, or weighted more heavily, than the second set of disk elements, and at another moment, disks in the second set of disk elements are strobed more brightly, or weighted more heavily, than the first set of disk elements, in producing output information.

15. A system, coupled to first and second imaging sensors, the system including processing circuitry configured to cause the system to perform acts including:

receiving first data from the first imaging sensor, the received first data representing image data captured from a scene by the first imaging sensor from a first viewpoint;

receiving second data from the second imaging sensor, the received second data representing image data captured from the scene by the second imaging sensor from a second viewpoint;

based at least in part on the received first data, generating first projected disk data corresponding to positions and energies of plural projected first disks, the first projected disk data taking into account a point spread function for the first imaging sensor, each of the plural projected first disks having a different spatial orientation, but all of said plural projected first disks facing the first viewpoint;

based at least in part on the received second data, generating second projected disk data corresponding to positions and energies of plural projected second disks, the second projected disk data taking into account a point spread function for the second imaging sensor, each of the plural projected second disks having a different spatial orientation, but all of said plural projected second disks facing the second viewpoint;

defining an array of image cells comprising output information, the output information depicting the scene from a virtual viewpoint, each of the image cells having a boundary and defining an area, each of the image cells corresponding to a region of the scene; and

for each of said cells, defining a value thereof by integrating across its area;

(a) contributions from one or more first projected disks that fall within its boundary, and (b) contributions from one or more second projected disks that fall within its boundary;

wherein the contribution from at least one of said projected disks that falls within the boundary of a cell is reduced in accordance with a reflectivity function, that is dependent on a spatial orientation of the projected disk relative to said virtual viewpoint.

16. A non-transitory computer readable medium containing instructions that configure a processor-based system to perform acts including:

receiving first data from a first imaging sensor, the received first data representing image data captured of a scene by the first imaging sensor from a first viewpoint;

receiving second data from a second imaging sensor, the received second data representing image data captured of the scene by the second imaging sensor from a second viewpoint;

based at least in part on the received first data, generating first projected disk data corresponding to positions and energies of plural projected first disks, the first projected disk data taking into account a point spread function for the first imaging sensor, each of the plural projected first disks having a different spatial orientation, but all of said plural projected first disks facing the first viewpoint;

based at least in part on the received second data, generating second projected disk data corresponding to positions and energies of plural projected second disks, the second projected disk data taking into account a point spread function for the second imaging sensor, each of the plural projected second disks having a different spatial orientation, but all of said plural projected second disks facing the second viewpoint;

defining an array of image cells comprising output information, the output information depicting the scene from a virtual viewpoint, each of the image cells having a boundary and defining an area, each of the image cells corresponding to a region of the scene; and

for each of said cells, defining a value thereof by integrating across its area;

(a) contributions from one or more first projected disks that fall within its boundary, and (b) contributions from one or more second projected disks that fall within its boundary;

wherein the contribution from at least one of said projected disks that falls within the boundary of a cell is reduced in accordance with a reflectivity function, that is dependent on a spatial orientation of the projected disk relative to said virtual viewpoint.

17. An image processing system comprising:

an input coupled to a first and second imaging sensors, through which first and second data depicting a scene from first and second respective viewpoints is provided to the system;

first means for generating data about first projected disks using the first data, the first projected disk data being a function of a point spread function for the first imaging sensor, each of the plural projected first disks having a different spatial orientation, but all of said plural projected first disks facing the first viewpoint;

second means for generating data about second projected disks using the second data, the second projected disk data being a function of a point spread function for the second imaging sensor, each of the plural projected second disks having a different spatial orientation, but all of said plural projected second disks facing the second viewpoint;

third means for defining an array of image cells comprising output information that depicts the scene from a virtual viewpoint, based on contributions from the first and second projected disks, wherein the contribution from at least one of said projected disks to one of said image cells is reduced in accordance with a reflectivity function, that is dependent on a spatial orientation of the projected disk relative to said virtual viewpoint.