Adaptive image acquisition and display using multi-focal display

• US 9,865,043 B2
• Filed: 03/04/2016
• Issued: 01/09/2018
• Est. Priority Date: 03/26/2008
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Robinson, M. Dirk, Yang, Junlan, Stork, David G., Wu, Wanmin, Llull, Patrick, Tosic, Ivana, Berkner, Kathrin, Bedard, Noah
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US15/061,938
• Publication Number:
  US 20160191905A1
• Time in Patent Office: 676 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/676 Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions

Abstract

Multiframe reconstruction combines a set of acquired images into a reconstructed image. Here, which images to acquire are selected based at least in part on the content of previously acquired images. In one approach, a set of at least three images of an object are acquired at different acquisition settings. For at least one of the images in the set, the acquisition setting for the image is determined based at least in part on the content of previously acquired images. Multiframe image reconstruction, preferably via a multi-focal display, is applied to the set of acquired images to synthesize a reconstructed image of the object.

Claims

1. A method for capturing and rendering an object on a multi-focal display, the method comprising:

acquiring a set of at least two images of an object at different acquisition settings;

for at least one of the images, determining the acquisition setting for the image based at least in part on content of previously acquired images and also at least in part on reconstruction of the object on a multi-focal display that renders objects at a plurality of renderable volumes, wherein determining the acquisition setting for the image is based on optimizing a metric that is a function of both the acquisition settings and of locations of the renderable volumes of the multi-focal display, by iteratively;

optimizing the metric as a function of the acquisition settings while holding constant the locations of the renderable volumes of the multi-focal display; and

optimizing the metric as a function of the locations of the renderable volumes of the multi-focal display while holding constant the acquisition settings; and

rendering the object on the multi-focal display from the acquired set of images of the object.

2. The method of claim 1 wherein the metric has a first term that accounts for an effect of capturing a set of images rather than a full three-dimensional object, and a second term that accounts for an effect of rendering the object on the multi-focal display rather than a full three-dimensional display.

3. The method of claim 2 wherein the metric includes a weighted sum of the first term and the second term.

4. The method of claim 3 wherein weights for the weighted sum are based at least in part on one or more of:

object depth, object spatial frequency, object contrast, and object texture.

5. The method of claim 1 wherein the metric has a single term that accounts for both an effect of capturing a set of images rather than a full three-dimensional object and for an effect of rendering the object on the multi-focal display rather than a full three-dimensional display.

6. A method for capturing and rendering an object on a multi-focal display, the method comprising:

acquiring a set of at least three images of an object at different acquisition settings;

for at least one of the images, determining the acquisition setting for the image based at least in part on content of previously acquired images and also at least in part on reconstruction of the object on a multi-focal display that renders objects at a plurality of renderable volumes;

wherein the steps of acquiring a set of images and determining an acquisition setting comprise;

acquiring at least two initial images at acquisition settings that do not depend on the content of previously acquired images; and

for every image acquired after the initial images, determining the acquisition setting for the image based at least in part on the content of previously acquired images and also at least in part on reconstruction of the object on the multi-focal display; and

rendering the object on the multi-focal display from the acquired set of images of the object.

7. The method of claim 6 wherein the images are acquired at different focus settings.

8. The method of claim 7 wherein the renderable volumes are focal planes at the same focus settings used to acquire the images.

9. The method of claim 6 wherein determining the acquisition setting for the image occurs in real-time.

10. The method of claim 6 wherein rendering the object on the multi-focal display comprises:

analyzing content of the acquired set of images of the object; and

selecting locations of the renderable volumes of the multi-focal display based on the content analysis.

11. The method of claim 10 wherein selecting locations of the renderable volumes occurs in real-time.

12. The method of claim 6 wherein, for every image acquired after the initial images, determining the acquisition setting for said image does not depend on whether any additional images will be acquired after said image.

13. The method of claim 6 wherein, for at least one of the images acquired after the initial images, determining the acquisition setting for said image is based at least in part on assuming that at least one additional image will be acquired after said image.

14. The method of claim 6 wherein the set of images contains K images;

and, for every image acquired after the initial images, determining the acquisition setting for said image is based at least in part on assuming a final set containing K images.

15. The method of claim 6 wherein, for at least one of the images acquired after the initial images, determining the acquisition setting for said image is based at least in part on increasing information captured by said image, compared to information captured by previously acquired images.

16. The method of claim 6 wherein the acquisition setting includes an aperture setting.

17. The method of claim 6 wherein the renderable volumes are focal planes.

18. The method of claim 6 wherein locations of the renderable volumes are fixed.

19. A system for capturing and rendering an object, the system comprising:

an adaptive image capture device for acquiring a set of at least three images of an object at different acquisition settings;

wherein, for at least one of the images, the acquisition setting for the image is determined based at least in part on content of previously acquired images and also at least in part on reconstruction of the object on a multi-focal display, comprising;

acquiring at least two initial images at acquisition settings that do not depend on the content of previously acquired images; and

for every image acquired after the initial images, determining the acquisition setting for the image based at least in part on the content of previously acquired images and also at least in part on reconstruction of the object on the multi-focal display; and

a multi-focal display for rendering the object on the multi-focal display from the acquired set of images of the object.