Determining an image capture payload burst structure

• US 9,087,391 B2
• Filed: 12/13/2012
• Issued: 07/21/2015
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Geiss, Ryan, Hasinoff, Samuel William
• Primary Examiner(s): Henn, Timothy J
• Assistant Examiner(s): PHAM, QUAN L
• Application Number:
  US13/713,734
• Publication Number:
  US 20140168474A1
• Time in Patent Office: 950 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 2562/0209 Special features of electrodes classified in A61B5/24, A61B5/25, A61B5/283, A61B5/291, A61B5/296, A61B5/053
• CPC Class Codes: A61B 2562/028 Microscale sensors, e.g. electromechanical sensors [MEMS]
• CPC Class Codes: A61B 5/0084 for introduction into the body, e.g. by catheters A61B5/1459 takes precedence
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/291 for electroencephalography [EEG]
• CPC Class Codes: A61B 5/377 using evoked responses
• CPC Class Codes: A61N 1/0529 Electrodes for brain stimulation
• CPC Class Codes: A61N 5/0601 Apparatus for use inside the body
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/583 with different integration times

Abstract

A first plurality of images of a scene may be captured. Each image of the first plurality of images may be captured using a different TET. Based at least on the first plurality of images, a long TET, a short TET, and a TET sequence that includes the long TET and the short TET may be determined. A second plurality of images of the scene may be captured. The images in the second plurality of images may be captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence. Based on one or more images in the image sequence, an output image may be constructed.

Claims

1. A method comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured using a different total exposure time (TET);

based at least on the first plurality of images, determining a long TET, a short TET, and a TET sequence that includes the long TET and the short TET;

capturing, by the image sensor, a second plurality of images of the scene, wherein the images in the second plurality of images are captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence, wherein the image sequence includes a three-image sub-sequence of a first long-TET image captured using the long TET, followed by a short-TET image captured using the short TET, followed by a second long-TET image captured using the long TET; and

based on one or more images in the image sequence, constructing an output image.

2. The method of claim 1, wherein the image sequence also includes a second three-image sub-sequence of a first third long-TET image captured using the long TET, followed by a second fourth long-TET image captured using the long TET, followed by a second short-TET image captured using the short TET.

3. The method of claim 1, wherein the image sequence includes a two-image sub-sequence of a primary short-TET image followed by a primary long-TET image, wherein the image sequence also includes one or more secondary short-TET images and one or more secondary long-TET images, wherein the primary short-TET image and the secondary short-TET images are captured using the short TET, wherein the primary long-TET image and the secondary long-TET images are captured using the long TET, and wherein constructing the output image comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of the one or more secondary short-TET images;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of the one or more secondary long-TET images; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

4. The method of claim 1, wherein the image sequence includes a long-TET image captured using the long TET, and wherein constructing the output image comprises:

determining that the long-TET image is a sharpest image of all images in the image sequence captured using the long TET;

based on the long-TET image being the sharpest image of all images in the image sequence captured using the long TET, selecting the long-TET image as a primary long-TET image; and

selecting, as a primary short-TET image, an image captured using the short TET that is adjacent to the primary long-TET image in the image sequence.

5. The method of claim 4, wherein constructing the output image further comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of one or more images from the image sequence that were captured using the short TET;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of one or more images from the image sequence that were captured using the long TET; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

6. The method of claim 4, wherein the primary short-TET image immediately precedes the primary long-TET image in the image sequence.

7. The method of claim 6, wherein the short-TET image that immediately precedes the primary long-TET image in the image sequence is selected as the primary short-TET image based on image capture ending at the edge of respective image capture epochs.

8. The method of claim 4, wherein the primary short-TET image immediately follows the primary long-TET image in the image sequence.

9. The method of claim 8, wherein the short-TET image that immediately follows the primary long-TET image in the image sequence is selected as the primary short-TET image based on image capture beginning at the edge of respective image capture epochs.

10. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured using a different total exposure time (TET);

based at least on the first plurality of images, determining a long TET, a short TET, and a TET sequence that includes the long TET and the short TET;

capturing, by the image sensor, a second plurality of images of the scene, wherein the images in the second plurality of images are captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence,

wherein the image sequence includes a three-image sub-sequence of a first long-TET image captured using the long TET, followed by a short-TET image captured using the short TET, followed by a second long-TET image captured using the long TET; and

based on one or more images in the image sequence, constructing an output image.

11. The article of manufacture of claim 10, wherein the image sequence also includes a second three-image sub-sequence of a first third long-TET image captured using the long TET, followed by a second fourth long-TET image captured using the long TET, followed by a second short-TET image captured using the short TET.

12. The article of manufacture of claim 10, wherein the image sequence includes a two-image sub-sequence of a primary short-TET image followed by a primary long-TET image, wherein the image sequence also includes one or more secondary short-TET images and one or more secondary long-TET images, wherein the primary short-TET image and the secondary short-TET images are captured using the short TET, wherein the primary long-TET image and the secondary long-TET images are captured using the long TET, and wherein constructing the output image comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of the one or more secondary short-TET images;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of the one or more secondary long-TET images; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

13. The article of manufacture of claim 10, wherein the image sequence includes a long-TET image captured using the long TET, and wherein constructing the output image comprises:

determining that the long-TET image is a sharpest image of all images in the image sequence captured using the long TET;

based on the long-TET image being the sharpest image of all images in the image sequence captured using the long TET, selecting the long-TET image as a primary long-TET image; and

selecting, as a primary short-TET image, an image captured using the short TET that is adjacent to the primary long-TET image in the image sequence.

14. The article of manufacture of claim 13, wherein constructing the output image further comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of one or more images from the image sequence that were captured using the short TET;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of one or more images from the image sequence that were captured using the long TET; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

15. The article of manufacture of claim 13, wherein the primary short-TET image immediately precedes the primary long-TET image in the image sequence.

16. A computing device comprising:

at least one processor;

an image sensor;

data storage; and

program instructions, stored in the data storage, that upon execution by the at least one processor cause the computing device to perform operations including;

capturing, by the image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured using a different total exposure time (TET);

based at least on the first plurality of images, determining a long TET, a short TET, and a TET sequence that includes the long TET and the short TET;

capturing, by the image sensor, a second plurality of images of the scene, wherein the images in the second plurality of images are captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence, wherein the image sequence includes a three-image sub-sequence of a first long-TET image captured using the long TET, followed by a short-TET image captured using the short TET, followed by a second long-TET image captured using the long TET; and

based on one or more images in the image sequence, constructing an output image.

17. The computing device of claim 16, wherein the image sequence also includes a second three-image sub-sequence of a first third long-TET image captured using the long TET, followed by a second fourth long-TET image captured using the long TET, followed by a second short-TET image captured using the short TET.

18. The computing device of claim 16, wherein the image sequence includes a two-image sub-sequence of a primary short-TET image followed by a primary long-TET image, wherein the image sequence also includes one or more secondary short-TET images and one or more secondary long-TET images, wherein the primary short-TET image and the secondary short-TET images are captured using the short TET, wherein the primary long-TET image and the secondary long-TET images are captured using the long TET, and wherein constructing the output image comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of the one or more secondary short-TET images;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of the one or more secondary long-TET images; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

19. The computing device of claim 16, wherein the image sequence includes a long-TET image captured using the long TET, and wherein constructing the output image comprises:

determining that the long-TET image is a sharpest image of all images in the image sequence captured using the long TET;

based on the long-TET image being the sharpest image of all images in the image sequence captured using the long TET, selecting the long-TET image as a primary long-TET image; and

selecting, as a primary short-TET image, an image captured using the short TET that is adjacent to the primary long-TET image in the image sequence.

20. The computing device of claim 19, wherein constructing the output image further comprises:

forming a combined short-TET image, wherein the combined short-TET image includes the primary short-TET image and at least part of one or more images from the image sequence that were captured using the short TET;

forming a combined long-TET image, wherein the combined long-TET image includes the primary long-TET image and at least part of one or more images from the image sequence that were captured using the long TET; and

forming the output image, wherein the output image includes at least part of the combined short-TET image and at least part of the combined long-TET image.

21. A method comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured using a different total exposure time (TET);

based at least on the first plurality of images, determining a long TET, a short TET, and a TET sequence that includes the long TET and the short TET;

capturing, by the image sensor, a second plurality of images of the scene, wherein the images in the second plurality of images are captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence, wherein the image sequence includes a three-image sub-sequence of a first long-TET image captured using the long TET, followed by a second long-TET image captured using the long TET, followed by a short-TET image captured using the short TET; and

based on one or more images in the image sequence, constructing an output image.

22. A method comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured using a different total exposure time (TET);

based at least on the first plurality of images, determining a long TET, a short TET, and a TET sequence that includes the long TET and the short TET;

capturing, by the image sensor, a second plurality of images of the scene, wherein the images in the second plurality of images are captured sequentially in an image sequence using a sequence of TETs corresponding to the TET sequence, wherein the image sequence includes a long-TET image captured using the long TET; and

based on one or more images in the image sequence, constructing an output image, wherein constructing the output image comprises (i) determining that the long-TET image is a sharpest image of all images in the image sequence captured using the long TET, (ii) based on the long-TET image being the sharpest image of all images in the image sequence captured using the long TET, selecting the long-TET image as a primary long-TET image, and (iii) selecting, as a primary short-TET image, an image captured using the short TET that is adjacent to the primary long-TET image in the image sequence.