Pass-through display of captured imagery

• US 11,024,082 B2
• Filed: 05/03/2016
• Issued: 06/01/2021
• Est. Priority Date: 05/04/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lee, Johnny Chung, Goldberg, Steven B.
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US15/145,362
• Publication Number:
  US 20160328882A1
• Time in Patent Office: 1,855 Days
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0025 for optical correction, e.g. distorsion, aberration
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A method includes sequentially outputting from an imaging sensor each pixel row of a set of pixel rows of an image captured by the imaging sensor. The method further includes displaying, at a display device, a pixel row representative of a first pixel row of the captured image prior to a second pixel row of the captured image being output by the imaging sensor. An apparatus includes an imaging sensor having a first lens that imparts a first type of spatial distortion, a display device coupled to the imaging sensor, the display to display imagery captured by the imaging sensor with the first spatial distortion, and an eyepiece lens aligned with the display, the eyepiece lens imparting a second type of spatial distortion that compensates for the first type of spatial distortion.

Claims

1. A method comprising:

sequentially outputting from a first imaging sensor each pixel row of a first set of pixel rows of a first image captured by the first imaging sensor;

buffering a subset of the first set of pixel rows in a buffer, the subset including a first pixel row of the first set of pixel rows;

modifying the subset of the first set of pixel rows in the buffer based on augmented reality overlay information associated with a position of the first pixel row to generate a pixel row representative of the first pixel row of the first image;

displaying, at a display device, the pixel row representative of the first pixel row of the first image prior to outputting subsequent rows from the first imaging sensor, the subsequent rows comprising a second pixel row of the first image; and

refraining from performing a second readout of a second image before a first readout of the first image completes.

2. The method of claim 1, wherein displaying the pixel row representative of the first pixel row comprises accessing the pixel row representative of the first pixel row from the buffer and driving a corresponding row of the display device with the accessed pixel row.

3. The method of claim 1, wherein modifying the buffered subset of the first set of pixel rows comprises:

receiving the augmented reality overlay information for one or more pixel rows of the subset of the first set of pixel rows from a remotely accessible augmented reality database.

4. The method of claim 1, wherein modifying the buffered subset of the first set of pixel rows comprises:

performing one or more filtering processes on the buffered subset of the first set of pixel rows.

5. The method of claim 4, wherein performing the one or more filtering processes comprises:

performing at least one of a spatial filtering process or a chromatic filtering process.

6. The method of claim 1, wherein buffering the subset of the first set of pixel rows in the buffer comprises:

buffering the first pixel row in a pixel row buffer, wherein displaying the pixel row representative of the first pixel row comprises;

accessing the pixel row representative of the first pixel row from the pixel row buffer; and

driving the display device with the accessed pixel row.

7. The method of claim 1, further comprising:

displaying, at the display device, a pixel row representative of a second pixel row prior to a third pixel row of the first image being output by the first imaging sensor.

8. The method of claim 1, further comprising:

sequentially outputting from a second imaging sensor each pixel row of a second set of pixel rows of a second image captured by the second imaging sensor; and

displaying, at the display device, a pixel row representative of a third pixel row of the second image prior to a fourth pixel row of the second image being output by the second imaging sensor.

9. The method of claim 8, wherein:

the first image is displayed in a first region of the display; and

the second image is displayed in a second region of the display concurrent with the display of the first image.

10. The method of claim 8, wherein:

the first image is displayed in a first region of the display at a first time;

the second image is displayed in a second region of the display at a second time different than the first time;

the second region is inactive at the first time; and

the first region is inactive at the second time.

11. An apparatus comprising:

a first imaging sensor having an output to sequentially output pixel rows of a first captured image;

a memory buffer configured to receive a subset of a first set of pixel rows from the first imaging sensor, the subset including a first pixel row;

a compositor that includes overlay circuitry configured to modify the first pixel row of the subset of the first set of pixel rows in the memory buffer based on augmented reality overlay information to generate a pixel row representative of the first pixel row; and

a display controller coupled to the output of the first imaging sensor, the display controller to begin sequential display of pixel rows of the first captured image, including the first pixel row, at a display device before a last pixel row of the first captured image is output by the first imaging sensor, and refraining from performing a second readout of a second captured image before a first readout of the first captured image completes.

12. The apparatus of claim 11, further comprising:

a pixel row buffer coupled to the output of the first imaging sensor, the pixel row buffer having a plurality of entries to buffer a subset of the pixel rows of the first captured image in the memory buffer,

wherein the compositor is to generate a modified subset of pixel rows; and

wherein the display controller is coupled to the pixel row buffer, the display controller is to sequentially display the pixel rows of the first captured image by sequentially accessing each pixel row of the modified subset of pixel rows from the pixel row buffer and is to drive a corresponding row of the display device with the accessed pixel row.

13. The apparatus of claim 12, wherein:

the compositor further is to receive the augmented reality overlay information for one or more pixel rows of the subset of the first set of pixel rows.

14. The apparatus of claim 12, wherein:

the compositor is to modify the buffered subset of pixel rows by performing one or more filtering processes on the buffered subset of pixel rows.

15. The apparatus of claim 14, wherein the one or more filtering processes comprises at least one of a spatial filtering process or a chromatic filtering process.

16. The apparatus of claim 11, further comprising:

a pixel row buffer to buffer a first pixel row of the first captured image in a pixel row buffer, wherein the display controller is coupled to the pixel row buffer and is to display the pixel row representative of the first pixel row by accessing the pixel row representative of the first pixel row from the buffer and is to drive the display device with the accessed pixel row.

17. The apparatus of claim 11, wherein:

the display controller further is to display, at the display device, a pixel row representative of a second pixel row prior to a third pixel row of the first captured image being output by the first imaging sensor.

18. The apparatus of claim 11, further comprising:

a second imaging sensor having an output to sequentially output pixel rows of a second captured image, wherein the display controller is coupled to the second imaging sensor and further is to begin sequential display of pixel rows of the second captured image at the display device before a last pixel row of the second captured image is output by the second imaging sensor.

19. The apparatus of claim 18, wherein:

the display controller is to display the first captured image in a first region of the display to display the second captured image in a second region of the display concurrent with the display of the first captured image.

20. The apparatus of claim 18, wherein:

the display controller is to display the first captured image in a first region of the display at a first time;

the display controller is to display the second captured image in a second region of the display at a second time different than the first time;

the second region is inactive at the first time; and

the first region is inactive at the second time.

21. The apparatus of claim 11, further comprising:

an eyepiece lens aligned with the display, the eyepiece lens imparting a first type of spatial distortion, wherein the first imaging sensor comprises a lens that imparts a second type of spatial distortion that is complementary to the first type of spatial distortion.

22. A head mounted display system comprising the apparatus of claim 11.

23. An apparatus comprising:

an imaging sensor having a lens that imparts a first type of spatial distortion;

a buffer coupled to the imaging sensor, the buffer configured to receive a subset of a first set of pixel rows of the imaging sensor, the subset including a first pixel row;

overlay circuitry configured to modify the first pixel row in the buffer based on augmented reality overlay information to generate a pixel row representative of the first pixel row;

a display device coupled to the imaging sensor, the display device to display imagery including the first pixel row prior to a second pixel row captured by the imaging sensor with the first type of spatial distortion, and refraining from performing a second readout of a second display imagery before a first readout of the display imagery completes; and

an eyepiece lens aligned with the display, the eyepiece lens imparting a second type of spatial distortion that compensates for the first type of spatial distortion, the pixel row representative of the first pixel row visible through the eyepiece lens.

24. The apparatus of claim 23, wherein the first type of spatial distortion is barrel distortion and the second type of spatial distortion is pincushion distortion.

25. The apparatus of claim 23, wherein the first type of spatial distortion is pincushion distortion and the second type of spatial distortion is barrel distortion.

26. A head mounted display comprising the apparatus of claim 23.