PASS-THROUGH DISPLAY OF CAPTURED IMAGERY

• US 20160328882A1
• Filed: 05/03/2016
• Published: 11/10/2016
• Est. Priority Date: 05/04/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Goldberg, Steven B., Lee, Johnny Chung
• Granted Patent: US 11,024,082 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0025 for optical correction, e.g. distorsion, aberration
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A method includes sequentially outputting from an imaging sensor each pixel row of a set of pixel rows of an image captured by the imaging sensor. The method further includes displaying, at a display device, a pixel row representative of a first pixel row of the captured image prior to a second pixel row of the captured image being output by the imaging sensor. An apparatus includes an imaging sensor having a first lens that imparts a first type of spatial distortion, a display device coupled to the imaging sensor, the display to display imagery captured by the imaging sensor with the first spatial distortion, and an eyepiece lens aligned with the display, the eyepiece lens imparting a second type of spatial distortion that compensates for the first type of spatial distortion.

Claims

1. A method comprising:

sequentially outputting from a first imaging sensor each pixel row of a first set of pixel rows of a first image captured by the first imaging sensor; and

displaying, at a display device, a pixel row representative of a first pixel row of the first image prior to a second pixel row of the first image being output by the first imaging sensor.

2. The method of claim 1, further comprising:

buffering a subset of the first set of pixel rows in a buffer, the subset including the first pixel row;

modifying the buffered subset of pixel rows in the buffer to generate the pixel row representative of the first pixel row; and

wherein displaying the pixel row representative of the first pixel row comprises accessing the pixel row representative of the first pixel row from the buffer and driving a corresponding row of the display device with the accessed pixel row.

3. The method of claim 2, wherein modifying the buffered subset of pixel rows comprises:

receiving augmented reality overlay information for one or more pixel rows of the first subset of pixel rows; and

modifying the buffered subset of pixel rows based on the augmented reality overlay information.

4. The method of claim 2, wherein modifying the buffered subset of pixel rows comprises:

performing one or more filtering processes on the buffered subset of pixel rows.

5. The method of claim 4, wherein performing one or more filtering processes comprises:

performing at least one of;

a spatial filtering process; and

a chromatic filtering process.

6. The method of claim 1, further comprising:

buffering the first pixel row in a pixel row buffer;

modifying the first pixel row in the pixel row buffer based on augmented reality overlay information associated with a position of the first pixel row to generate the pixel row representative of the first pixel row; and

wherein displaying the pixel row representative of the first pixel row comprises accessing the pixel row representative of the first pixel row from the buffer and driving the display device with the accessed pixel row.

7. The method of claim 1, further comprising:

displaying, at the display device, a pixel row representative of a second pixel row prior to a third pixel row of the first image being output by the first imaging sensor.

8. The method of claim 1, further comprising:

sequentially outputting from a second imaging sensor each pixel row of a second set of pixel rows of a second image captured by the second imaging sensor; and

displaying, at the display device, a pixel row representative of a third pixel row of the second image prior to a fourth pixel row of the second image being output by the second imaging sensor.

9. The method of claim 8, wherein:

the first image is displayed in a first region of the display; and

the second image is displayed in a second region of the display concurrent with the display of the first image.

10. The method of claim 8, wherein:

the first image is displayed in a first region of the display at a first time;

the second image is displayed in a second region of the display at a second time different than the first time;

the second region is inactive at the first time; and

the first region is inactive at the second time.

11. An apparatus comprising:

a first imaging sensor having an output to sequentially output pixel rows of a first captured image; and

a display controller coupled to the output of the first imaging sensor, the display controller to begin sequential display of pixel rows of the first captured image at a display device before a last pixel row of the first captured image is output by the first imaging sensor.

12. The apparatus of claim 11, further comprising:

a pixel row buffer coupled to the output of the first imaging sensor, the pixel row buffer having a plurality of entries to buffer a subset of the pixel rows of the first captured image in a buffer;

a compositor coupled to the pixel row buffer, the compositor to modify the buffered subset of pixel rows to generate a modified subset of pixel rows; and

wherein the display controller is coupled to the pixel row buffer, the display controller is to sequentially display the pixel rows of the first captured image by sequentially accessing each pixel row of the modified subset of pixel rows from the pixel row buffer and is to drive a corresponding row the display device with the accessed pixel row.

13. The apparatus of claim 12, wherein:

the compositor further is to receive augmented reality overlay information for one or more pixel rows of the subset of pixel rows; and

the compositor is to modify the buffered subset of pixel rows based on the augmented reality overlay information.

14. The apparatus of claim 12, wherein:

the compositor is to modify the buffered subset of pixel rows by performing one or more filtering processes on the buffered subset of pixel rows.

15. The apparatus of claim 14, wherein the one or more filtering processes comprises at least one of:

a spatial filtering process; and

a chromatic filtering process.

16. The apparatus of claim 11, further comprising:

a pixel row buffer to buffer a first pixel row of the first captured image in a pixel row buffer;

a compositor coupled to the pixel row buffer, the compositor to modify the first pixel row in the pixel row buffer based on augmented reality overlay information associated with a position of the first pixel row to generate a pixel row representative of the first pixel row; and

wherein the display controller is coupled to the pixel row buffer and is to display the pixel row representative of the first pixel row by accessing the pixel row representative of the first pixel row from the buffer and is to drive the display device with the accessed pixel row.

17. The apparatus of claim 11, wherein:

the display controller further is to display, at the display device, a pixel row representative of a second pixel row prior to a third pixel row of the first captured image being output by the first imaging sensor.

18. The apparatus of claim 11, further comprising:

a second imaging sensor having an output to sequentially output pixel rows of a second captured image; and

wherein the display controller is coupled to the second imaging sensor and further is to begin sequential display of pixel rows of the second captured image at the display device before a last pixel row of the second captured image is output by the second imaging sensor.

19. The apparatus of claim 18, wherein:

the display controller is to display the first captured image in a first region of the display to display the second captured image in a second region of the display concurrent with the display of the first captured image.

20. The apparatus of claim 18, wherein:

the display controller is to display the first captured image in a first region of the display at a first time;

the display controller is to display the second captured image in a second region of the display at a second time different than the first time;

wherein the second region is inactive at the first time; and

wherein the first region is inactive at the second time.

21. The apparatus of claim 11, further comprising:

an eyepiece lens aligned with the display, the eyepiece lens imparting a first type of spatial distortion; and

wherein the first imaging sensor comprises a lens that imparts a second type of spatial distortion that is complementary to the first type of spatial distortion.

22. A head mounted display system comprising the apparatus of claim 11.

23. An apparatus comprising:

an imaging sensor having a lens that imparts a first type of spatial distortion;

a display device coupled to the imaging sensor, the display device to display imagery captured by the imaging sensor with the first type of spatial distortion; and

an eyepiece lens aligned with the display, the eyepiece lens imparting a second type of spatial distortion that compensates for the first type of spatial distortion.

24. The apparatus of claim 23, wherein the first type of spatial distortion is barrel distortion and the second type of spatial distortion is pincushion distortion.

25. The apparatus of claim 23, wherein the first type of spatial distortion is pincushion distortion and the second type of spatial distortion is barrel distortion.

26. A head mounted display comprising the apparatus of claim 23.