Post-rendering image transformation using parallel image transformation pipelines

• US 10,360,832 B2
• Filed: 08/14/2017
• Issued: 07/23/2019
• Est. Priority Date: 08/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ozguner, Tolga, Comparan, Miguel, Johnson, Christopher Jon, Bradford, Jeffrey Powers
• Primary Examiner(s): Caschera, Antonio A
• Application Number:
  US15/676,906
• Publication Number:
  US 20190051229A1
• Time in Patent Office: 708 Days
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G09G 2310/0235 Field-sequential colour display
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/2003 Display of colours specific for liquid crystal displays G09G3/3607

Abstract

Techniques for post-rendering image transformation including outputting an image frame including a plurality of first pixels by sequentially generating and outputting multiple color component fields including a first color component field and a second color component field by applying one or more two-dimensional (2D) image transformations to at least one portion of the plurality of source pixels by first, second, and third image transformation pipelines, to generate transformed pixel color data for the first color component field and the second color component field.

Claims

1. An electronic device comprising:

a post-rendering image transformation module including a plurality of image transformation pipelines each arranged to generate transformed pixel color data for a single color component by applying one or more two-dimensional (2D) image transformations to a rendered source image, the plurality of image transformation pipelines including first, second, and third image transformation pipelines;

a source image data retrieval module arranged to retrieve pixel color values for the rendered source image in response to pixel color data requests received from the plurality of image transformation pipelines; and

control logic arranged to operate in a first configuration in which the post-rendering image transformation module is configured to output a first image frame including a plurality of first pixels by sequentially generating and outputting a plurality of color component fields including a first color component field and a second color component field, wherein;

each of the plurality of first pixels has pixel color values for at least a first color component and a second color component different than the first color component,

the first color component field includes first pixel color values for the first color component for the plurality of first pixels,

the pixel color values included in the first color component field specify an intensity value for the first color component,

the generating the first color component field includes generating a first portion of the first pixel color values by the first image transformation pipeline, a second portion of the first pixel color values by the second image transformation pipeline, and a third portion of the first pixel color values by the third image transformation pipeline,

the second color component field includes second pixel color values for the second color component for the plurality of first pixels,

the pixel color values included in the second color component field specify an intensity value for the second color component,

the generating the second color component field includes generating a first portion of the second pixel color values by the first image transformation pipeline, a second portion of the second pixel color values by the second image transformation pipeline, and a third portion of the second pixel color values by the third image transformation pipeline, and

the first image transformation pipeline, the second image transformation pipeline, and the third image transformation pipeline are different pipelines.

2. The electronic device according to claim 1, further comprising a field sequential color display device arranged to receive and display the plurality of color component fields in the sequence output by the post-rendering image transformation module.

3. The electronic device according to claim 2, further comprising:

a source image rendering module configured to render the source image from a three-dimensional model based on an initial predicted pose of an end user of the electronic device; and

an image transformation generation module configured to;

obtain an updated predicted pose of the end user, and

dynamically generate the one or more 2D image transformations based on at least the updated predicted pose.

4. The electronic device according to claim 3, further comprising:

a head-mounted display device;

one or more head-mounted sensor devices; and

a pose estimation module configured to;

receive first sensor data from the one or more head-mounted sensor devices,

generate the initial predicted pose based on at least the received first sensor data,

receive second sensor data from the one or more head-mounted sensor devices after receiving the first sensor data, and

generate the updated predicted pose based on at least the received second sensor data.

5. The electronic device according to claim 4, wherein:

the pose estimation module is further configured to;

receive third sensor data from the one or more head-mounted sensor devices after receiving the second sensor data, and

generate a second updated predicted pose based on at least the received third sensor data; and

the image transformation generation module is further configured to;

obtain the second updated predicted pose of the end user, and

dynamically re-generate the one or more 2D image transformations based on at least the second updated predicted pose for use in the generation of the second color component field.

6. The electronic device according to claim 1, wherein:

the control logic is further arranged to operate in a second configuration in which the post-rendering image transformation module is configured to output a second image frame by concurrently generating pixel color values for a third color component by the first image transformation pipeline, generating pixel color values for a fourth color component by the second image transformation pipeline, and generating pixel color values for a fifth color component by the third image transformation pipeline;

the control logic is further arranged to receive a display mode selection indicating a first display mode or a second display mode; and

the control logic is further arranged to selectively operate in the first configuration in response to the display mode selection indicating the first display mode and to operate in the second configuration in response to the display mode selection indicating the second display mode; and

the third, fourth, and fifth color components are different color components.

7. The electronic device according to claim 6, wherein:

in the first configuration, the device is configured to separate the rendered source image, storing pixel color values for the first color component for the rendered source image as a first component source image data and storing pixel color values for the second color component for the rendered source image as a second component source image data separate from the first component source image data; and

in the first configuration, the source image data retrieval module is configured to retrieve pixel color values for the first color component from the stored first component source image data and to retrieve pixel color values for the second color component from the stored second component source image data.

8. The electronic device according to claim 7, wherein in the second configuration, the source data retrieval module is configured to retrieve pixel color values for the first color component by obtaining pixel data words each encoding intensity values for the first and second color components, and to retrieve pixel color values for the second color component by obtaining pixel data words each encoding intensity values for the first and second color components.

9. The electronic device according to claim 8, wherein:

in the first configuration, the device is configured to store the first component source image data as first compressed image data encoding pixel color values for the first color component, and to store the second component source image data as second compressed image data separate from the first compressed image data and encoding pixel color values for the second color component;

in the second configuration, the device is configured to store the rendered source image as third compressed image data encoding pixel color values for the first and second color components; and

in the second configuration, the source data retrieval module is configured to, in response to receiving a pixel color data request, retrieve a portion of the third compressed image data corresponding to the received pixel color data request, decompress the retrieved portion of the third compressed image data to obtain pixel data words each encoding intensity values for the first and second color components.

10. The electronic device according to claim 6, further comprising:

a source image rendering module configured to render the source image from a three-dimensional model based on an initial predicted pose of an end user of the electronic device; and

an image transformation generation module configured to;

obtain an updated predicted pose of the end user, and

dynamically generate the one or more 2D image transformations based on at least the updated predicted pose.

11. The device according to claim 10, further comprising:

a head-mounted display device;

one or more head-mounted sensor devices; and

a pose estimation module configured to;

receive first sensor data from the one or more head-mounted sensor devices,

generate the initial predicted pose based on at least the received first sensor data,

receive second sensor data from the one or more head-mounted sensor devices after receiving the first sensor data, and

generate the updated predicted pose based on at least the received second sensor data.

12. The device according to claim 6, wherein:

the plurality of color component fields consists of a first total number of color component fields;

the second image frame includes a plurality of second pixels, each of the second pixels having pixel color values for a second total number of color components; and

the first total number is different than the second total number.

13. The device according to claim 6, further comprising a pixel data output buffer arranged to receive pixel color values generated by the first image transformation module, receive pixel color values generated by the second image transformation module, and receive pixel color values generated by the third image transformation module, wherein

the control logic is arranged to, while operating in the first configuration, configure the pixel data output buffer to, for each of a plurality of pixels for the first color component field, arrange the pixel color values received by the pixel data output buffer from the first, second, and third image transformation modules according to the locations of their respective first pixels in the first image frame, and

the control logic is arranged to, while operating in the second configuration, configure the pixel data output buffer to, for each of a second plurality of second pixels included in the second image frame, output a pixel data word including a pixel color value received by the pixel data output buffer from the first image transformation module, a second pixel color value received by the pixel data output buffer from the second image transformation module, and a pixel color value received by the pixel data output buffer from the third image transformation module.

14. The electronic device according to claim 1, wherein:

when the control logic is operating in the first configuration, the post-rendering image transformation module is configured to output the first pixel color values in rows or portions of rows;

the first portion of the first pixel color values comprises pixel color values for a first contiguous group of columns of the first pixels;

the second portion of the first pixel color values comprises pixel color values for a second contiguous group of columns of the first pixels that does not overlap the first contiguous group of columns; and

the third portion of the first pixel color values comprises pixel color values for a third contiguous group of columns of the first pixels that does not overlap the first or second contiguous groups of columns.

15. The device according to claim 1, further comprising an image transformation assignment module configured to operate in a third configuration configured to:

configure the first image transformation pipeline to apply one or more first 2D image transformations to generate the first portion of the first pixel color values;

configure the second image transformation pipeline to apply one or more second 2D image transformations to generate the second portion of the first pixel color values;

configure the third image transformation pipeline to apply one or more third 2D image transformations to generate the third portion of the first pixel color values;

configure the first image transformation pipeline to apply one or more fourth 2D image transformations to generate the first portion of the second pixel color values, the fourth 2D image transformations being different than the first 2D image transformations;

configure the second image transformation pipeline to apply one or more fifth 2D image transformations to generate the second portion of the second pixel color values, the fifth 2D image transformations being different than the second 2D image transformations; and

configure the third image transformation pipeline to apply one or more sixth 2D image transformations to generate the third portion of the second pixel color values, the sixth 2D image transformations being different than the third 2D image transformations.

16. A method for operating an electronic device for post-rendering image transformation, the method comprising:

determining, at the device, that a source image has been rendered, the rendered source image including a plurality of source pixels; and

outputting, from the device, an image frame including a plurality of first pixels by sequentially generating and outputting a plurality of color component fields including a first color component field and a second color component field by applying one or more two-dimensional (2D) image transformations to at least one portion of the plurality of source pixels by first, second, and third image transformation pipelines, to generate, at the device, transformed pixel color data for the first color component field and the second color component field, wherein;

each of the plurality of first pixels has pixel color values for at least a first color component and a second color component different than the first color component,

the first color component field includes first pixel color values for the first color component for the plurality of first pixels,

the pixel color values included in the first color component field specify an intensity value for the first color component,

the generating the transformed pixel color data for the first color component field includes generating a first portion of the first pixel color values by the first image transformation pipeline, a second portion of the first pixel color values by the second image transformation pipeline, and a third portion of the first pixel color values by the third image transformation pipeline,

the second color component field includes second pixel color values for the second color component for the plurality of first pixels,

the pixel color values included in the second color component field specify an intensity value for the second color component,

the generating the transformed pixel color data for second color component field includes generating, at the device, a first portion of the second pixel color values by the first image transformation pipeline, a second portion of the second pixel color values by the second image transformation pipeline, and a third portion of the second pixel color values by the third image transformation pipeline, and

the first image transformation pipeline, the second image transformation pipeline, and the third image transformation pipeline are different pipelines.

17. The method of claim 16, further comprising displaying via a field sequential color display device the plurality of color component fields in the outputted sequence.

18. The method of claim 16, further comprising:

obtaining an initial predicted pose for an end user;

rendering the source image from a three-dimensional model based on the initial predicted pose; and

obtaining an updated predicted pose for the end user; and

dynamically generating the one or more 2D image transformations based on at least the updated predicted pose.

19. The method of claim 18, further comprising:

receiving first sensor data from one or more head-mounted sensor devices;

generating the initial predicted pose based on at least the received first sensor data;

receiving second sensor data from the head-mounted sensor devices after receiving the first sensor data;

generating the updated predicted pose based on at least the received second sensor data; and

displaying via a head-mounted field sequential color display the plurality of color component fields in the outputted sequence.

20. The method of claim 19, further comprising:

receiving third sensor data from the one or more head-mounted sensor devices after receiving the second sensor data;

generating a second updated predicted pose based on at least the received third sensor data; and

dynamically re-generating the one or more 2D image transformations based on at least the second updated predicted pose for use in the generation of the second color component field.