Reducing latency in an augmented-reality display

• US 9,576,397 B2
• Filed: 09/10/2012
• Issued: 02/21/2017
• Est. Priority Date: 09/10/2012
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Blackberry Limited
• Inventors: Johansson, Karl-Anders Reinhold, Grdenfors, Dan Zacharias
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Vu, Khoa
• Application Number:
  US13/608,879
• Publication Number:
  US 20140071116A1
• Time in Patent Office: 1,625 Days
• Field of Search: 345/419
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/0492 Change of orientation of the displayed image, e.g. upside-down, mirrored

Abstract

Disclosed are methods and systems for generating display pixel data so as to reduce latency when rendering a representation of a graphic on a display, such as for augmented-reality applications. The method comprises: receiving a set of display pixel coordinate-pairs at the graphics processing unit; applying a transform matrix to the set of display pixel coordinate-pairs to obtain a set of graphic pixel coordinate-pairs, the transform matrix calculated using orientation data received from an external reference; retrieving a set of graphic pixel data associated with the set of graphic pixel coordinate-pairs; and, determining a set of display pixel data based on the retrieved set of graphic pixel data.

Claims

1. A method, implemented on a graphics processing unit, of generating display pixel data for rendering a representation of a graphic on a display, the method comprising:

receiving a set of display pixel coordinate-pairs at the graphics processing unit, wherein the set of display pixel coordinate-pairs is on the same scan-line in a raster scan process on the display;

applying a transform matrix to the set of display pixel coordinate-pairs to obtain a set of graphic pixel coordinate-pairs, the transform matrix calculated using orientation data received from an external reference, wherein the transform matrix is a mapping from a two-dimensional coordinate system of the display to a three-dimensional planar region in space identified by the orientation data and a projection onto a two-dimensional plane;

retrieving a set of graphic pixel data associated with the set of graphic pixel coordinate-pairs;

determining a set of display pixel data based on the retrieved set of graphic pixel data;

rendering the set of display pixel data; and

applying, using a second warp unit, a second transform matrix to a second set of display pixel coordinate-pairs to obtain a second set of graphic pixel coordinate-pairs and retrieving a second set of graphic pixel data from the image in memory, the second set of graphic pixel data associated with the second set of graphic pixel coordinate-pairs.

2. The method of claim 1, further comprising updating the transform matrix using updated orientation data from the external reference at least once per scan-line.

3. The method of claim 1, wherein one or more of the set of graphic pixel coordinate-pairs is not on the three-dimensional planar region projected on the display.

4. The method of claim 3, wherein the set of graphic pixel data associated with the one or more of the set of graphic pixel coordinate-pairs that is not on the three-dimensional planar region on the display is null.

5. The method of claim 1, wherein the set of display pixel coordinate-pairs comprise a single pixel coordinate-pair.

6. The method of claim 1, further comprising applying a pre-determined custom matrix to the set of display pixel data.

7. The method of claim 1, further comprising:

receiving a subsequent set of display pixel coordinate-pairs after retrieving the set of graphic pixel data associated with the set of graphic pixel coordinate-pairs, wherein the subsequent set of display pixel coordinate-pairs is on a next scan-line in a raster scan process on the display;

applying a subsequent transform matrix to the subsequent set of display pixel coordinate-pairs to obtain subsequent set of graphic pixel coordinate-pairs, the subsequent transform matrix calculated using further orientation data received from the external reference;

retrieving the subsequent set of graphic pixel data associated with the subsequent set of graphic pixel coordinate-pairs; and

,

determining a subsequent set of display pixel data based on the retrieved subsequent set of graphic pixel data for rendering on the display.

8. The method of claim 7, wherein the subsequent set of display pixel coordinate-pairs is exclusive of the set of display pixel coordinate-pairs.

9. The method of claim 7, where the subsequent transform matrix represents a subsequent three-dimensional planar region on the display corresponding to a subsequent three-dimensional plane identified by the further orientation data.

10. The method of claim 7, wherein the graphics processing unit is associated with wearable glasses, and wherein the display comprises a first display portion, viewable by a first eye of a wearer of the glasses, and a second display portion, viewable by a second eye of the wearer of the glasses.

11. The method of claim 10, wherein the determined set of display pixel data is for rendering on the first display portion, the method further comprising:

applying a second transform matrix to the set of display pixel coordinate-pairs to obtain a second set of graphic pixel coordinate-pairs, the second transform matrix calculated using the orientation data received from the external reference;

retrieving a second set of graphic pixel data associated with the second set of graphic pixel coordinate-pairs; and

determining a second set of display pixel data based on the retrieved second set of graphic pixel data, the second set of display pixel data for rendering on the second display portion.

12. The method of claim 11, wherein the external reference comprises a plurality of infrared LED lights and wherein the orientation data comprises three-dimensional coordinates of the LED lights on a three-dimensional plane, and wherein the transform matrix and the second transform matrix are calculated based on the relative position of at least two of the infrared LED lights.

13. The method of claim 1, further comprising:

receiving the orientation data from the external reference, the orientation data identifying a three-dimensional plane; and

,

calculating the transform matrix using the orientation data, the transform matrix for orienting the set of display pixel coordinate-pairs on the three-dimensional plane and then identifying the set of graphic pixel coordinate-pairs projected onto the three-dimensional planar region that correspond to the set of display pixel coordinate-pairs on the three-dimensional plane.

14. The method of claim 1, further comprising:

applying a further transform matrix to the set of display pixel coordinate-pairs to obtain a further set of graphic pixel coordinate-pairs, the further transform matrix calculated using orientation data received from the external reference, the further transform matrix associated with a second three-dimensional planar region on the display, wherein the second three-dimensional planar region is different from the three-dimensional planar region;

retrieving a further set of graphic pixel data associated with the further set of graphic pixel coordinate-pairs; and

,

determining a further set of display pixel data based on the retrieved further set of graphic pixel data.

15. The method of claim 14, wherein applying the transform matrix to the set of display pixel coordinate-pairs and applying the further transform matrix to the set of display pixel coordinate-pairs is performed in parallel.

16. The method of claim 14, further comprising:

receiving further orientation data from the external reference, the further orientation data identifying a second three-dimensional plane; and

,

calculating the further transform matrix using the further orientation data, the second transform matrix associated with the second three-dimensional planar region on the display corresponding to the second three-dimensional plane.

17. The method of claim 16 wherein the further orientation data is the same as the orientation data and wherein the second three-dimensional plane is different from the three-dimensional plane.

18. The method of claim 1 wherein the set of graphic pixel data is retrieved from a graphic stored in memory associated with the graphics processing unit.

19. A graphics processing unit associated with a memory, the graphics processing unit comprising:

a controller configured to calculate a transform matrix based on orientation data received from an external reference, the orientation data representing a three-dimensional plane;

a warp unit in communication with the controller configured to apply the transform matrix to a set of display pixel coordinate-pairs to calculate a set of graphic pixel coordinate-pairs, wherein the set of display pixel coordinate-pairs is on the same scan-line in a raster scan process on the display, and wherein the transform matrix is a mapping from a two-dimensional coordinate system of the display to a three-dimensional planar region in space identified by the orientation data and a projection onto a two-dimensional plane;

a pixel-fetch module in communication with the warp unit configured to retrieve a set of graphic pixel data from a graphic stored in memory, the set of graphic pixel data associated with the set of graphic pixel coordinate-pairs;

a blending module in communication with the pixel-fetch module configured to determine the set of display pixel data associated with the set of display pixel coordinate-pairs, the display pixel data calculated based on the retrieved set of graphic pixel data;

a rendering module in communication with the blending module configured to render the set of display pixel data on the display; and

a second warp unit in communication with the controller and configured to apply a second transform matrix to a second set of display pixel coordinate-pairs to obtain a second set of graphic pixel coordinate-pairs and wherein the pixel-fetch module is configured to retrieve a second set of graphic pixel data from the image in memory, the second set of graphic pixel data associated with the second set of graphic pixel coordinate-pairs.

20. The graphics processing unit of claim 19, further comprising an orientation module configured to:

receive the orientation data from the external reference, the orientation data identifying a three-dimensional plane; and

,

calculate the transform matrix using the orientation data, the transform matrix for identifying the set of graphic pixel coordinate-pairs projected onto the three-dimensional planar region visible at the set of display pixel coordinate-pairs.

21. The graphics processing unit of claim 19, wherein the second warp unit is configured to apply the second transform matrix to the second set of display pixel coordinate-pairs and the warp unit is configured to apply the transform matrix to the set of display pixel coordinate-pairs in parallel.

22. The graphics processing unit of claim 21, wherein the display comprises wearable glasses having a first display portion, viewable by a first eye of a wearer of the glasses, and a second display portion, viewable by a second eye of the wearer of the glasses, wherein the rendering module is configured to render the set of display pixel data on the first display portion and to render the second set of display pixel data to the second display portion, thereby rendering a stereoscopic image on the display.

23. The graphics processing unit of claim 21, wherein the controller is configured to calculate the second transform matrix based on the orientation data received from the external reference, the second transform matrix associated with a second three-dimensional planar region on the display corresponding to the second three-dimensional plane.

24. A system for generating and displaying an image, the system comprising:

a memory;

a processor for executing instructions stored on the memory;

a display; and

,

a graphics processing unit connected to the memory, display and processor, the graphics processing unit comprising;

a control processor configured to calculate a transform matrix based on orientation data received from an external reference identifying a three-dimensional plane;

a warp unit in communication with the control processor configured to apply the transform matrix to a set of display pixel coordinate-pairs to obtain a set of graphic pixel coordinate-pairs, wherein the set of display pixel coordinate-pairs is on the same scan-line in a raster scan process on the display, and wherein the transform matrix is a mapping from a two-dimensional coordinate system of the display to a three-dimensional planar region in space identified by the orientation data and a projection onto a two-dimensional plane;

a pixel-fetch module in communication with the warp unit configured to retrieve a set of graphic pixel data from an image stored in memory, the set of graphic pixel data associated with the set of graphic pixel coordinate-pairs;

a blending module in communication with the pixel-fetch module configured to determine the set of display pixel data associated with the set of display pixel coordinate-pairs for a rendering on the display; and

a second warp unit in communication with the controller for applying a second transform matrix to a second set of display pixel coordinate-pairs to obtain a second set of graphic pixel coordinate-pairs and wherein the pixel-fetch module is configured to retrieve a second set of graphic pixel data from the image in memory, the second set of graphic pixel data associated with the second set of graphic pixel coordinate-pairs.

25. The system of claim 24, wherein the display comprises wearable glasses.

26. The system of claim 24, further comprising a sensor configured to receive the orientation data from the external reference, the sensor coupled to the control processor.