Technique for recording augmented reality data

• US 10,573,078 B2
• Filed: 03/16/2018
• Issued: 02/25/2020
• Est. Priority Date: 03/17/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Nourai, Reza, Liebenow, Michael Harold, Taylor, Robert Blake, Wyatt, Robert
• Primary Examiner(s): Tseng, Charles
• Application Number:
  US15/924,144
• Publication Number:
  US 20180268611A1
• Time in Patent Office: 711 Days
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]

Abstract

Disclosed is an improved approach for generated recordings from augmented reality systems from the perspective of a camera within the system. Instead of re-using rendered virtual content from the perspective of the user'"'"'s eyes for AR recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That additional virtual content is combined with image frames generated by a camera to form the AR recording.

Claims

1. A method for generating an image recording, comprising

capturing a camera image frame with a camera;

identifying pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from a user'"'"'s perspective;

rendering camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera'"'"'s perspective; and

composing the image recording by combining the camera image frame with the camera pose virtual content,

wherein a single inertial measurement unit (IMU) generates both the user pose data and the camera pose data.

2. The method of claim 1, wherein a single graphics processing unit (GPU) processing cycle includes processing to:

render the user pose virtual content based on the user pose data, and render the camera pose virtual content based on the camera pose data.

3. The method of claim 2, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for a same frame.

4. The method of claim 2, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for different frames.

5. The method of claim 1, wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.

6. The method of claim 1, wherein a media encoder encodes one or more image frames that combine the camera image frame with the camera pose virtual content.

7. The method of claim 6, wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

8. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for generating an image recording comprising:

capturing a camera image frame with a camera;

identifying pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from a user'"'"'s perspective;

rendering camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera'"'"'s perspective; and

composing the image recording by combining the camera image frame with the camera pose virtual content,

wherein a single inertial measurement unit (IMU) generates both the user pose data and the camera pose data.

9. The computer program product of claim 8, wherein a single graphics processing unit (GPU) processing cycle includes processing to:

render the user pose virtual content based on the user pose data, and render the camera pose virtual content based on the camera pose data.

10. The computer program product of claim 9, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for a same frame.

11. The computer program product of claim 9, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for different frames.

12. The computer program product of claim 8, wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.

13. The computer program product of claim 8, wherein a media encoder encodes one or more image frames that combine the camera image frame with the camera pose virtual content.

14. The computer program product of claim 13, wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

15. A system for generating an image recording, comprising

an augmented realty display device that displays three dimensional content;

a camera that captures a camera image frame;

an inertial measurement unit (IMU) that identifies pose data corresponding to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

a single graphics processing unit (GPU) that renders both user pose virtual content and camera pose virtual content, the user pose virtual content being rendered for immediate display to a user in the augmented reality device, wherein the user pose virtual content corresponds to first virtual content from a user'"'"'s perspective and the camera pose virtual content corresponds to second virtual content from the camera'"'"'s perspective; and

a compositor that composes the image recording by combining the camera image frame with the camera pose virtual content.

16. The system of claim 15, further comprising a media encoder that encodes one or more image frames that combine the camera image frame with the camera pose virtual content.

17. The system of claim 16, wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

18. The system of claim 15, wherein a single GPU processing cycle includes processing to:

render the user pose virtual content based on the user pose data, and render the camera pose virtual content based on the camera pose data.

19. The system of claim 18, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for a same frame.

20. The system of claim 18, wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data rendered during the single GPU processing cycle is for different frames.

21. The system of claim 15, wherein the compositor generates the image recording by combining the camera image frame with the camera pose virtual content.