System and method for display of augmented reality

• US 10,497,180 B1
• Filed: 07/03/2018
• Issued: 12/03/2019
• Est. Priority Date: 07/03/2018
• Status: Active Grant

• Current Assignee: ‎AI-EKSP, OOO Co.
• Original Assignee: ‎AI-EKSP, OOO Co.
• Inventors: Perfilev, Denis, Samsonov, Pavel
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US16/026,908
• Time in Patent Office: 518 Days
• Field of Search: None
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 2219/2008 Assembling, disassembling
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/39 involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

Method of displaying augmented reality, including initializing the user device, launching a utility for determining device coordinates and device orientation; transmitting, to a server, user device parameters, the device coordinates and the device orientation; σMeH codec information for encoding received data; transmitting zero coordinates to server; continuously sending updated device coordinates and updated device orientation to the server; on server, inserting virtual objects into a virtual scene, using the zero coordinates and the updated device coordinates and orientation. Virtual camera duplicates movements of camera of the user device, and renders an image including inserted virtual objects on transparent background; encoding and transmitting the encoded image to the user device; continuing to transmit encoded images based on updated virtual objects, updated coordinates and orientation, as a video stream. On the user device, decoding the video stream; overlaying it onto camera images to generate a combined video; and displaying the combined video.

Claims

1. A method of displaying augmented reality on a user device, the method comprising:

initializing the user device, including launching a utility for determining device coordinates and device orientation;

connecting to a server over a network;

transmitting, to the server, user device parameters, the device coordinates and the device orientation;

requesting from the server, or transmitting to the server, codec information for encoding received data from the server;

transmitting, to the server, values of zero coordinates;

continuously sending updated device coordinates and updated device orientation to the server;

on the server, inserting virtual objects into a virtual scene that is maintained by the server, wherein the inserting uses the values of the zero coordinates and the updated device coordinates and updated device orientation,

wherein the virtual scene includes a virtual camera that duplicates all movements of the camera of the user device;

on the server, the virtual camera renders an image that includes the inserted virtual objects on a transparent background;

encoding the image on the server;

transmitting the encoded image to the user device;

continuing the transmit encoded images based on updated virtual objects, the updated device coordinates and updated device orientation, as a video stream;

on the user device, decoding the video stream;

overlaying the video stream onto images generated by the camera of the user device to generate a combined video; and

displaying the combined video on the user device.

2. The method of claim 1, wherein the user device parameters include screen resolution, operating system and data transmission protocol.

3. The method of claim 1, wherein the device coordinates, orientation and scale include a 4×

4 matrix transform.

4. The method of claim 1, wherein the zero coordinates are determined by aiming a camera of the user device at a marker.

5. The method of claim 1, wherein the zero coordinates are determined by selecting any point on the display that shows an image from the camera of the user device.

6. The method of claim 1, wherein the zero coordinates are specified by the user.

7. The method of claim 1, wherein the encoding of the image on the server encodes the image with the alpha channel.

8. The method of claim 1, wherein the encoding includes encoding of rendered objects as a first channel, and encoding of a transparency mask as a second channel.

9. The method of claim 1, wherein the encoding includes encoding of rendered objects encoding of a transparency mask into a single channel, using a codec that supports an alpha channel.

10. A system for displaying augmented reality, the system comprising:

a user device having a camera, a camera position acquisition unit, a camera image acquisition unit, a position and orientation determination unit, a data transmission unit, a data receiving unit, an image overlay unit, and a display;

a server having an image generation unit;

wherein, upon initialization, the user device launches a utility for determining device coordinates and device orientation using the position and orientation determination unit, and transmits values of zero coordinates to the server;

wherein the user device transmits, to the server over the network, user device parameters, the device position and the device orientation;

wherein the user device requests, from the server, codec information for decoding received data from the server;

wherein the user device continuously sends updated device coordinates and updated device orientation to the server;

wherein the server inserts virtual objects into a virtual scene that is maintained by the server, wherein the inserting uses the values of the zero coordinates and the updated device coordinates and updated device orientation,

wherein the virtual scene includes a virtual camera that duplicates all movements of the camera of the user device;

wherein the virtual camera renders an image that includes the inserted virtual objects on a transparent background;

wherein the server encodes the image using the codec and transmits the encoded image to the user device;

wherein the server continues transmitting encoded images based on updated virtual objects, the updated device coordinates and updated device orientation, as a video stream;

wherein the data receiving unit decodes the video stream;

wherein the image overlay unit overlays the video stream onto images generated by the camera of the user device to generate a combined video; and

wherein the user device shows the combined video on the display.

11. The system of claim 10, wherein the user device parameters include screen resolution, operating system and data transmission protocol.

12. The system of claim 10, wherein the device coordinates, orientation and scale include a 4×

4 matrix transform.

13. The system of claim 10, wherein the zero coordinates are determined by aiming a camera of the user device at a marker.

14. The system of claim 10, wherein the zero coordinates are determined by selecting any point on the display that shows an image from the camera of the user device.

15. The system of claim 10, wherein the zero coordinates are specified by the user.

16. The system of claim 10, wherein the encoding of the image on the server encodes the image with the alpha channel.

17. The system of claim 10, wherein the encoding includes encoding of rendered objects as a first channel, and encoding of a transparency mask as a second channel.

18. The system of claim 10, wherein the encoding includes encoding of rendered objects encoding of a transparency mask into a single channel, using a codec that supports an alpha channel.