THREE-DIMENSIONAL DISPLAY METHOD, TERMINAL DEVICE, AND STORAGE MEDIUM

• US 20200134927A1
• Filed: 12/31/2019
• Published: 04/30/2020
• Est. Priority Date: 09/03/2018
• Status: Active Grant

• Current Assignee: Guangdong Virtual Reality Technology Co., Ltd.
• Original Assignee: Guangdong Virtual Reality Technology Co., Ltd.
• Inventors: HUANG, Sibin, DAI, Jingwen, HE, Jie
• Granted Patent: US 11,380,063 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 30/22 of the stereoscopic type
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A three-dimensional display method is provided and comprises: acquiring target space coordinates of a target marker in a physical space; converting the target space coordinates into rendering coordinates in a virtual space; acquiring data of a virtual object to be displayed, and rendering, based on the data of the virtual object and the rendering coordinates, the virtual object to obtain a left-eye display content and a right-eye display content of the virtual object; and displaying the left-eye display content and the right-eye display content. The left-eye display content is projected onto a first optical lens, the right-eye display content is projected onto a second optical lens, and the first optical lens and the second optical lens respectively reflect the left-eye display content and the right-eye display content to human eyes. The method can achieve alignment display of the virtual object with the target marker, and a stereoscopic display.

Claims

1. A three-dimensional display method, comprising:

determining, by a terminal device, target space coordinates of a target marker in a physical space;

converting, by the terminal device, the target space coordinates into rendering coordinates in a virtual space;

acquiring, by the terminal device, data of a virtual object to be displayed;

rendering, by the terminal device, the virtual object based on the data of the virtual object and the rendering coordinates to obtain a left-eye display content and a right-eye display content of the virtual object; and

displaying, by the terminal device, the left-eye display content and the right-eye display content, wherein the left-eye display content is projected onto a first optical lens, the right-eye display content is projected onto a second optical lens, and the first optical lens and the second optical lens respectively reflect the left-eye display content and the right-eye display content to a user'"'"'s eyes.

2. The method of claim 1, wherein the converting the target space coordinates into the rendering coordinates in the virtual space comprises:

reading stored conversion parameters between a first spatial coordinate system and a second spatial coordinate system, wherein the first spatial coordinate system is a spatial coordinate system with a tracking camera as an origin in the physical space, and the second spatial coordinate system is a spatial coordinate system with a virtual camera as an origin in the virtual space; and

converting the target space coordinates into the rendering coordinates in the virtual space based on the conversion parameters.

3. The method of claim 2, wherein the method further comprises acquiring the conversion parameters, wherein the acquiring the conversion parameters comprises:

displaying a virtual marker;

acquiring first coordinates of a physical marker in the first spatial coordinate system when an alignment determination operation by a user is detected, wherein the alignment determination operation indicates that the virtual marker is aligned with the physical marker;

acquiring second coordinates of the virtual marker in the second spatial coordinate system; and

calculating the conversion parameters between the first spatial coordinate system and the second spatial coordinate system based on the first coordinates of the physical marker and the second coordinates of the virtual marker corresponding to the physical marker.

4. The method of claim 3, further comprising:

after acquiring the conversion parameters, fine adjusting at least one of a first camera parameter of the tracking camera and a second camera parameter of the virtual camera.

5. The method of claim 3, wherein the physical marker is disposed in a movement controllable mechanism, and the movement controllable mechanism is connected to the terminal device;

the method further comprises;

prior to the acquiring the first coordinates of the physical marker in the first spatial coordinate system, receiving a movement control operation by the user;

transmitting a movement instruction to the movement controllable mechanism based on the movement control operation; and

moving the movement controllable mechanism based on the movement instruction.

6. The method of claim 2, wherein the virtual camera comprises a left virtual camera and a right virtual camera;

and the rendering the virtual object to obtain a left-eye display content and a right-eye display content of the virtual object comprises;

calculating pixel coordinates in the left virtual camera of the virtual object and pixel coordinates in the right virtual camera of the virtual object based on the rendering coordinates; and

generating the left-eye display content according to the pixel coordinates in the left virtual camera, and the right-eye display content according to the pixel coordinates in the right virtual camera.

7. The method of claim 1, wherein the displaying the left-eye display content and the right-eye display content comprises:

processing the left-eye display content and the right-eye display content based on an optical distortion model to obtain a left-eye pre-distorted image corresponding to the left-eye display content and a right-eye pre-distorted image corresponding to the right-eye display content, the optical distortion model being configured to fit an optical distortion generated by an optical lens; and

displaying the left-eye pro-distorted image and the right-eye pre-distorted image, wherein the left-eye pre-distorted image is projected onto the first optical lens and reflected by the first optical lens to a left eye of the user, and the right-eye pre-distorted image is projected onto the second optical lens and reflected by the second optical lens to a right eye of the user, so as to form a virtual image of an undistorted three-dimensional display content.

8. The method of claim 7, the method further comprising constructing the optical distortion model, wherein the constructing the optical distortion model comprises:

reading optical manufacturer data of the optical lens, the optical manufacturer data including coordinate data of an experimental image and coordinate data of an distorted virtual image corresponding to the experimental image; and

performing a polynomial fitting on the coordinate data of the experimental image and the coordinate data of the distorted virtual image to construct the optical distortion model.

9. The method of claim 8, wherein the performing the polynomial fitting comprises:

calculating a first distortion parameter and a second distortion parameter of the optical distortion model based on the coordinate data of the experimental image and the coordinate data of the distorted virtual image corresponding to the experimental image, wherein the first distortion parameter is a coefficient for fitting a distortion of the optical lens in a first direction, and the second distortion parameter is a coefficient for fitting a distortion of the optical lens in a second direction; and

constructing the optical distortion model based on the first distortion parameter and the second distortion parameter.

10. The method of claim 8, further comprising:

after the reading the optical manufacturer data of the optical lens, adjusting the coordinate data of the distorted virtual image based on a display parameter, wherein the display parameter comprises at least one of a zoom ratio, a screen size, a pixel size, or a position of an optical center of a screen of the terminal device.

11. A terminal device, comprising:

one or more processors;

a memory, the memory being coupled to the processor and storing a computer program stored therein, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform operations of;

acquiring target space coordinates of a target marker in a physical space;

converting the target space coordinates into rendering coordinates in a virtual space;

acquiring data of a virtual object to be displayed, and rendering the virtual object based on the data on the virtual object and the rendering coordinates; and

generating, by the terminal device, a left-eye display content and a right-eye display content of the virtual object according to the rendered virtual object;

a display, configured to display the left-eye display content and the right-eye display content; and

an optical module comprising a first optical lens and a second optical lens, wherein the left-eye display content is projected onto a first optical lens, the right-eye display content is projected onto a second optical lens, and the first optical lens and the second optical lens respectively reflect the left-eye display content and the right-eye display content to a user'"'"'s eyes.

12. The terminal device of claim 11, wherein the converting the target space coordinates into the rendering coordinates in the virtual space comprises:

reading stored conversion parameters between a first spatial coordinate system and a second spatial coordinate system, wherein the first spatial coordinate system is a spatial coordinate system with a tracking camera as an origin in the physical space, and the second spatial coordinate system is a spatial coordinate system with a virtual camera as an origin in the virtual space; and

converting the target space coordinates into the rendering coordinates in the virtual space based on the conversion parameters.

13. The terminal device of claim 12, wherein the processor is further configured to perform an operation of acquiring the conversion parameters, and the operation of acquiring the conversion parameters comprises:

displaying, by the display, a virtual marker;

acquiring first coordinates of a physical marker in the first spatial coordinate system when an alignment determination operation by a user is detected, wherein the alignment determination operation indicates that the virtual marker is aligned with the physical marker, and the virtual marker corresponds to the physical marker;

acquiring second coordinates of the virtual marker in a second spatial coordinate system; and

calculating the conversion parameters between the first spatial coordinate system and the second spatial coordinate system based on the first coordinates of the physical marker and the second coordinates of the virtual marker corresponding to the physical marker.

14. The terminal device of claim 11, wherein the displaying the left-eye display content and the right-eye display content comprises:

processing the left-eye display content and the right-eye display content based on an optical distortion model to obtain a left-eye pre-distorted image corresponding to the left-eye display content and a right-eye pre-distorted image corresponding to the right-eye display content, the optical distortion model being configured to fit an optical distortion generated by an optical lens; and

displaying the left-eye pre-distorted image and the right-eye pre-distorted image, wherein the left-eye pre-distorted image is projected onto the first optical lens and reflected by the first optical lens to a left eye of the user, and the right-eye pre-distorted image is projected onto the second optical lens and reflected by the second optical lens to a right eye of the user, so as to form a virtual image of an undistorted three-dimensional display content.

15. The terminal device of claim 14, wherein the processor is further configured to perform an operation of constructing the optical distortion model, and the operation of constructing the optical distortion model comprises:

reading optical manufacturer data of the optical lens, the optical manufacturer data comprising coordinate data of an experimental image and coordinate data of an distorted virtual image corresponding to the experimental image; and

performing a polynomial fitting on the coordinate data of the experimental image and the coordinate data of the distorted virtual image to construct the optical distortion model.

16. A computer readable storage medium having a program code stored thereon, wherein the program code, when executed by a processor, causes the processor to perform blocks of:

determining, by a terminal device, target space coordinates of a target marker in a physical space;

converting, by the terminal device, the target space coordinates into rendering coordinates in a virtual space;

acquiring, by the terminal device, data of a virtual object to be displayed;

rendering, by the terminal device, the virtual object based on the data of the virtual object and the rendering coordinates to obtain a left-eye display content and a right-eye display content of the virtual object; and

displaying, by the terminal device, the left-eye display content and the right-eye display content, wherein the left-eye display content is projected onto a first optical lens, the right-eye display content is projected onto a second optical lens, the first optical lens and the second optical lens respectively reflect the left-eye display content and the right-eye display content to a user'"'"'s eyes.

17. The computer readable medium of claim 16, wherein said converting the target space coordinates into the rendering coordinates in the virtual space comprises:

reading stored conversion parameters between a first spatial coordinate system and a second spatial coordinate system, wherein the first spatial coordinate system is a spatial coordinate system with a tracking camera as an origin in the physical space, and the second spatial coordinate system is a spatial coordinate system with a virtual camera as an origin in the virtual space; and

converting the target space coordinates into the rendering coordinates in the virtual space based on the conversion parameters.

18. The computer readable medium of claim 17, wherein the program code, when being called by the processor, further causes the processor to perform an operation of acquiring the conversion parameters, and the operation of acquiring the conversion parameters comprises:

displaying a virtual marker;

acquiring first coordinates of a physical marker in the first spatial coordinate system when an alignment determination operation by a user is detected, wherein the alignment determination operation indicates that the virtual marker is aligned with the physical marker;

acquiring second coordinates of the virtual marker in the second spatial coordinate system; and

calculating the conversion parameters between the first spatial coordinate system and the second spatial coordinate system based on the first coordinates of the physical marker and the second coordinates of the virtual marker corresponding to the physical marker.

19. The computer readable medium of claim 16, wherein said displaying the left-eye display content and the right-eye display content comprises:

processing the left-eye display content and the right-eye display content based on an optical distortion model to obtain a left-eye pre-distorted image corresponding to the left-eye display content and a right-eye pre-distorted image corresponding to the right-eye display content, the optical distortion model being configured to fit an optical distortion generated by an optical lens; and

displaying the left-eye pre-distorted image and the right-eye pre-distorted image, wherein the left-eye pre-distorted image is projected onto the first optical lens and reflected by the first optical lens to a left eye of the user, and the right-eye pre-distorted image is projected onto the second optical lens and reflected by the second optical lens to a right eye of the user, so as to form a virtual image of an undistorted three-dimensional display content.

20. The computer readable medium of claim 19, wherein the program code, when being called by the processor, further causes the processor to perform an operation of constructing the optical distortion model, and the operation of constructing the optical distortion model comprises:

reading optical manufacturer data of the optical lens, the optical manufacturer data including coordinate data of an experimental image and coordinate data of an distorted virtual image corresponding to the experimental image; and

performing a polynomial fitting on the coordinate data of the experimental image and the coordinate data of the distorted virtual image to construct the optical distortion model.