Distance adaptive holographic displaying method and device based on eyeball tracking

• US 10,101,807 B2
• Filed: 11/28/2015
• Issued: 10/16/2018
• Est. Priority Date: 11/28/2014
• Status: Active Grant

• Current Assignee: Shenzhen Smart Superv Co., Ltd.
• Original Assignee: Shenzhen Magic Eye Technology Co., Ltd.
• Inventors: Liu, Meihong, Gao, Wei, Xu, Wanliang
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Abimbola, Kehinde O
• Application Number:
  US14/953,342
• Publication Number:
  US 20160154458A1
• Time in Patent Office: 1,053 Days
• CPC Class Codes: G03H 1/0005 Adaptation of holography to specific applications holographic optical element G02B5/32; holographic scanner G02B26/106; recognition using holographic mask G06V10/88; holographic memories G11B7/0065, G11C13/042
• CPC Class Codes: G03H 2001/0088 for video-holography, i.e. integrating hologram acquisition, transmission and display
• CPC Class Codes: G03H 2210/30 3D object
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A distance adaptive holographic displaying method and device based on eyeball tracking are disclosed. The method includes: detecting a human face of a user and acquiring positions of eyes of the user, wherein the positions of the eyes of the user comprise a distance between the eyes and a displaying screen; adjusting the size of an object in a holographic scene along a z-axis direction in the holographic three-dimensional display according to the distance, and displaying the object in the displaying screen; adjusting an emergent direction of the light emitted by the displaying screen to the direction of the left eye and the right eye of the eyes so that the different individual holographic three-dimensional images of the left eye and the right eye are imaged into the left eye and the right eye respectively; and tracking the change of the positions of the eyes, and repeating the aforesaid steps to dynamically adjust viewpoints and the displayed content. The present disclosure can adapt to different groups of people at different distances and conform to the visual principle of human beings, thereby improving user experiences.

Claims

1. A distance adaptive holographic displaying method based on eyeball tracking, the distance adaptive holographic displaying method applied in a distance adaptive holographic displaying device comprising a camera and a distance meter;

the distance adaptive holographic displaying method comprising the following steps of;

controlling the camera to track change of positions of eyes of a user, moving the distance meter together with the camera to measure a distance between the eyes and the distance meter, and calculating a distance between the eyes and a displaying screen of the device;

adjusting the size of an object in a holographic scene along a z-axis direction in the holographic three-dimensional display according to the distance between the eyes and the displaying screen, adjusting viewpoints of different individual holographic three-dimensional images that need to be projected into a left eye and a right eye of the eyes according to the size of the object in the holographic scene along the z-axis direction, and displaying the viewpoints in the displaying screen;

adjusting an emergent direction of the light emitted by the displaying screen to the direction of the left eye and the right eye of the eyes so that the different individual holographic three-dimensional images of the left eye and the right eye are imaged into the left eye and the right eye respectively; and

tracking the change of the positions of the eyes, and repeating the aforesaid steps to dynamically acquire the different individual holographic three-dimensional images of the left eye and the right eye so that, when the eyes are moving, the left eye and the right eye can always acquire individual holographic three-dimensional images having the correct viewpoints of the holographic three-dimensional display and of the size of the object in the holographic scene along the z-axis direction;

wherein if the distance between the eyes and the displaying screen is smaller than a first threshold value, the size of the object in the holographic scene is not enlarged along the z-axis direction; and

if the distance between the eyes and the display screen is larger than a second threshold value, the size of the object in the holographic scene is not reduced along the z-axis direction.

2. The method of claim 1, wherein adjusting the size of an object in a holographic scene along a z-axis direction in the holographic three-dimensional display according to the distance further comprises adjusting the content of the object in the holographic scene in the holographic three-dimensional display, wherein the content of the object in the holographic scene is the appearance content of the object within a region of the object that can be covered by the sight of the both human eyes, and the step of adjusting the size of an object in a holographic scene along a z-axis direction in the individual holographic three-dimensional display according to the distance comprises the following steps of:

determining whether the distance is getting larger or smaller; and

if the distance is getting larger, reducing the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display, and meanwhile, enlarging the region of the object that can be covered by the sight of the both human eyes so as to obtain the content of the object in the holographic scene in the adjusted holographic three-dimensional display; and

otherwise, enlarging the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display, and meanwhile, reducing the region of the object that can be covered by the sight of the both human eyes so as to obtain the content of the object in the holographic scene in the adjusted holographic three-dimensional display.

3. The method of claim 2, wherein the step of determining whether the distance is getting larger or smaller further comprises the following steps of:

determining whether the distance is smaller than a first threshold value, and if the distance is smaller than the first threshold value, then the distance is getting smaller, and the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display will not be enlarged and the content of the object in the holographic scene in the holographic three-dimensional display will not be adjusted; and

determining whether the distance is larger than a second threshold value, and if the distance is larger than the second threshold value, then the distance is getting larger, and the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display will not be reduced and the content of the object in the holographic scene in the holographic three-dimensional display will not be adjusted.

4. The method of claim 1, wherein a three-dimensional standard coordinate system is established by taking a center of the displaying screen as an origin, with a xy coordinate plane being parallel to the displaying screen, a positive direction of a x-axis being horizontal and directed to a right direction, a positive direction of a y-axis being vertical to the x-axis and directed downwards, and a positive direction of the z-axis being perpendicular to the xy coordinate plane and directed towards a direction of the human eyes;

when the eyes are moving, each of the distances is acquired in the following way;

acquiring, according to data obtained by detecting the human face, the following coordinate values corresponding to the eyeballs of the eyes;

(x_L1,y_L1) and (x_R1,y_R1), (x_L2,y_L2) and (x_R2,y_R2), . . . , (x_Ln,y_Ln) and (x_Rn,y_Rn), wherein L represents the left eye, R represents the right eye, n represents that the eyes move from position 1 to position n, and a three-dimensional standard coordinate system is adopted; and

acquiring a distance Z between the eyes and the displaying screen according to a distance between the distance meter and the eyes measured by the distance meter when the eyes are moving, a coordinate value of the distance meter in the coordinate system, and coordinate values corresponding to the eyeballs of the eyes;

or acquiring the distance Z between the eyes and the displaying screen according to the distance between the distance meter and the eyes, and an angle of an optical axis of the distance meter with respect to a plane of the displaying screen;

or directly taking the distance between the distance meter and the eyes as the distance Z between the eyes and the displaying screen.

5. The method of claim 1, further comprising the following step of:

acquiring different individual holographic sounds that need to be projected into a left ear and a right ear of ears according to the viewpoints of the holographic three-dimensional display and the size of the object in the holographic scene along the z-axis direction, and outputting the different individual holographic sounds corresponding to the left ear and the right ear respectively.

6. A distance adaptive holographic displaying device based on eyeball tracking, comprising:

a camera, controlled to track change of positions of eyes of a user;

a distance meter, moved together with the camera to measure a distance between the eyes and the distance meter for a calculation of a distance between the eyes and the displaying screen of the displaying device;

a displaying module, being configured to adjust the size of an object in a holographic scene along a z-axis direction in the holographic three-dimensional display according to the distance, to adjust viewpoints of different individual holographic three-dimensional images that need to be projected into a left eye and a right eye of the eyes according to the size of the object in the holographic scene along the z-axis direction, and to display the viewpoints in the displaying screen; and

an adjusting module, being configured to adjust an emergent direction of the light emitted by the displaying screen to the direction of the left eye and the right eye of the eyes so that the different individual holographic three-dimensional images of the left eye and the right eye are imaged into the left eye and the right eye respectively;

wherein the displaying module adjusts the different individual holographic three-dimensional images of the left eye and the right eye dynamically so that, when the eyes are moving, the left eye and the right eye can always acquire individual holographic three-dimensional images having the correct viewpoints of the holographic three-dimensional display and of the size of the object in the holographic scene along the z-axis direction by means of the adjusting module;

wherein if the distance between the eyes and the displaying screen is smaller than a first threshold value, the displaying module does not enlarge the size of the object in the holographic scene along the z-axis direction; and

if the distance between the eyes and the displaying screen is larger than a second threshold value, the displaying module does not reduce the size of the object in the holographic scene.

7. The device of claim 6, wherein the displaying module comprises a displaying unit, an adjusting unit and a first determining unit, and the adjusting unit is configured to adjust the size of an object in a holographic scene along a z-axis direction in the holographic three-dimensional display and the content of the object in the holographic scene in the holographic three-dimensional display according to the distance, wherein the content of the object in the holographic scene is the appearance content of the object within a region of the object that can be covered by the sight of the both human eyes;

the first determining unit is configured to determine whether the distance is getting larger or smaller;

if the first determining unit determines that the distance is getting larger, the adjusting unit is controlled to reduce the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display, and meanwhile, enlarge the region of the object that can be covered by the sight of the both human eyes so as to obtain the content of the object in the holographic scene in the adjusted holographic three-dimensional display; and

otherwise, the adjusting unit is controlled to enlarge the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display, and meanwhile, reduce the region of the object that can be covered by the sight of the both human eyes so as to obtain the content of the object in the holographic scene in the adjusted holographic three-dimensional display; and

the displaying unit is configured to display the holographic three-dimensional image adjusted by the adjusting unit.

8. The device of claim 7, wherein the displaying module further comprises a second determining unit configured to determine whether the distance is smaller than a first threshold value, and if the distance is smaller than the first threshold value, then the distance is getting smaller, and the adjusting unit is controlled neither to enlarge the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display nor to adjust the content of the object in the holographic scene in the holographic three-dimensional display;

and the second determining unit is further configured to determine whether the distance is larger than a second threshold value, and if the distance is larger than the second threshold value, then the distance is getting larger, and the adjusting unit is controlled neither to reduce the size of the object in the holographic scene along the z-axis direction in the holographic three-dimensional display nor to adjust the content of the object in the holographic scene in the holographic three-dimensional display.

9. The device of claim 6, further comprising a calculating unit, wherein the calculating unit is specifically configured to:

establish a three-dimensional standard coordinate system by taking a center of the displaying screen as an origin, with a xy coordinate plane being parallel to the displaying screen, a positive direction of a x-axis being horizontal and directed to a right direction, a positive direction of a y-axis being vertical to the x-axis and directed downwards, and a positive direction of the z-axis being perpendicular to the xy coordinate plane and directed towards a direction of the human eyes;

acquire, according to data obtained by detecting the human face by the image capturing unit, the following coordinate values corresponding to the eyeballs of the eyes;

(x_L1,y_L1) and (x_R1,y_R1), (x_L2,y_L2) and (x_R2,y_R2), . . . , (x_Ln,y_Ln) and (x_Rn,y_Rn), wherein L represents the left eye, R represents the right eye, n represents that the eyes move from position 1 to position n, the xy coordinate plane is parallel to the surface of the displaying screen, and a three-dimensional standard coordinate system is adopted; and

acquire a distance Z between the eyes and the displaying screen according to a distance between the distance meter and the eyes measured by the distance meter when the eyes are moving, a coordinate value of the distance meter in the coordinate system, and coordinate values corresponding to the eyeballs of the eyes;

or acquire the distance Z between the eyes and the displaying screen according to the distance between the distance meter and the eyes, and an angle of an optical axis of the distance meter with respect to a plane of the displaying screen;

or directly take the distance between the distance meter and the eyes as the distance Z between the eyes and the displaying screen.

10. The device of claim 6, further comprising a sound adjusting module specifically configured to adjust different individual holographic sounds that need to be projected into a left ear and a right ear of ears according to the viewpoints of the holographic three-dimensional display and the size of the object in the holographic scene along the z-axis direction, and to output the different individual holographic sounds corresponding to the left ear and the right ear respectively.