Transparent display device, control method thereof and controller thereof

• US 10,643,359 B2
• Filed: 12/12/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute, Intellectual Property Innovation Corporation
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lee, Cheng-Chung, Chen, Kuang-Jung, Wang, Sheng-Po, Chen, Heng-Yin
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Tran, Kim Thanh T
• Application Number:
  US15/838,363
• Publication Number:
  US 20180165857A1
• Time in Patent Office: 875 Days
• Field of Search: 345633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A transparent display device, a control method thereof and a controller thereof are provided. The transparent display device includes a transparent display, a first sensor set, a second sensor set and a controller. The first sensor set senses user information in a first side. The second sensor set senses scene information in a second side. The controller analyzes and determines a location and a gaze direction of a user according to the user information, calculates a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, determines a target object corresponding to the first fixation point according to the scene information, inquires target object information corresponding to the target object, and presents the target object information according to the second fixation point by the transparent display.

Claims

1. A transparent display device, comprising:

a transparent display, comprising a first side and a second side opposite to the first side, wherein a display area of the transparent display is visually transparent;

a first sensor set, sensing user information of a user located in the first side;

a second sensor set, sensing scene information comprising an integration of an image capturing plane of the second sensor set and a plurality of objects in the image capturing plane in the second side; and

a controller, coupled to the transparent display, the first sensor set and the second sensor set, wherein

the controller receives the user information sensed by the first sensor set to analyze and determine a location and a gaze direction of the user, and the controller calculates a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, wherein the first fixation point is defined as a point on the image capturing plane integrated in the scene information that is calculated according to the location and the gaze direction of the user, and the second fixation point is defined as a position on the transparent display that is calculated according to the location of the user, a gaze angle of the user and a distance between the user and the transparent display,

the controller determines a target object corresponding to the first fixation point according to the scene information, inquires target object information corresponding to the target object, and controls the transparent display to present the target object information according to the second fixation point, wherein the target object corresponding to the first fixation point is one of the objects in the image capturing plane that is determined by the controller as to be gazed by the user according to the scene information, and the target object information comprises information related to the target object that is stored in the controller and configured to be displayed by the transparent display according to the second fixation point.

2. The transparent display device as claimed in claim 1, wherein the controller obtains the location and the gaze direction of the user through the first sensor set, and extends the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point, searches whether the scene information comprises the target object according to a location of the first fixation point, and extends the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

3. The transparent display device as claimed in claim 2, wherein the gaze direction is generated according to the location of the user and a gaze angle of the user or according to touch information coordinates produced by the user in the first side of the transparent display.

4. The transparent display device as claimed in claim 1, wherein the controller calculates a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.

5. The transparent display device as claimed in claim 1, wherein the first sensor set comprises:

a first image capturing device, coupled to the controller, and obtaining an image of the first side of the transparent display, wherein the image comprises the user.

6. The transparent display device as claimed in claim 5, wherein the first sensor set further comprises a touch sensor,

the touch sensor obtains a touch point of the user in the first side of the transparent display, and the touch point corresponds to the second fixation point on the transparent display.

7. The transparent display device as claimed in claim 1, wherein the controller provides a user interface according to the gaze direction, a touch operation, voice information of the user or a gesture of the user, so as to provide information for the scene information through the transparent display.

8. The transparent display device as claimed in claim 1, wherein the controller cuts the scene information to obtain a plurality of sub scene information by using a clustering algorithm, determines that the first fixation point of the user is located at concerned sub scene information of one of the sub scene information, and takes the concerned sub scene information as the target object.

9. The transparent display device as claimed in claim 1, wherein the second sensor set comprises a second image capturing device.

10. A control method of a transparent display device, wherein the transparent display device comprises a transparent display, a first sensor set and a second sensor set, the control method comprising:

sensing user information of a user located in a first side of the transparent display;

sensing scene information comprising an integration of an image capturing plane of the second sensor set and a plurality of objects in the image capturing plane in a second side of e transparent display, wherein the second side is opposite to the side;

analyzing and determining a location and a gaze direction of the user according to the user information;

calculating a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, wherein the first fixation point is defined as a point on the image capturing plane integrated in the scene information that is calculated according to the location and the gaze direction of the user, and the second fixation point is defined as a position on the transparent display that is calculated according to the location of the user, a gaze angle of the user and a distance between the user and the transparent display;

determining a target object corresponding to the first fixation point according to the scene information, wherein the target object corresponding to the first fixation point is one of the objects in the image capturing plane that is determined to be gazed by the user according to the scene information; and

inquiring target object information corresponding to the target object, and controlling the transparent display to present the target object information according to the second fixation point, wherein the target object information comprises information related to the target object that is stored in the controller and configured to be displayed by the transparent display according to the second fixation point.

11. The control method of the transparent display device as claimed in claim 10, wherein the step of calculating the first fixation point in the scene information and the second fixation point on the transparent display according to the location and the gaze direction of the user comprises:

extending the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point;

searching whether the scene information comprises the target object according to a location of the first fixation point; and

extending the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

12. The control method of the transparent display device as claimed in claim 10, further comprising:

controlling an image capturing device in the second sensor set to translate to a location closest to the location of the user after the location of the user is obtained.

13. The control method of the transparent display device as claimed in claim 10, further comprising:

calculating a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.

14. The control method of the transparent display device as claimed in claim 10, further comprising:

providing a user interface according to the gaze direction, a touch operation, voice information of the user or a gesture of the user, so as to provide information for the scene information through the transparent display.

15. The control method of the transparent display device as claimed in claim 10, wherein the step of determining the target object corresponding to the first fixation point according to the scene information comprises:

cutting the scene information to obtain a plurality of sub scene information by using a clustering algorithm;

determining that the first fixation point of the user is located at concerned sub scene information of one of the sub scene information; and

taking the concerned sub scene information as the target object.

16. A controller, adapted to a transparent display device, and configured to:

receive user information of a user located in a first side of the transparent display device, and analyze and determine a location and a gaze direction of the user according to the user information;

receive scene information comprising an integration of a captured image of a second side of the transparent display device and a plurality of objects in the captured image;

calculate a first fixation point in the scene information and a second fixation point on a transparent display of the transparent display device according to the location and the gaze direction of the user, wherein the first fixation point is defined as a point on an image capturing plane of the captured image integrated in the scene information that is calculated according to the location and the gaze direction of the user, and the second fixation point is defined as a position on the transparent display that is calculated according to the location of the user, a gaze angle of the user and a distance between the user and the transparent display;

determine a target object corresponding to the first fixation point according to the scene information, wherein the target object corresponding to the first fixation point is one of the objects in the captured image that is determined to be gazed by the user according to the scene information;

inquire target object information corresponding to the target object, wherein the target object information comprises information related to the target object that is stored in the controller; and

control the transparent display to present the target object information according to the second fixation point.

17. The controller as claimed in claim 16, further configured to:

extend the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point, search whether the scene information comprises the target object according to a location of the first fixation point, and extend the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

18. The controller as claimed in claim 17, wherein the gaze direction is generated according to the location of the user and a gaze angle of the user or according to touch information coordinates produced by the user in the first side of the transparent display.

19. The controller as claimed in claim 16, further configured to:

determine an object distance between the target object in the first fixation point and the transparent display, and determine a scene mode according to the object distance, and selectively obtain the required user information from a plurality of devices in a first sensor set according to the scene mode,

calculate, from the captured image, coordinates of the first fixation point on the image capturing plane of a second sensor set according to the scene mode.

20. The controller as claimed in claim 16, further configured to calculate a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.