TRANSPARENT DISPLAY DEVICE, CONTROL METHOD THEREOF AND CONTROLLER THEREOF

• US 20180165857A1
• Filed: 12/12/2017
• Published: 06/14/2018
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute, Intellectual Property Innovation Corporation
• Original Assignee: Industrial Technology Research Institute, Intellectual Property Innovation Corporation
• Inventors: Lee, Cheng-Chung, Chen, Kuang-Jung, Wang, Sheng-Po, Chen, Heng-Yin
• Granted Patent: US 10,643,359 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A transparent display device, a control method thereof and a controller thereof are provided. The transparent display device includes a transparent display, a first sensor set, a second sensor set and a controller. The first sensor set senses user information in a first side. The second sensor set senses scene information in a second side. The controller analyzes and determines a location and a gaze direction of a user according to the user information, calculates a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, determines a target object corresponding to the first fixation point according to the scene information, inquires target object information corresponding to the target object, and presents the target object information according to the second fixation point by the transparent display.

Claims

1. A transparent display device, comprising:

a transparent display, comprising a first side and a second side opposite to the first side, wherein a display area of the transparent display is visually transparent;

a first sensor set, sensing user information in the first side;

a second sensor set, sensing scene information in the second side; and

a controller, coupled to the transparent display, the first sensor set and the second sensor set, wherein

the controller receives the user information sensed by the first sensor set to analyze and determine a location and a gaze direction of a user, and the controller calculates a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, determines a target object corresponding to the first fixation point according to the scene information, inquires target object information corresponding to the target object, and controls the transparent display to present the target object information according to the second fixation point.

2. The transparent display device as claimed in claim 1, wherein the controller obtains the location and the gaze direction of the user through the first sensor set, and extends the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point, searches whether the scene information comprises the target object according to a location of the first fixation point, and extends the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

3. The transparent display device as claimed in claim 2, wherein the gaze direction is generated according to the location of the user and a gaze angle of the user or according to touch information coordinates produced by the user in the first side of the transparent display.

4. The transparent display device as claimed in claim 1, wherein the controller calculates a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.

5. The transparent display device as claimed in claim 1, wherein the first sensor set comprises:

a first image capturing device, coupled to the controller, and obtaining an image of the first side of the transparent display, wherein the image comprises the user.

6. The transparent display device as claimed in claim 5, wherein the first sensor set further comprises a touch sensor,

the touch sensor obtains a touch point of the user in the first side of the transparent display, and the touch point corresponds to the second fixation point on the transparent display.

7. The transparent display device as claimed in claim 1, wherein the controller provides a user interface according to the gaze direction, a touch operation, voice information of the user or a gesture of the user, so as to provide information for the scene information through the transparent display.

8. The transparent display device as claimed in claim 1, wherein the controller cuts the scene information to obtain a plurality of sub scene information by using a clustering algorithm, determines that the first fixation point of the user is located at concerned sub scene information of one of the sub scene information, and takes the concerned sub scene information as the target object.

9. The transparent display device as claimed in claim 1, wherein the second sensor set comprises a second image capturing device.

10. A control method of a transparent display device, wherein the transparent display device comprises a transparent display, a first sensor set and a second sensor set, the control method comprising:

sensing user information in a first side of the transparent display;

sensing scene information in a second side of the transparent display, wherein the second side is opposite to the first side;

analyzing and determining a location and a gaze direction of a user according to the user information;

calculating a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user;

determining a target object corresponding to the first fixation point according to the scene information; and

inquiring target object info'"'"' cation corresponding to the target object, and controlling the transparent display to present the target object information according to the second fixation point.

11. The control method of the transparent display device as claimed in claim 10, wherein the step of calculating the first fixation point in the scene information and the second fixation point on the transparent display according to the location and the gaze direction of the user comprises:

extending the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point;

searching whether the scene information comprises the target object according to a location of the first fixation point; and

extending the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

12. The control method of the transparent display device as claimed in claim 10, further comprising:

controlling an image capturing device in the second sensor set to translate to a location closest to the location of the user after the location of the user is obtained.

13. The control method of the transparent display device as claimed in claim 10, further comprising:

calculating a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.

14. The control method of the transparent display device as claimed in claim 10, further comprising:

providing a user interface according to the gaze direction, a touch operation, voice information of the user or a gesture of the user, so as to provide information for the scene information through the transparent display.

15. The control method of the transparent display device as claimed in claim 10, wherein the step of determining the target object corresponding to the first fixation point according to the scene information comprises:

cutting the scene information to obtain a plurality of sub scene information by using a clustering algorithm;

determining that the first fixation point of the user is located at concerned sub scene information of one of the sub scene information; and

taking the concerned sub scene information as the target object.

16. A controller, adapted to a transparent display device, comprising:

an input control module, coupled to a first sensor set of the transparent display device, receiving user information in a first side of the transparent display device, and analyzing and determining a location and a gaze direction of a user according to the user information;

a viewing area capturing module, coupled to a second sensor set of the transparent display device, and receiving scene information in a second side of the transparent display device;

an algorithm analysis module, coupled to the input control module and the viewing area capturing module, calculating a first fixation point in the scene information and a second fixation point on the transparent display according to the location and the gaze direction of the user, and determining a target object corresponding to the first fixation point according to the scene information;

an object identification module, coupled to the algorithm analysis module, and inquiring target object information corresponding to the target object; and

an info nation display module, coupled to a transparent display in the transparent display device, and controlling the transparent display to present the target object information according to the second fixation point.

17. The controller as claimed in claim 16, wherein the algorithm analysis module extends the gaze direction to a place of the scene information in the second side of the transparent display to obtain the first fixation point, searches whether the scene information comprises the target object according to a location of the first fixation point, and extends the gaze direction to the transparent display to obtain the second fixation point on the transparent display.

18. The controller as claimed in claim 17, wherein the gaze direction is generated according to the location of the user and a gaze angle of the user or according to touch information coordinates produced by the user in the first side of the transparent display.

19. The controller as claimed in claim 16, wherein the input control module determines an object distance between the target object in the first fixation point and the transparent display, and determines a scene mode according to the object distance, and the input control module selectively obtains the required user information from a plurality of devices in the first sensor set according to the scene mode,

the algorithm analysis module calculates coordinates of the first fixation point on an image capturing plane of the second sensor set according to the scene mode.

20. The controller as claimed in claim 16, wherein the viewing area capturing module calculates a field of vision range viewed through the transparent display according to the location of the user, so as to obtain the scene information in the field of vision range from the second sensor set.