METHOD FOR DISPLAYING VIRTUAL IMAGE, STORAGE MEDIUM AND ELECTRONIC DEVICE THEREFOR

• US 20180239425A1
• Filed: 02/21/2018
• Published: 08/23/2018
• Est. Priority Date: 02/21/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jang, Won-Nam
• Granted Patent: US 11,068,050 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An electronic device comprises a first camera module, a display device, and a processor functionally connected with the first camera module and the display device. The processor may be configured to determine a size of at least one eye area included in an image obtained by the first camera module while displaying a virtual image using the display device and to control a degree of display of the virtual image corresponding to the size of the at least one eye area. Other various embodiments are also available.

Claims

1. An electronic device, comprising:

a first camera module;

a display device; and

a processor that is functionally connected with each of the first camera module and the display device, wherein the processor is configured to determine a size of at least one eye area included in a first image obtained by the first camera module while displaying a first virtual image using the display device and to control a degree of display of the first virtual image based on the determined size of the at least one eye area.

2. The electronic device of claim 1, wherein the processor is further configured to vary a degree of display of at least one from among a transparency, a highlight, an inversion, a color change, and a contour of the first virtual image based on the determined size of the at least one eye area and to display the first virtual image that has the varied degree of display.

3. The electronic device of claim 1, wherein the processor is further configured to vary a degree of display of at least one from among a size and a ratio of the first virtual image based on the determined size of the at least one eye area and to display the first virtual image that has the varied degree of display.

4. The electronic device of claim 1, wherein when the display device is a projection display device, the processor is further configured to display the first virtual image of which the degree of display has been varied to be distinguished from an external object by using the projection display device.

5. The electronic device of claim 1, wherein the first camera module is positioned on a first surface of the electronic device, wherein the electronic device further comprises a second camera module positioned on a second surface of the electronic device which is positioned opposite the first surface, and wherein the processor is further configured to vary the degree of display of a second virtual image that is associated with a second image obtained by the second camera module based on the determined size of the at least one eye area and to display the second virtual image on a portion of the second image by using the display device.

6. The electronic device of claim 5, wherein the processor is further configured to control to output the second virtual image that differs in at least one from among a transparency and a resolution to be distinguished from the second image, based on the determined size of the at least one eye area.

7. The electronic device of claim 5, wherein the second image includes an image for a left eye and an image for a right eye.

8. The electronic device of claim 1, wherein the processor is further configured to determine whether a state of an eye closing is maintained for a predetermined time by using the determined size of the at least one eye area and to release the display of the first virtual image when the state of the eye closing is maintained for the predetermined time.

9. The electronic device of claim 8, wherein the processor is further configured to determine whether the state of the eye closing is maintained for a predetermined time by using the determined size of the at least one eye area after releasing the display of the first virtual image and to display the first virtual image by using the display device when the state of the eye closing is maintained for the predetermined time.

10. The electronic device of claim 1, wherein the processor is further configured to determine whether the size of the at least one eye area is less than or equal to a mean eye area size and to vary the degree of display of the first virtual image based on the determined size of the at least one eye area when a state in which the size of the at least one eye area is determined as being less than or equal to the mean eye area size is maintained for a predetermined time and to display the first virtual image that has the varied degree of display.

11. The electronic device of claim 10, wherein the processor is further configured to stepwise increase at least one from among a transparency and a highlight of the first virtual image based on the determined size of the at least one eye area and to display the first virtual image that has the increased at least one from among the transparency and the highlight.

12. The electronic device of claim 5, wherein the processor is further configured to obtain a third image captured for at least one external object by the second camera module, to recognize the at least one external object included in the third image, to associate each of the at least one external object with at least one virtual image based on a result of the recognition, and to control to display the at least one virtual image associated with each of the at least one external object.

13. The electronic device of claim 12, wherein the at least one virtual image is an advertisement image that corresponds to the at least one external object.

14. A method for displaying a first virtual image in an electronic device, the method comprising:

obtaining a first image by using a first camera module while displaying the first virtual image;

determining a size of at least one eye area included in the obtained first image; and

controlling a degree of display of the first virtual image based on the determined size of the at least one eye area.

15. The method of claim 14, wherein the controlling the degree of display of the first virtual image includes varying a degree of display of at least one from among a transparency, a highlight, an inversion, a color change, and a contour of the first virtual image based on to the determined size of the at least one eye area and displaying the first virtual image that has the varied degree of display.

16. The method of claim 14, wherein the controlling the degree of display of the first virtual image includes varying a degree of display of at least one from among a size and a ratio of the first virtual image based on the determined size of the at least one eye area and displaying the first virtual image that has the varied degree of display.

17. The method of claim 14, wherein when the electronic device includes a projection display device, the controlling the degree of display of the first virtual image includes displaying the first virtual image of which the degree of display has been varied to be distinguished from an external object by using the projection display device.

18. The method of claim 14, further comprising obtaining a second image by a second camera module positioned on a second surface which is positioned opposite a first surface of the electronic device upon which the first camera module is positioned and varying the degree of display of a second virtual image associated with the second image based on the determined size of the at least one eye area and displaying the second virtual image on a portion of the first image.

19. The method of claim 18, wherein the varying the degree of display of the second virtual image associated with the second image and the displaying the second virtual image include displaying the second virtual image that stepwise differs in at least one from among a transparency and a resolution based on the determined size of the at least one eye area to be distinguished from the second image.

20. The method of claim 14, further comprising:

determining whether a state of an eye closing is maintained for a predetermined time by using the determined size of the at least one eye area; and

releasing the display of the first virtual image when the state of the eye closing is maintained for the predetermined time.

21. The method of claim 14, wherein the varying the degree of display of the first virtual image and the displaying the first virtual image include

determining whether the size of the at least one eye area is less than or equal to a mean eye area size;

and

when a state in which the size of the at least one eye area is determined as being less than or equal to the mean eye area size is maintained for a predetermined time, varying the degree of display of the first virtual image based on the determined size of the at least one eye area and displaying the first virtual image that has the varied degree of display.

22. A non-transitory storage medium storing instructions, the instructions configured to be executed by at least one processor to enable the at least one processor to perform at least one operation, the at least one operation comprising:

obtaining an image by using a first camera module while displaying a virtual image;

determining a size of at least one eye area included in the obtained image; and

controlling a degree of display of the virtual image based on the determined size of the at least one eye area.

23. An electronic device, comprising:

a first camera module configured to obtain a first image;

a display device configured to display each of the first image and a first virtual image; and

a processor configured to determine at least one parameter that relates to an eye area included in the first image and to adjust at least one characteristic of the first virtual image based on the determined at least one parameter,

wherein the display device is further configured to display the first image in conjunction with the first virtual image for which the at least one characteristic has been adjusted.

24. The electronic device of claim 23, wherein the at least one parameter that relates to the eye area includes at least one from among a height of a pupil, a diameter of the pupil, a height of an iris, a diameter of the iris, a first state in which an eye included in the eye area is fully open, a second state in which the eye included in the eye area is partially closed, and a third state in which the eye included in the eye area is fully closed.

25. The electronic device of claim 23, wherein the at least one characteristic of the first virtual image includes at least one from among a transparency, a highlight, an inversion, a color, and a contour.

26. The electronic device of claim 23, wherein the processor is further configured to determine whether an eye included in the eye area maintains a state of an eye closure for at least a predetermined amount of time and to further adjust the at least one characteristic of the first virtual image based on whether the eye is determined to have maintained the state of the eye closure,

wherein the state of the eye closure includes at least one from among a first state in which the eye is fully open, a second state in which the eye is partially closed, and a third state in which the eye is fully closed.

27. The electronic device of claim 26, wherein the processor is further configured to determine when the state of the eye closure changes, and to further adjust the at least one characteristic of the first virtual image based on a change in the state of the eye closure.