Method and apparatus for controlling virtual monitor

• US 20120139914A1
• Filed: 12/06/2011
• Published: 06/07/2012
• Est. Priority Date: 12/06/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jeon, Kwang Ha
• Granted Patent: US 9,911,230 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2210/21 Collision detection, intersection
• CPC Class Codes: G06T 2210/52 Parallel processing

Abstract

An apparatus and method for controlling a virtual monitor in an augmented reality display system are provided. The method for controlling a virtual monitor in an augmented reality display system, including a camera for photographing an image for determining a location of the virtual monitor and for photographing an image of a user'"'"'s finger, and a graphic device including window images, includes parallel-processing the image for determining a location of the virtual monitor and the window images and mapping the window images to a set location to create the virtual monitor, extracting a finger tip from the image of the user'"'"'s finger, stencil-buffering an image of the finger tip not to render the image of the finger in the virtual monitor, determining whether the finger tip collides with the stencil-buffered virtual monitor, and controlling an operation of the virtual monitor according to the determination of the finger tip collision.

Claims

1. An apparatus for controlling a virtual monitor in an augmented reality display system, the apparatus comprising:

a camera for photographing an image for determining a location of the virtual monitor and for photographing an image of a user'"'"'s finger;

a graphic device including window images;

a Parallel Tracking And Mapping (PTAM) processor for parallel-processing the image for determining a location of the virtual monitor and the window images and for mapping the window images to a set location to create the virtual monitor;

a finger tip extractor for extracting a finger tip from the image of the user'"'"'s finger;

a stencil buffer for stencil-buffering an image of the finger tip not to render the image of the finger in the virtual monitor;

a collision determinator for determining whether the finger tip collides with the stencil-buffered virtual monitor; and

a function processor for controlling an operation of the virtual monitor according to whether the finger tip collides with the stencil-buffered virtual monitor.

2. The apparatus of claim 1, further comprising an image compensator for receiving and compensating for the image of the user'"'"'s finger using a Rentinex algorithm to compensate for and to transfer, to the finger tip extractor, a unique color in the image of the user'"'"'s finger changed due to illumination.

3. The apparatus of claim 2, wherein the Retinex algorithm performs an operation of removing an influence of the illumination using the following equation:

B(x,y)=R(x,y)*I(x,y)

where, B is a color, R is a unique reflection, I is the illumination, and (x,y) is the location of a pixel.

4. The apparatus of claim 2, wherein the PTAM processor tracks the image for determining the location to extract characteristic points, forms a three-dimensional space using the extracted characteristic points to determine a location of the virtual monitor, forms a plane at the determined location of the virtual monitor using openGL, and maps the window image on the formed plane to form the virtual monitor.

5. The apparatus of claim 4, wherein the PTAM processor repeats parallel processing the image for determining the location and the window images according to a user setting to form virtual multi-monitors at different locations.

6. The apparatus of claim 5, wherein the finger tip extractor separates a finger from the compensated image of a finger using a Convex hull algorithm, and determines a direction of a finger from a center of an area and a center of a contour to extract the finger tip.

7. The apparatus of claim 6, wherein the function processor comprises a head mount display unit, controls an operation of the virtual monitor according to a determination result of the collision processor, and expresses the operation of the virtual monitor in a three-dimensional space.

8. A method for controlling a virtual monitor in an augmented reality display system, including a camera for photographing an image for determining a location of the virtual monitor and for photographing an image of a user'"'"'s finger, and a graphic device including window images, the method comprising:

parallel-processing the image for determining a location of the virtual monitor and the window images and mapping the window images to a set location to create the virtual monitor;

extracting a finger tip from the image of the user'"'"'s finger;

stencil-buffering an image of the finger tip not to render the image of the finger in the virtual monitor;

determining whether the finger tip collides with the stencil-buffered virtual monitor; and

controlling an operation of the virtual monitor according to whether the finger tip collides with the stencil-buffered virtual monitor.

9. The method of claim 8, further comprising receiving and compensating for the image of the user'"'"'s finger using a Rentinex algorithm to compensate for a unique color in the image of the user'"'"'s finger changed due to illumination.

10. The method of claim 9, wherein the using of the Retinex algorithm comprises performing an operation of removing an influence of the illumination using the following equation:

B(x,y)=R(x,y)*I(x,y)

where, B is a color, R is a unique reflection, I is the illumination, and (x,y) is the location of a pixel.

11. The method of claim 9, wherein the parallel-processing of the image for determining a location of the virtual monitor and the window images comprises:

tracking the image for determining the location to extract characteristic points;

forming a three-dimensional space using the extracted characteristic points to determine a location of the virtual monitor;

forming a plane at the determined location of the virtual monitor using openGL; and

mapping the window image on the formed plane to form the virtual monitor.

12. The method of claim 11, wherein the parallel-processing of the image for determining a location of the virtual monitor and the window images comprises:

repeating parallel processing of the image for determining the location and the window images according to a user setting to form virtual multi-monitors at different locations.

13. The method of claim 12, herein the extracting of the finger tip comprises:

separating a finger from the compensated image of a finger using a Convex hull algorithm; and

determining a direction of the finger from a center of an area and a center of a contour to extract the finger tip.

14. The method of claim 13, wherein, the controlling of the operation of the virtual monitor comprises:

controlling an operation of the virtual monitor according to a determination result of the collision processor; and

expressing the operation of the virtual monitor in a three-dimensional space such that a user views the expressed operation of the virtual monitor.