Systems and Methods for Implementing Augmented Reality

• US 20120154638A1
• Filed: 12/20/2011
• Published: 06/21/2012
• Est. Priority Date: 12/21/2010
• Status: Active Grant

• Current Assignee: Cyberlink Corporation
• Original Assignee: Cyberlink Corporation
• Inventors: Chou, Chen-Wei
• Granted Patent: US 9,721,387 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method implemented in a video playback system is described for incorporating augmented reality (AR) into a video stream. The method comprises determining a target pattern, determining an inner pattern in the target pattern, determining a relationship between the target pattern and the inner pattern, and receiving, by the video playback system, frames of the video stream. For each frame within the frame sequence, binarization is performed according to a predetermined threshold. Based on whether a location of the target pattern can be determined, a location of the inner pattern is determined. Based on the location of the inner pattern on received frames and the determined relationship between the target pattern and the inner pattern, a location of the target pattern is determined. The method further comprises displaying a virtual object with the target pattern on an output device based on the location of the target pattern.

Claims

1. A method implemented in a video playback system for incorporating augmented reality into a video stream, comprising:

determining a target pattern;

determining an inner pattern in the target pattern;

determining a relationship between the target pattern and the inner pattern;

receiving, by the video playback system, frames of the video stream;

for each frame within the frame sequence, performing binarization according to a predetermined threshold;

based on whether a location of the target pattern can be determined, determining a location of the inner pattern;

based on the location of the inner pattern on received frames and the determined relationship between the target pattern and the inner pattern, determining a location of the target pattern; and

displaying a virtual object with the target pattern on an output device based on the location of the target pattern.

2. The method of claim 1, wherein determining the location of the inner pattern and determining the location of the target pattern are performed if the location of the target pattern cannot be determined.

3. The method of claim 1, wherein determining the relationship between the target pattern and the inner pattern is performed according to a spacing and relative location of each of the corners of the inner pattern with respect to corners of the outer boundary of the target pattern.

4. The method of claim 1, wherein determining the relationship between the target pattern and the inner pattern is performed according to a transformation formula for transforming a mask of the target pattern to a mask of the inner pattern.

5. The method of claim 1, wherein the step of determining a location of the target pattern based on the location of the inner pattern on the received frames and the determined relationship between the target pattern and the inner pattern further comprises:

utilizing the location of the inner pattern on the received frames to compute parameters of a camera in the video playback system based on information derived during camera calibration; and

determining a location of the target pattern based on each of;

the location of the inner pattern on received frames;

the determined relationship between the target pattern and the inner pattern; and

the computed parameters of the camera.

6. The method of claim 1, wherein determining an inner pattern comprises determining one or more of a color and an intensity of a boundary of the inner pattern, wherein one or more of the color and intensity of the boundary is different than one or more of a color and an intensity of a connective region of the target pattern defined between the inner pattern and a remainder of the target pattern.

7. The method of claim 1, wherein determining the inner pattern is performed by inversing the color or intensity of the target pattern.

8. The method of claim 1, wherein the inner pattern is offset relative to a center of the target pattern or orientation relative to the target pattern or scale relative to the target pattern.

9. A video system for incorporating augmented reality (AR) into a video stream, comprising:

a marker module configured to determine a target pattern within the video stream, the marker module further configured to determine an inner pattern of the target pattern, wherein the marker module is further configured to determine a relationship between the target pattern and the inner pattern according to a spacing and relative location of each of the corners of the inner pattern with respect to corners of the outer boundary of the target pattern;

a tracker module configured to perform a tiered search of the target pattern within the video stream;

a location estimator configured to determine a location of the target pattern according to a search result of the tracker module and the relationship between the target pattern and the inner pattern determined by the marker module; and

an effects module configured to integrate a virtual object into the video stream according to the location of the target pattern.

10. The system of claim 9, wherein the tiered search performed by the tracker module comprises:

a first search of the target pattern; and

a second search of the inner pattern of the target pattern, the second search being performed only if the first search of the target pattern is unsuccessful.

11. The system of claim 10, wherein the first search and the second search are performed on a binarized form of the video stream.

12. The system of claim 10, wherein the first search and the second search are based on a pattern matching algorithm.

13. The system of claim 12, wherein the pattern matching algorithm comprises utilizing template matching for performing pattern matching,

wherein template matching is based on a first vector V1, a second vector V2, a value L1, and a value L2,

wherein L1 represents an absolute distance between the two vectors (V1-V2), and

wherein L2 represents a square of the distance between the two vectors (V1-V2)².

14. The system of claim 9, wherein the marker module is configured to determine the relationship between the target pattern and the inner pattern during a calibration phase performed by the video system.

15. The system of claim 9, wherein the location estimator is configured to determine a location of the target pattern based on one of:

a location of the target pattern if the first search by the tracker module is successful; and

an estimated location of the target pattern if the first search by the tracker module is unsuccessful, the estimated location being based on the location of the inner pattern and the determined relationship.

16. The system of claim 15, wherein the estimated location of the target pattern is based on predetermined characteristics of the target pattern, the predetermined characteristics comprising one or more of:

a size of the target pattern with respect to a field of view associated with the video stream, a size of the inner pattern with respect to the target pattern, and placement of the inner pattern with respect to the target pattern.

17. The system of claim 15, wherein the estimated location of the target pattern comprises a position of the target pattern calculated by the tracker module based on:

predetermined characteristics of the target pattern, and

the determined location of the second search.

18. A method implemented in a video playback system for incorporating augmented reality (AR) into a video stream, comprising:

determining a target pattern;

determining an inner pattern in the target pattern;

determining a relationship between the target pattern and the inner pattern;

receiving, by the video playback system, a frame sequence of the video stream;

performing a tiered search on a location of the target pattern; and

displaying a virtual object based on the location of the target pattern.

19. The method of claim 18, wherein the tiered search comprises:

a first tier search comprising searching a location of the target pattern as a whole; and

a second tier search comprising searching a location of the inner pattern.

20. The method of claim 18, wherein the second tier search is performed only if the first tier search is unsuccessful in determining a location of the target pattern, wherein the second tier search comprises determining a location of the inner pattern and based on the location of the inner pattern on the received frame and the determined relationship between the target pattern and the inner pattern.

21. The method of claim 18, further comprising:

for each frame within the frame sequence, performing binarization according to a predetermined threshold.

22. The method of claim 18, wherein performing binarization comprises assigning a binary value to each pixel within each frame according to one of:

edge detection, contrast, and detection of a region of interest (ROI).

23. The method of claim 18, wherein performing binarization comprises assigning a binary value to each pixel within each frame according to a threshold defined according to one of:

luminance value, saturation value, intensity value, hue value, and brightness value.

24. The method of claim 18, wherein the first tier search comprises matching a region within the frame to a template of the target pattern.

25. The method of claim 24, wherein matching a region within the frame to a template of the target pattern comprises matching a binarized region within the frame to a binary template of the target pattern.

26. The method of claim 24, wherein matching a region within the frame to a template of the target pattern is based on a first vector V1, a second vector V2, a value L1, and a value L2,

wherein L1 represents an absolute distance between the two vectors (V1-V2), and

wherein L2 represents a square of the distance between the two vectors (V1-V2)².

27. The method of claim 18, wherein determining the relationship between the target pattern and the inner pattern is performed according to a spacing and relative location of each of the corners of the inner pattern with respect to corners of the outer boundary of the target pattern.

28. The method of claim 18, wherein determining an inner pattern comprises determining one or more of a color and an intensity of a boundary of the inner pattern, wherein one or more of the color and intensity of the boundary is different than one or more of a color and an intensity of a connective region of the target pattern defined between the inner pattern and a remainder of the target pattern.

29. The method of claim 18, wherein the inner pattern is offset relative to a center of the target pattern or orientation relative to the target pattern or scale relative to the target pattern.