TEXT-BASED 3D AUGMENTED REALITY

• US 20120092329A1
• Filed: 06/28/2011
• Published: 04/19/2012
• Est. Priority Date: 10/13/2010
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Koo, Hyung-Il, You, Kisun, Lee, Te-Won, Baik, Young-Ki
• Application Number:
  US13/170,758
• Publication Number:
  US 20120092329A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

A particular method includes receiving image data from an image capture device and detecting text within the image data. In response to detecting the text, augmented image data is generated that includes at least one augmented reality feature associated with the text.

Claims

1. A method comprising:

receiving image data from an image capture device;

detecting text within the image data; and

in response to detecting the text, generating augmented image data that includes at least one augmented reality feature associated with the text.

2. The method of claim 1, wherein the text is detected without examining the image data to locate predetermined markers and without accessing a database of registered natural images.

3. The method of claim 1, wherein the image capture device comprises a video camera of a portable electronic device.

4. The method of claim 3, further comprising displaying the augmented image data at a display device of the portable electronic device.

5. The method of claim 1, wherein the image data corresponds to a frame of video data that includes the image data, and further comprising, in response to detecting the text, transitioning from a text detection mode to a tracking mode.

6. The method of claim 5, wherein a text region is tracked in the tracking mode relative to at least one other salient feature of the video data during multiple frames of the video data.

7. The method of claim 6, further comprising determining a pose of the image capture device, wherein the text region is tracked in three dimensions and wherein the augmented image data is positioned in the multiple frames according to a position of the text region and the pose.

8. The method of claim 1, wherein detecting the text includes estimating an orientation of a text region according to a projection profile analysis.

9. The method of claim 1, wherein detecting the text includes adjusting a text region to reduce a perspective distortion.

10. The method of claim 9, wherein adjusting the text region includes applying a transform that maps corners of a bounding box of the text region into corners of a rectangle.

11. The method of claim 9, wherein detecting the text includes:

generating proposed text data via optical character recognition; and

accessing a dictionary to verify the proposed text data.

12. The method of claim 11, wherein the proposed text data includes multiple text candidates and confidence data associated with the multiple text candidates, and wherein a text candidate corresponding to an entry of the dictionary is selected as verified text according to a confidence value associated with the text candidate.

13. The method of claim 1, wherein the at least one augmented reality feature is incorporated within the image data.

14. An apparatus comprising:

a text detector configured to detect text within image data received from an image capture device; and

a renderer configured to generate augmented image data, the augmented image data including augmented reality data to render at least one augmented reality feature associated with the text.

15. The apparatus of claim 14, wherein the text detector is configured to detect the text without examining the image data to locate predetermined markers and without accessing a database of registered natural images.

16. The apparatus of claim 14, further comprising the image capture device, wherein the image capture device comprises a video camera.

17. The apparatus of claim 16, further comprising:

a display device configured to display the augmented image data; and

a user input device, wherein the at least one augmented reality feature is a three-dimensional object and wherein the user input device enables user control of the three-dimensional object displayed at the display device.

18. The apparatus of claim 14, wherein the image data corresponds to a frame of video data that includes the image data, and wherein the apparatus is configured to transition from a text detection mode to a tracking mode in response to detecting the text.

19. The apparatus of claim 18, further comprising a tracking module configured to track a text region relative to at least one other salient feature of the video data during multiple frames of the video data while in the tracking mode.

20. The apparatus of claim 19, wherein the tracking module is further configured to determine a pose of the image capture device, wherein the text region is tracked in three dimensions and wherein the augmented image data is positioned in the multiple frames according to a position of the text region and the pose.

21. The apparatus of claim 14, wherein the text detector is configured to estimate an orientation of a text region according to a projection profile analysis.

22. The apparatus of claim 14, wherein the text detector is configured to adjust a text region to reduce a perspective distortion.

23. The apparatus of claim 22, wherein the text detector is configured to adjust the text region by applying a transform that maps corners of a bounding box of the text region into corners of a rectangle.

24. The apparatus of claim 22, wherein the text detector further comprises:

a text recognizer configured to generate proposed text data via optical character recognition; and

a text verifier configured to access a dictionary to verify the proposed text data.

25. The apparatus of claim 24, wherein the proposed text data includes multiple text candidates and confidence data associated with the multiple text candidates, and wherein the text verifier is configured to select as verified a text candidate corresponding to an entry of the dictionary according to a confidence value associated with the text candidate.

26. An apparatus comprising:

means for detecting text within image data received from an image capture device; and

means for generating augmented image data, the augmented image data including augmented reality data to render at least one augmented reality feature associated with the text.

27. A computer readable storage medium storing program instructions that are executable by a processor, the program instructions comprising:

code for detecting text within image data received from an image capture device; and

code for generating augmented image data, the augmented image data including augmented reality data to render at least one augmented reality feature associated with the text.

28. A method of tracking text in image data, the method comprising:

receiving image data from an image capture device, the image data including text;

processing at least a portion of the image data to locate corner features of the text; and

in response to a count of the located corner features not satisfying a threshold, processing a first region of the image data that includes a first corner feature to locate additional salient features of the text.

29. The method of claim 28, further comprising iteratively processing regions of the image data that include one or more of the located corner features until a count of the located additional salient features and the located corner features satisfies the threshold.

30. The method of claim 28, wherein the located corner features and the located additional salient features are located within a first frame of the image data, and further comprising tracking the text in a second frame of the image data based on the located corner features and the located additional salient features.

31. The method of claim 28, wherein the first region is centered on the first corner feature and wherein processing the first region includes applying a filter to locate at least one of an edge and a contour within the first region.

32. A method of tracking text in multiple frames of image data, the method comprising:

receiving image data from an image capture device, the image data including text;

identifying a set of features of the text in a first frame of the image data, the set of features including a first feature set and a second feature;

identifying a mapping that corresponds to a displacement of the first feature set in a current frame of the image data as compared to the first feature set in the first frame; and

in response to determining the mapping does not correspond to a displacement of the second feature in the current frame as compared to the second feature in the first frame, processing a region around a predicted location of the second feature in the current frame according to the mapping to determine whether the second feature is located within the region.

33. The method of claim 32, wherein processing the region includes applying a similarity measure to compensate for at least one of a geometric deformation and an illumination change between the first frame and the current frame.

34. The method of claim 33, wherein the similarity measure includes a normalized cross-correlation.

35. The method of claim 32, further comprising adjusting the mapping in response to locating the second feature within the region.

36. A method of estimating a pose of an image capture device, the method comprising:

receiving image data from the image capture device, the image data including text;

identifying a distorted bounding region enclosing at least a portion the text, the distorted bounding region at least partially corresponding to a perspective distortion of a regular bounding region enclosing the portion of the text;

determining a pose of the image capture device based on the distorted bounding region and a focal length of the image capture device; and

generating augmented image data including at least one augmented reality feature to be displayed at a display device, the at least one augmented reality feature positioned within the augmented image data according to the pose of the image capture device.

37. The method of claim 36, wherein identifying the distorted bounding region includes:

identifying pixels of the image data that correspond to the portion of the text; and

determining borders of the distorted bounding region to define a substantially smallest area that includes the identified pixels.

38. The method of claim 37, wherein the regular bounding region is rectangular and wherein the borders of the distorted bounding region form a quadrangle.