Generating accurate augmented reality objects in relation to a real-world surface via a digital writing device

• US 10,297,088 B2
• Filed: 09/26/2017
• Issued: 05/21/2019
• Est. Priority Date: 09/26/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Rhodes, Tenell, Miller, Gavin S. P., Ceylan Aksit, Duygu, Ito, Daichi
• Primary Examiner(s): Wu, Chong
• Application Number:
  US15/716,450
• Publication Number:
  US 20190096129A1
• Time in Patent Office: 602 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The present disclosure includes systems, methods, computer readable media, and devices that can generate accurate augmented reality objects based on tracking a writing device in relation to a real-world surface. In particular, the systems and methods described herein can detect an initial location of a writing device, and further track movement of the writing device on a real-world surface based on one or more sensory inputs. For example, disclosed systems and methods can generate an augmented reality object based on pressure detected at a tip of a writing device, based on orientation of the writing device, based on motion detector elements of the writing device (e.g., reflective materials, emitters, or object tracking shapes), and/or optical sensors. The systems and methods further render augmented reality objects within an augmented reality environment that appear on the real-world surface based on tracking the movement of the writing device.

Claims

1. In a digital medium environment for generating augmented reality objects, a computer-implemented method of creating and arranging a digital object on an augmented reality representation of a real-world drawing space comprising:

detecting, by way of a pressure sensor of a writing device, an initial contact of the writing device with a real-world surface;

determining, in response to the detected initial contact and by way of a locator element associated with the writing device, an initial location of the writing device;

detecting real-world movement of a tip of the writing device relative to the initial location by;

tracking motion of a reflective motion detector element located on the writing device and that is operable to reflect light within a range of light wavelengths; and

determining, during the real-world movement, a position of the tip of the writing device relative to the reflective motion detector element; and

rendering, by way of an augmented reality device and based on the detected real-world movement of the tip of the writing device, an augmented reality object such that the augmented reality object appears to be drawn on the real-world surface.

2. The computer-implemented method of claim 1, further comprising analyzing, by way of the augmented reality device, a real-world environment to identify at least one real-world surface acceptable for composing.

3. The computer-implemented method of claim 2, further comprising providing, by way of the augmented reality device and in response to analyzing the at least one real-world surface, an indication that the at least one real-world surface is acceptable for composing.

4. The computer-implemented method of claim 1, further comprising:

determining that the real-world movement of the writing device is above a speed threshold;

in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first motion detector element; and

in response to determining that the real-world movement is no longer above the speed threshold, determining the real-world movement of the writing device based on a second motion detector element.

5. The computer-implemented method of claim 1, further comprising:

detecting an environmental circumstance associated with the writing device that affects tracking the writing device; and

determining, based on the detected environmental circumstance, one or more techniques to utilize to monitor the writing device.

6. The computer-implemented method of claim 5, wherein:

detecting the environmental circumstance comprises detecting that the writing device is writing on a non-reflective surface; and

determining the one or more techniques to utilize to monitor the writing device comprises selecting, based on detecting that the writing device is writing on the non-reflective surface, one or more of the reflective motion detector element, an emitter operable to project a visual indicator onto the non-reflective surface, an optical sensor operable to capture images corresponding to positional changes of the writing device, or a machine learning model.

7. The computer-implemented method of claim 1, wherein rendering the augmented reality object comprises:

detecting, by way of the pressure sensor associated with the writing device, that the writing device is contacting the real-world surface; and

generating, by way of the augmented reality device and in response to detecting that the writing device is contacting the real-world surface, a digital mark that appears to be drawn on the real-world surface and that follows the detected real-world movement of the writing device.

8. The computer-implemented method of claim 7, further comprising:

detecting, by way of the pressure sensor associated with the writing device, a pressure force applied to the writing device;

detecting, by way of an inertial measurement unit associated with the writing device, a tilt of the writing device; and

adjusting a compositional attribute of the digital mark generated by way of the augmented reality device based on the detected pressure force applied to the writing device and the detected tilt of the writing device.

9. The computer-implemented method of claim 1, further comprising generating, based on the rendered augmented reality object, a three-dimensional augmented reality model corresponding to the augmented reality object, wherein the three-dimensional augmented reality model is manipulable by a user within a three-dimensional augmented reality environment.

10. A non-transitory computer readable storage medium comprising instructions thereon that, when executed by at least one processor, cause a computer system to:

detect, by way of a pressure sensor of a writing device, an initial contact of the writing device with a real-world surface;

determine, in response to the detected initial contact and by way of a locator element associated with the writing device, an initial location of the writing device;

detect real-world movement of a tip of the writing device relative to the initial location by;

tracking motion of a motion detector element located on the writing device, wherein the motion detector element comprises at least one of a reflective element operable to reflect a light within a range of light wavelengths or an emitter operable to project a trackable visual indicator onto the real-world surface; and

determining, during the real-world movement, a position of the tip of the writing device relative to the motion detector element; and

render, by way of an augmented reality device and based on the detected real-world movement of the tip of the writing device, an augmented reality object such that the augmented reality object appears to be drawn on the real-world surface.

11. The non-transitory computer readable storage medium of claim 10, further comprising instructions thereon that, when executed by the at least one processor, cause the computer system to analyze, by way of the augmented reality device, a real-world environment to identify at least one real-world surface acceptable for composing.

12. The non-transitory computer readable storage medium of claim 11, further comprising instructions thereon that, when executed by the at least one processor, cause the computer system to provide, by way of the augmented reality device and in response to analyzing the at least one real-world surface, an indication that the at least one real-world surface is acceptable for composing.

13. The non-transitory computer readable storage medium of claim 10, further comprising instructions thereon that, when executed by the at least one processor, cause the computer system to:

determine that the real-world movement of the writing device is above a speed threshold;

in response to determining that the real-world movement is above the speed threshold, determine the real-world movement of the writing based on a first motion detector; and

in response to determining that the real-world movement is no longer above the speed threshold, determine the real-world movement of the writing device based on a second motion detector.

14. The non-transitory computer readable storage medium of claim 10, wherein the locator element comprises one or more of:

a reflective element operable to reflect a range of light wavelengths measurable by the augmented reality device or an emitter operable to project a visual indicator onto the real-world surface.

15. The non-transitory computer readable storage medium of claim 10, wherein the motion detector element is the reflective element and further comprising instructions that, when executed by the at least one processor, cause the computer system to detect the real-world movement of the writing device by tracking motion of the reflective element using a sensor operable to detect light within the range of light wavelengths.

16. The non-transitory computer readable storage medium of claim 10, wherein detecting the real-world movement of the writing device comprises training a machine learning model to predict, based on analyzing writing motions of a plurality of individuals, where a user will move the writing device.

17. A writing device for composing augmented reality objects for display by way of an augmented reality device based on tracking movement of the writing device on a real-world surface, the writing device comprising:

a housing having a tip on a first end of the housing;

a pressure sensor within the housing that, when triggered, indicates a pressure force applied to the tip;

an inertial measurement unit within the housing, comprising at least one accelerometer, that detects at least one of;

a location of the writing device or a tilt of the writing device;

a locator element operable to indicate a location of the writing device, the locator element being affixed to the housing and comprising at least one of;

an emitter that projects a visual indicator onto a writing surface or a reflective element that reflects a range of light wavelengths measurable by an augmented reality device;

a reflective motion detector element operable to indicate real-world movement the a tip of the housing based on a position of the tip of the housing relative to the reflective motion detector element; and

a transceiver element affixed to the housing that sends information associated with at least one of;

the pressure sensor, the inertial measurement unit, the locator element, or the reflective motion detector element.

18. The writing device of claim 17, wherein the locator element comprises the reflective element and the reflective element comprises an infrared element that reflects infrared light wavelengths.

19. The writing device of claim 17, wherein the locator element comprises the emitter and the emitter comprises a laser emitter.

20. The writing device of claim 17, further comprising an optical sensor that tracks movement of the writing device based on light deflection from the writing surface.