Optical stylus interaction

• US 9,354,748 B2
• Filed: 02/13/2012
• Issued: 05/31/2016
• Est. Priority Date: 02/13/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lutian, John M., Wang, Liang, Powell, Karlton D., Bathiche, Steven Nabil
• Primary Examiner(s): Moon, Seokyun
• Assistant Examiner(s): Acha, III, Josemarie G
• Application Number:
  US13/371,725
• Publication Number:
  US 20130207937A1
• Time in Patent Office: 1,569 Days
• Field of Search: 345179-184
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/042 by opto-electronic means

Abstract

Optical stylus interaction techniques are described. In an implementation, a display of a computing device includes optical sensors capable of detecting images projected by a stylus. A stylus may include a projection system to project various images used to convey interaction information that may be interpreted by the computing device. Based on recognition of different projected images, a context for interaction of the stylus may be ascertained and corresponding operations may be performed by the computing device. This may include resolving a spatial position of the stylus relative to the display device as well as movement of the stylus and images that defines various stylus-based gestures. In addition, the environment for optical stylus interaction enables a writing mode that emulates natural writing by mapping different projectable images to changes in pressure applied to the stylus when in contact with the display.

Claims

1. A computing device comprising:

a display device including one or more optical sensors configured to detect images projected onto a surface of the display device by a stylus;

an image decoder module configured to;

decode a detected image to ascertain a corresponding context for interaction with the computing device using the stylus by recognizing the detected image from multiple different images that each represent different contexts, the multiple different images each corresponding to a graphic that results from illuminating a series of image elements of the stylus differently than the series of image elements is illuminated for other graphics; and

cause performance of one or more actions by the computing device that are associated with the ascertained context.

2. A computing device as described in claim 1, wherein the display device comprises a sensor-in-pixel (SIP) panel having the optical sensors associated with respective pixels of the display device.

3. A computing device as described in claim 1, wherein the image decoder module is further configured to resolve the spatial position of the stylus based upon decoding of the detected image.

4. A computing device as described in claim 1, wherein:

the image decoder module is further configured to recognize a gesture input via the stylus based upon decoding of the detected image; and

the one or more actions correspond to the recognized gesture.

5. A computing device as described in claim 1, wherein:

the context comprises an operational mode for interaction with the computing device using the stylus that is defined by the detected image; and

performance of one or more actions comprises switching to the operational mode.

6. A computing device as described in claim 1, wherein the image decoder module is further configured to detect movement of the stylus based upon decoding of the detected image.

7. A computing device as described in claim 1, wherein:

the detected image defines a pressure level for a writing mode of the stylus; and

the one or more actions comprise emulating pressure sensitive writing by adjusting attributes of digital ink traces applied via the stylus to match the pressure level defined by the detected image.

8. A computing device as described in claim 1, wherein the image decoder module is further configured to determine an identity of the stylus based upon an identifier for the stylus that is projected as part of the detected image.

9. A method implemented by one or more modules at least partially in hardware of a computing device, the method comprising:

detecting one or more images projected onto a display device associated with the computing device using a stylus;

resolving a spatial position of the stylus relative to the display device by;

decoding the one or more projected images that, when decoded, reveal information that is indicative of the spatial position relative to the display device, the decoding including analyzing the one or more projected images relative to known patterns of a mapping of multiple different projectable images that each represent different information indicative of the spatial position, at least one of the multiple different projectable images corresponding to a graphic containing asymmetric content that is analyzable to facilitate resolution of the spatial position; and

ascertaining projection characteristics of the one or more projected images that are further indicative of the spatial position relative to the display device; and

performing one or more actions corresponding to the spatial position that is resolved by the decoding of the one or more projected images and the ascertaining of the projection characteristics of the one or more projected images.

10. A method as described in claim 9, further comprising:

recognizing an image projected by the stylus that is indicative of a writing mode of the stylus and conveys information regarding a pressure level to apply to digital ink traces input via the stylus; and

in response to recognition of the image indicative of the writing mode, activating the writing mode and causing digital ink traces input via the stylus to have attributes defined for the pressure level conveyed by the image.

11. A method as described in claim 9, wherein the images are detected via one or more optical sensors incorporated with the display device.

12. A method as described in claim 9, wherein the detecting and the decoding of the one or more projected images by the computing device are based on optical projections of the one or more projected images by the stylus that do not involve radio frequency (RF) communications.

13. A method as described in claim 9, wherein ascertaining the projection characteristics of the one or more projected images to resolve the spatial position comprises determining a height of the stylus relative to the display based at least in part upon size calculations for the one or more projected images.

14. A method as described in claim 9, wherein ascertaining the projection characteristics of the one or more projected images to resolve the spatial position comprises processing the asymmetric content of the one or more projected images to identify a rotational position of the stylus around an optical axis along which the one or more projected images are projected.

15. A method as described in claim 9, wherein ascertaining the projection characteristics of the one or more projected images to resolve the spatial position comprises resolving the spatial position for six degrees of freedom of the stylus based on analysis of image size, distortion of image elements, and orientation of the asymmetric content for the one or more projected images.

16. A method as described in claim 9, wherein decoding the one or more projected images to resolve the spatial position comprises recognizing a gesture based upon a detection of movement of the stylus in accordance with the spatial position.

17. A method as described in claim 9, wherein the one or more actions comprise control functions to manipulate operations of the computing device determined based in part upon the spatial position.

18. A stylus comprising:

a projection system;

one or more image elements embodying image information for corresponding images to enable projection of the images via the projection system to a target computing device; and

a mode manager module configured to cause the projection system to project different images using the image information embodied by the one or more image elements in response to manipulation of the stylus, the different images each comprising a different combination of graphical content and projected to convey different interaction information for the stylus optically for decoding by the target computing device, the different projected images interpretable by the target computing device with projection characteristics of detected projected images to recognize one or more of a spatial position of the stylus, an activated mode for the stylus, or a gesture produced by manipulation of the stylus.

19. A stylus as described in claim 18, wherein:

the one or more image elements comprise diffractive optical elements having image information to reproduce recorded holographic images; and

the projection system comprises a laser projection system that employs a laser diode to illuminate the diffractive optical elements and form the corresponding images to convey the interaction information optically to the target computing device.

20. A stylus as described in claim 18, further comprising a pressure switch to measure pressure applied to a tip of the stylus, wherein the mode manager module is further configured to:

determine a pressure level measured by the pressure switch; and

cause the projection system to project a particular image indicative of the determined pressure level to enable the target computing device to adjust attributes of digital ink traces input via the stylus to match the determined pressure level that is conveyed optically by the particular image.