Gestural annotations

• US 9,552,345 B2
• Filed: 02/28/2014
• Issued: 01/24/2017
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Nicholas Yen-Cherng, Sellen, Abigail Jane, Yoon, Dongwook
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): Cortes, Howard
• Application Number:
  US14/192,936
• Publication Number:
  US 20150248388A1
• Time in Patent Office: 1,061 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 65/401 wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Gestural annotation is described, for example where sensors such as touch screens and/or cameras monitor document annotation events made by a user of a document reading and/or writing application. In various examples the document annotation events comprise gestures recognized from the sensor data by a gesture recognition component. For example, the gestures may be in-air gestures or touch screen gestures. In examples, a compressed record of the sensor data is computed using at least the recognized gestures, document state and timestamps. In some examples the compressed record of the sensor data is used to facilitate consumption of the annotation events in relation to the document by a second user. In some examples the sensor data comprises touch sensor data representing electronic ink; and in some examples the sensor data comprises audio data capturing speech of a user.

Claims

1. A method at a computing device comprising:

rendering, by an application that at least one of reads or writes documents, a document at a display device, the application executing on the computing device or at a remote computing entity in communication with the computing device;

receiving sensor data from one or more sensors monitoring user annotation events associated with the rendered document, whilst also,

monitoring state data of the application;

recognizing one or more gestures by analyzing the received sensor data;

storing the state data with associated timestamps;

storing the recognized gestures with associated timestamps; and

calculating and storing a compressed record of the sensor data on the basis of the recognized gestures, the state data, and the timestamps, wherein the compressed record of the sensor data comprises identifiers of a plurality of views of the document having associated recognized gestures, the plurality of views of the document together forming a storyboard.

2. The method according to claim 1, wherein the compressed record of the sensor data comprises thumbnail image locations of one or more coverage regions of a gesture with respect to one or more thumbnail images of parts of the document.

3. The method according to claim 1, wherein the compressed record of the sensor data comprises locations within the document of one or more presence marks which may be used to indicate presence of a gesture with respect to the document.

4. The method according to claim 1, wherein the compressed record of the sensor data comprises the location and size of a region in the document which is associated with one of the recognized gestures, and where that recognized gesture is known to control manipulation of the document in the application.

5. The method according to claim 1, wherein the compressed record of the sensor data comprises identifiers indicating whether the recognized gestures are known to control manipulation of the document in the application.

6. The method according to claim 1, wherein the sensor data comprises one or more of:

touch sensor data, image data, or audio data.

7. The method according to claim 1 wherein the compressed record of the sensor data further comprises metadata describing the sensor data.

8. The method according to claim 7 wherein the metadata comprises any of:

an identifier of the user, or a location of the computing device.

9. The method according to claim 1, wherein the compressed record of the sensor data further comprises an animation of at least part of one of the recognized gestures.

10. The method according to claim 1, wherein the sensor data comprises touch sensor data and the compressed record of the sensor data further comprises electronic ink identifiers.

11. The method according to claim 1, wherein the sensor data comprises audio sensor data and the compressed record of the sensor data further comprises a transcript of speech of the user calculated from the audio sensor data.

12. The method according to claim 1 wherein the sensor data comprises audio sensor data and the compressed record of the sensor data further comprises identifiers of audio annotations of the user.

13. A method at a computing device comprising:

rendering a document at a display device using output of an application which is executing on the computing device or at a remote computing entity in communication with the computing device, the application configured to at least one of read or write documents;

accessing a compressed record of sensor data, the sensor data being from one or more sensors that monitored first user annotation events associated with the rendered document, the compressed record of sensor data comprising information about gestures recognized from the sensor data, state data of the application at the time of sensing the sensed data, identifiers of a plurality of views of the document having associated recognized gestures, and timestamps of the sensed data and of the state data;

using the compressed record of sensor data to enable a second user to consume the annotation events in conjunction with the rendered document, wherein using the compressed record includes displaying the plurality of views of the document as a storyboard.

14. A method according to claim 13 wherein the compressed record of the sensor data comprises thumbnail image locations of one or more coverage regions of a gesture with respect to one or more thumbnail images of parts of the document;

and using the compressed record comprises displaying the coverage regions overlaid on the thumbnail images.

15. A method according to claim 13 wherein:

the compressed record of the sensor data comprises metadata describing the sensor data; and

using the compressed record further includes searching the document on the basis of the metadata.

16. A computing device comprising:

one or more processors arranged to render a document at a display device using output of an application which is executing on the computing device or at a remote computing entity in communication with the computing device, the application configured to at least one of read or write documents;

one or more sensors monitoring user annotation events associated with the rendered document;

the one or more processors arranged to monitor state data of the application during monitoring of the annotation events by the sensors;

a gesture recognition component arranged to recognize gestures by analyzing the received sensor data; and

a memory storing the state data with associated timestamps, the memory storing the recognized gestures with associated timestamps;

the one or more processors arranged to calculate a compressed record of the sensor data on the basis of the recognized gestures, the state data, and the timestamps;

the one or more processors arranged to display representations of the recognized gestures superimposed on representations of portions of the rendered document associated with the recognized gestures, wherein individual ones of the representations of the recognized gestures are distinguishable from each other.

17. A computing device according to claim 16 the sensors comprising one or more of:

a camera, a touch sensor, or a microphone.

18. A computing device according to claim 16 the gesture recognition component being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, or a graphics processing unit.

19. A computing device according to claim 16, wherein the recognized gestures are associated with the portions of the rendered document by commands associated with the recognized gestures that operated on the portions of the rendered document.