MAKING STATIC PRINTED CONTENT DYNAMIC WITH VIRTUAL DATA

• US 20130147836A1
• Filed: 12/07/2011
• Published: 06/13/2013
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Granted Patent: US 9,182,815 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

The technology provides embodiments for making static printed content being viewed through a see-through, mixed reality display device system more dynamic with display of virtual data. A printed content item, for example a book or magazine, is identified from image data captured by cameras on the display device, and user selection of a printed content selection within the printed content item is identified based on physical action user input, for example eye gaze or a gesture. A task in relation to the printed content selection can also be determined based on physical action user input. Virtual data for the printed content selection is displayed in accordance with the task. Additionally, virtual data can be linked to a work embodied in a printed content item. Furthermore, a virtual version of the printed material may be displayed at a more comfortable reading position and with improved visibility of the content.

Claims

1. A method for making static printed content dynamic with virtual data using a see-through, near eye, mixed reality display device comprising:

identifying a printed content item in a field of view of a see-through, near eye, mixed reality display device;

identifying user selection of a printed content selection within the printed content item based on physical action user input;

determining a task for the printed content selection based on physical action user input;

performing the task; and

displaying virtual data related to the printed content selection in accordance with the task.

2. The method of claim 1 wherein the task is bringing the printed content selection up-to-date which further comprises:

identifying the printed content selection as being within an out-of-date section within the printed content item;

retrieving virtual data with up-to-date information for the section including the printed content selection; and

displaying the virtual data with up-to-date information in a position related to the position of the printed content selection.

3. The method of claim 1 wherein performing the task further comprises performing a search task by

identifying a marker of the printed content selection within a subdivision of the printed content item from a plurality of markers for the subdivision based on physical action user input data;

selecting a search task for the printed content selection based on physical action user input data;

requesting and receiving at least one keyword associated with the marker from one or more printed content related datastores;

generating a search query based on the at least one keyword associated with the marker;

sending the search query to a search engine; and

displaying search results in the field of view of the see-through, near-eye, mixed reality display device.

4. The method of claim 1 wherein the task is an annotation task and performing the task further comprises:

displaying a virtual key entry input device for the user;

generating annotation data based on user data entry with the virtual key entry input device; and

displaying the annotation data at a position indicated by physical action user input.

5. The method of claim 1 wherein the task is an annotation task and performing the requested task further comprises:

the printed content selection is a first printed content selection;

receiving user input indicating a physical action of the user requesting virtual data of the first printed content selection be made and designated as an annotation;

identifying user selection of a second printed content selection as a selection to be annotated;

receiving user input indicating a physical action of the user indicating a position for displaying the annotation made of the first printed content selection in relation to the second printed content selection; and

displaying the annotation at the indicated position.

6. The method of claim 5 wherein the first printed content selection is handwritten text and the annotation made of the first printed content selection is in a computer standardized text data format.

7. The method of claim 1 wherein the task is a group read task and performing the group read task further comprises:

receiving user input identifying a user associated with the see through, mixed reality display device system as either a reader or a participant;

responsive to the user input identifying the user as a reader,

displaying a virtual version of the printed content item, and

responding to physical action user input with respect to either the virtual version of the printed content item or the printed content item if the printed content item is within the field of view of cameras of the display device system; and

responsive to the user input identifying the user as a participant, displaying virtual data in relation to the printed content item in the field of view responsive to one or more notifications from another see-through, near eye, mixed reality display device system of the reader.

8. The method of claim 1 wherein the task is for providing a virtual version of printed content which is linked to the user field of view which further comprises:

displaying image data including content of the printed content selection at a position in the user field of view independent of a position of the printed content selection; and

updating display of the image data responsive to physical action input of the user.

9. The method of claim 8 further comprising:

determining whether an electronic version of the printed content item is available;

responsive to the electronic version of the printed content item being available, displaying image data of the electronic version;

responsive to the electronic version of the printed content item not being available,

storing image data of the printed content selection,

outputting instructions to the user via the display device system to look at one or more sections of the printed content item including content to be available in a current reading session;

capturing image data of each of the one or more sections while the respective section is in the field of view of outward facing cameras of the display device system; and

storing the captured image data.

10. The method of claim 8 further comprising:

updating display of the image data responsive to layout change requests of the user.

11. A see-through, near-eye, mixed reality display device system for making static printed material dynamic comprising:

a respective see-through display for each eye positioned by a support structure;

at least one outward facing camera positioned on the support structure for capturing image data in a field of view of the respective see-through displays;

one or more software controlled processors being communicatively coupled to a search engine having access to datastores including content, layout and virtual data for works and printed content items embodying the works;

the one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving image data;

the one or more software controlled processors being for identifying user selection of a printed content selection based on physical action user input and image data;

the one or more software controlled processors being for identifying a printed content item including the printed content selection and a work including a medium independent version of the printed content selection based on formulating one or more queries based on the image data and sending the one or more queries to the search engine; and

the one or more software controlled processors causing at least one communicatively coupled image generation unit to display virtual data associated with the printed content selection or the medium independent version of the printed content selection by each respective see-through display optically coupled to the at least one image generation unit.

12. The system of claim 11 further comprising one or more eye tracking

assemblies positioned by the support structure for capturing image data of each eye and communicatively coupled to the one or more software controlled processors for sending image data of each eye;

the one or more software controlled processors identifying from the image data of each eye a physical action of gaze duration on a printed content object in the field of view of the see-through displays; and

responsive to the gaze duration exceeding a threshold, selecting the printed content object as a printed content selection.

13. The system of claim 11 further comprising:

a memory for storing data and software including software for converting text in image data to text in a computer standardized text data storage format; and

wherein the one or more software controlled processors being for identifying a printed content item including the printed content selection and a work including a medium independent version of the printed content selection based on formulating one or more queries based on the image data and sending the one or more queries to the search engine further comprises

the one or more software controlled processors identifying one or more version identifying sections of the printed content item in image data,

the one or more software controlled processors formulating and sending a query based on the one or more version identifying sections to the communicatively coupled search engine, and

the one or more software controlled processors receiving a printed content item version identifier.

14. The system of claim 13 further comprising:

the one or more software controlled processors requesting and receiving from the datastores based on the printed content item version identifier, position data for the printed content selection within a layout of the printed content item, a medium independent work identifier, work position data for the medium independent version of the printed content selection in terms of a medium independent subdivision, and any applicable medium independent work version identifier and work version position data in terms of the medium independent subdivision.

15. The system of claim 14 further comprising:

the one or more software controlled processors searches for virtual data related to the printed content selection based on the work identifier, the work version identifier and the printed content item version identifier.

16. The system of claim 14 further comprising:

the one or more software controlled processors receiving user input linking user generated virtual annotation data to the printed content selection in the printed content item; and

the one or more software controlled processors link user generated virtual annotation data to the printed content selection in the printed content item and to the medium independent version of the printed content selection in the work.

17. One or more processor readable storage devices having instructions encoded thereon for causing one or more processors to execute a method for improving readability of static printed material with virtual data using a see-through, near eye, mixed reality display device system, the method comprising:

identifying printed material in a field of view of the see-through, near eye, mixed reality display device system based on image data captured by one or more outward facing cameras of the display device system;

determining whether readability criteria is satisfied for the printed material positioned in the field of view; and

responsive to readability criteria not being satisfied,

displaying a virtual version of the printed material in the field of view which satisfies readability criteria; and

responding to physical action user input with respect to either or both of the virtual version of the printed material or the printed material if the printed material is still within a field of view of the one or more outward facing cameras.

18. The one or more processor readable storage devices of claim 17

wherein the readability criteria includes comfortable reading position criteria and wherein determining whether readability criteria is satisfied for the printed material positioned in the field of view further comprises:

identifying a position of the printed material in the field of view based on the image data captured by the one or more outward facing cameras;

determining whether the position of the printed material in the field of view satisfies comfortable reading position criteria; and

responsive to the position of the printed material not satisfying comfortable reading position criteria,

determining a new position for satisfying comfortable reading position criteria,

generating image data of a virtual version of the printed material, and

displaying the virtual version of the printed material at the new position satisfying comfortable reading position criteria by the see-through, near eye, mixed reality display device system.

19. The one or more processor readable storage devices of claim 18 wherein the comfortable reading position criteria includes at least one of the following:

an angular position criteria from a respective reference point of a see through display for each eye included in the see-through, mixed reality display device system;

a depth distance from the respective reference point of the see through display for each eye; and

an orientation of content data data in the field of view of the display device.

20. The one or more processor readable storage devices of claim 17 wherein the readability criteria includes visibility criteria and wherein determining whether readability criteria is satisfied for the printed material positioned in the field of view further comprises:

identifying a size of content data of the printed material in the field of view of the see through, near eye, mixed reality display device system;

determining whether visibility criteria is satisfied based on the size of the content data; and

responsive to visibility criteria not being satisfied, generating a virtual version of the printed material with a changed size of the content data which satisfies visibility criteria, and displaying the virtual version of the printed material with the changed size.