Making static printed content dynamic with virtual data

• US 9,182,815 B2
• Filed: 12/07/2011
• Issued: 11/10/2015
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US13/313,368
• Publication Number:
  US 20130147836A1
• Time in Patent Office: 1,434 Days
• Field of Search: 345 7- 8, 345632-633, 715/757, 715/863
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

The technology provides embodiments for making static printed content being viewed through a see-through, mixed reality display device system more dynamic with display of virtual data. A printed content item, for example a book or magazine, is identified from image data captured by cameras on the display device, and user selection of a printed content selection within the printed content item is identified based on physical action user input, for example eye gaze or a gesture. A task in relation to the printed content selection can also be determined based on physical action user input. Virtual data for the printed content selection is displayed in accordance with the task. Additionally, virtual data can be linked to a work embodied in a printed content item. Furthermore, a virtual version of the printed material may be displayed at a more comfortable reading position and with improved visibility of the content.

Claims

1. A method for making static printed content dynamic with virtual data using a near eye, mixed reality display device comprising:

identifying a real printed content item in a field of view of a near eye, mixed reality display device;

identifying user selection of a real printed content selection within the real printed content item based on physical action user input;

determining a task requested for the real printed content selection based on physical action user input;

performing the task; and

displaying virtual data related to the real printed content selection in accordance with the task wherein the task further comprises displaying a virtual version of the real printed content selection at a first position in the field of view of the near-eye, mixed reality display device which is independent of a second position of the real printed content item with respect to the field of view of the display device, and updating display of the virtual version of the real printed content selection responsive to physical action input of the user.

2. The method of claim 1 wherein the task is bringing the printed content selection up-to-date which further comprises:

identifying the printed content selection as being within an out-of-date section within the printed content item;

retrieving virtual data with up-to-date information for the section including the printed content selection; and

displaying the virtual data with up-to-date information in a position related to the position of the printed content selection.

3. The method of claim 1 wherein the task is an annotation task and performing the task further comprises:

displaying a virtual key entry input device for the user;

generating annotation data based on user data entry with the virtual key entry input device; and

displaying the annotation data at a position indicated by physical action user input.

4. The method of claim 1 wherein the task is an annotation task and performing the requested task further comprises:

the printed content selection is a first printed content selection;

receiving user input indicating a physical action of the user requesting virtual data of the first printed content selection be made and designated as an annotation;

identifying user selection of a second printed content selection as a selection to be annotated;

receiving user input indicating a physical action of the user indicating a position for displaying the annotation made of the first printed content selection in relation to the second printed content selection; and

displaying the annotation at the indicated position.

5. The method of claim 4 wherein the first printed content selection is handwritten text and the annotation made of the first printed content selection is in a computer standardized text data format.

6. The method of claim 1 wherein the task is a group read task and performing the group read task further comprises:

receiving user input identifying a user associated with the mixed reality display device system as either a reader or a participant;

responsive to the user input identifying the user as a reader,

displaying a virtual version of the printed content item, and

responding to physical action user input with respect to either the virtual version of the real printed content item or the real printed content item if the real printed content item is within the field of view of at least one camera of the display device; and

responsive to the user input identifying the user as a participant, displaying virtual data in relation to the real printed content item in the field of view responsive to one or more notifications from another near eye, mixed reality display device of the reader.

7. The method of claim 1 wherein displaying a virtual version of the real printed content selection at a first position in the field of view of the near-eye, mixed reality display device which is independent of a second position of the real printed content selection with respect to the field of view of the display device further comprises:

determining whether an electronic version of the printed content item is available;

responsive to the electronic version of the printed content item being available, loading at least a section of the printed content item including the printed content selection into a memory accessible by a processor of the near-eye, mixed reality display device and displaying image data of the electronic version the printed content selection;

responsive to the electronic version of the printed content item not being available,

storing image data of the printed content selection currently in the user field of view,

outputting instructions to the user via the display device to look at one or more sections of the printed content item including content to be available in a current reading session;

capturing image data of each of the one or more sections while the respective section is in the field of view of outward facing cameras of the display device system; and

storing the captured image data.

8. The method of claim 1 further comprising:

updating display of the image data responsive to layout change requests of the user.

9. The method of claim 1 wherein the near-eye, mixed reality display device is a see-through, near-eye, mixed reality display device.

10. One or more processor readable non-volatile storage devices having instructions encoded thereon for causing one or more processors to execute a method for making static printed content dynamic with virtual data using a near eye, mixed reality display device, the method comprising:

identifying a printed content item in a field of view of a near eye, mixed reality display device;

identifying user selection of a printed content selection within the printed content item based on physical action user input;

determining a task for the printed content selection based on physical action user input;

performing the task; and

displaying virtual data related to the printed content selection in accordance with the task wherein the task is a group read task and performing the group read task further comprises;

receiving user input identifying a user associated with the mixed reality display device system as either a reader or a participant;

responsive to the user input identifying the user as a reader,

displaying a virtual version of the printed content item, and

responding to physical action user input with respect to either the virtual version of the printed content item or the printed content item if the printed content item is within the field of view of cameras of the display device system, and

responsive to the user input identifying the user as a participant,

displaying virtual data in relation to the printed content item in the field of view responsive to one or more notifications from another near eye, mixed reality display device of the reader.

11. The one or more processor readable non-volatile storage devices of claim 10 wherein the near-eye, mixed reality display device is a see-through, near-eye, mixed reality display device.

12. A near-eye, mixed reality display device system for making static printed material dynamic comprising:

a respective display for each eye positioned by a support structure;

at least one outward facing camera positioned on the support structure for capturing image data in a field of view of the respective displays;

one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving image data;

the one or more software controlled processors being operable for identifying a real printed content item in the field of view of the respective displays;

the one or more software controlled processors being communicatively coupled to one or more natural user input capture devices for receiving data for identifying physical action user input;

the one or more software controlled processors being operable for identifying physical action user input indicating user selection of a real printed content selection within the real printed content item based on the data received from the one or more natural user input capture devices;

the one or more software controlled processors being operable for identifying physical action user input indicating a task is requested for the real printed content selection;

an image generation unit communicatively coupled to the one or more software processors and being operable for control by the one or more software processors;

the one or more software controlled processors controlling performance of the task; and

the image generation under control of the one or more software controlled processors being operable for displaying virtual data related to the real printed content selection in accordance with the task wherein the task further comprises the image generation under control of the one or more software controlled processors being operable for displaying a virtual version of the real printed content selection at a first position in the field of view of the near-eye, mixed reality display device which is independent of a second position of the real printed content item with respect to the field of view of the display device, and updating display of the virtual version of the real printed content selection responsive to physical action input of the user.

13. The system of claim 12 wherein the respective display for each eye positioned by a support structure is a see-through display.