UPDATING PRINTED CONTENT WITH PERSONALIZED VIRTUAL DATA

• US 20130147838A1
• Filed: 01/09/2012
• Published: 06/13/2013
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Granted Patent: US 9,229,231 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/228 of application context

Abstract

The technology provides for updating printed content with personalized virtual data using a see-through, near-eye, mixed reality display device system. A printed content item, for example a book or magazine, is identified from image data captured by cameras on the display device, and user selection of a printed content selection within the printed content item is identified based on physical action user input, for example eye gaze or a gesture. Virtual data is selected from available virtual data for the printed content selection based on user profile data, and the display device system displays the selected virtual data in a position registered to the position of the printed content selection. In some examples, a task related to the printed content item is determined based on physical action user input, and personalized virtual data is displayed registered to the printed content item in accordance with the task.

Claims

1. A method for updating printed content with personalized virtual data using a see-through, near-eye, mixed reality display device comprising:

identifying a printed content item in a field of view of a see-through, near-eye, mixed reality display device;

identifying user selection of a printed content selection within the printed content item based on physical action user input;

determining whether virtual data is available for the printed content selection;

responsive to virtual data being available for the printed content selection, selecting virtual data from the available virtual data based on user profile data; and

displaying the virtual data in a position registered to a position of the printed content selection.

2. The method of claim 1 further comprising determining a current state of being based on eye tracking data and updating the current state of being stored in user profile data;

and

wherein selecting virtual data based on user profile data includes selecting virtual data based on the current state of being.

3. The method of claim 1 wherein,

user profile data includes identity data including a user age;

and

wherein selecting virtual data from the available virtual data based on user profile data includes selecting age appropriate virtual data based on the user age.

4. The method of claim 1 wherein

user profile data includes a current user location;

and

wherein selecting virtual data from the available virtual data based on user profile data includes selecting virtual data based on the current user location.

5. The method of claim 1 further comprising:

accessing data related to the printed content selection and another person identified in user profile data of the user; and

wherein selecting virtual data from the available virtual data based on user profile data includes selecting the data about the other person for the virtual data.

6. The method of claim 1 further comprising:

storing the virtual data with a registration to content embodied in the printed content selection in a layout of a medium independent version of a work embodied in the printed content item.

7. The method of claim 1

wherein selecting virtual data from the available virtual data based on user profile data includes selecting virtual data based on a current state of being of the user.

8. A see-through, near-eye, mixed reality display device system for updating printed content with personalized virtual data comprising:

a see-through display positioned by a support structure;

at least one outward facing camera positioned on the support structure for capturing image data of a field of view of the see-through display;

one or more software controlled processors having access to stored user profile data and being communicatively coupled to a search engine having access to one or more datastores including content, layout and virtual data for works and printed content items embodying the works;

the one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving image data of the field of view and being for identifying user physical action user selecting a printed content selection in a printed content item based on image data of the field of view;

the one or more software controlled processors being for selecting virtual data from the one or more datastores based on user profile data and the printed content selection; and

the one or more software controlled processors causing at least one communicatively coupled image generation unit for the see-through display to display the virtual data.

9. The system of claim 8 wherein the one or more software controlled processors are communicatively coupled to one or more remote computer systems executing an information push service which updates user profile data of the user.

10. The system of claim 8 wherein the one or more software controlled processors being for identifying physical action user selection of a printed content selection in a printed content item based on the image data of the field of view further comprises identifying one or more gestures identifying selection of the printed content selection from the image data of the field of view.

11. The system of claim 8 further comprising one or more eye tracking assemblies positioned by the support structure for capturing image data of each eye and communicatively coupled to the one or more software controlled processors for sending image data of each eye;

wherein identifying physical action user selection of a printed content selection in a printed content item based on image data of the field of view further comprises

the one or more software controlled processors identifying from the image data of each eye a physical action of gaze duration;

the one or more software controlled processors identifying from the image data of the field a view that a point of gaze is a printed content object in the field of view of the see-through displays; and

responsive to the gaze duration exceeding a threshold, selecting the printed content object as a printed content selection.

12. The system of claim 8 further comprising:

a memory for storing data and software including software for converting text in image data to text in a computer standardized text data storage format; and

wherein the one or more software controlled processors being for identifying a printed content item including the printed content selection and a work including a medium independent version of the printed content selection based on formulating one or more queries based on the image data and sending the one or more queries to the search engine further comprises

the one or more software controlled processors identifying one or more version identifying sections of the printed content item in image data,

the one or more software controlled processors formulating and sending a query based on the one or more version identifying sections to the communicatively coupled search engine, and

the one or more software controlled processors receiving a printed content item version identifier.

13. The system of claim 12 further comprising:

the one or more software controlled processors requesting and receiving from the datastores based on the printed content item version identifier, position data for the printed content selection within a layout of the printed content item, a medium independent work identifier, work position data for the medium independent version of the printed content selection in terms of a medium independent subdivision, and any applicable medium independent work version identifier and work version position data in terms of the medium independent subdivision.

14. The system of claim 13 further comprising:

the one or more software controlled processors searches for virtual data candidates related to the printed content selection based on the work identifier, the work version identifier, and the printed content item version identifier; and

the one or more software controlled processors selects one of the virtual data candidates as virtual data for display based on user profile data.

15. The system of claim 14 further comprising:

the one or more software controlled processors identifying a task to be performed with respect to the printed content selection based on identifying one or more gestures from the image data of the field of view; and

displaying the virtual data selected for display based on user profile data related to the content selection in accordance with the task.

16. One or more processor readable storage devices having instructions encoded thereon for causing one or more processors to execute a method for updating printed content with personalized virtual data using a see-through, near-eye, mixed reality display device, the method comprising:

identifying a printed content item in a field of view of a see-through, near-eye, mixed reality display device;

determining a task related to the printed content item based on physical action user input;

performing the task; and

displaying personalized virtual data registered to the printed content item in accordance with the task.

17. The one or more processor readable storage devices of claim 16 wherein performing the task further comprises performing a learning assistance task by

monitoring audio data captured by a microphone of the see-through, near-eye mixed reality device for one or more words of the printed content item;

identifying by speech recognition software a question by the user with respect to the one or more words of the printed content item;

formulating a query based on the question;

searching one or more datastores including virtual data for the printed content item based on the query; and

responsive to receiving virtual data based on the query, displaying the virtual data in a position registered to a position of the one or more words in the printed content item.

18. The one or more processor readable storage devices of claim 16

wherein the virtual data is formatted on one or more virtual pages appearing to be inserted as one or more adjacent pages to a page including the one or more words.

19. The one or more processor readable storage devices of claim 16 wherein performing the task further comprises identifying a word for which a user has previously requested a definition in a section of the printed content item in the image data of the field of view;

and

displaying the definition for the word in a position registered to the word in the section.

20. The one or more processor readable storage devices of claim 16 wherein performing the task further comprises identifying a word a user has previously requested a definition for in a section of the printed content item in the image data of the field of view;

and

displaying a synonym for the word in a position registered to the word in the section.