Updating printed content with personalized virtual data

• US 9,229,231 B2
• Filed: 01/09/2012
• Issued: 01/05/2016
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Primary Examiner(s): Brier, Jeffery A
• Assistant Examiner(s): BUTTRAM, ALAN T
• Application Number:
  US13/346,674
• Publication Number:
  US 20130147838A1
• Time in Patent Office: 1,457 Days
• Field of Search: 345/7, 345/8, 345/632, 345/633, 715/757, 715/863
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/228 of application context

Abstract

The technology provides for updating printed content with personalized virtual data using a see-through, near-eye, mixed reality display device system. A printed content item, for example a book or magazine, is identified from image data captured by cameras on the display device, and user selection of a printed content selection within the printed content item is identified based on physical action user input, for example eye gaze or a gesture. Virtual data is selected from available virtual data for the printed content selection based on user profile data, and the display device system displays the selected virtual data in a position registered to the position of the printed content selection. In some examples, a task related to the printed content item is determined based on physical action user input, and personalized virtual data is displayed registered to the printed content item in accordance with the task.

Claims

1. A machine-implemented method of augmenting real printed content with personalized virtual data that is personalized for a user wearing a near-eye, mixed reality display device, the method comprising:

retrieving identity data for a user wearing a near-eye, mixed reality display device;

identifying a printed content item viewable through and in a field of view of the near-eye, mixed reality display device, the identified item being real and providing real printed content;

identifying user selection of a printed content portion within the real printed content of the printed content item based on physical action user input;

determining whether personalizable virtual data is available for the user selected printed content portion;

responsive to the personalizable virtual data being determined to be available for the user selected printed content portion, selecting personalized virtual data from the determined to be available virtual data based on user profile data corresponding to the retrieved identity data for the user wearing the near-eye, mixed reality display device, wherein the selecting of the personalized virtual data from the available personalizable data includes selecting the personalized virtual data based on a current state of being of the user, the current state of being of the user including an emotional state of the user; and

causing display of the selected, personalized virtual data in the field of view of the near-eye, mixed reality display device and in a position registered to a position of the user selected printed content portion of the automatically identified printed content item.

2. The method of claim 1 further comprising determining a current state of being of the user based on eye tracking data and updating the current state of being stored in user profile data.

3. The method of claim 1 wherein,

the retrieved identity data includes including a user age;

and

wherein the selecting of personalized virtual data from the available personalizable virtual data based on user profile data includes selecting age appropriate virtual data based on the user age.

4. The method of claim 1 wherein,

the retrieved identity data includes a current user location;

and

wherein the selecting of personalized virtual data from the available personalizable virtual data based on user profile data includes selecting virtual data based on the current user location.

5. The method of claim 1 further comprising:

accessing data related to the printed content selection and another person identified in the user profile data of the user; and

wherein the selecting of personalized virtual data from the available personalizable virtual data based on user profile data includes selecting the data about the other person for the virtual data.

6. The method of claim 1 further comprising:

storing the virtual data with a registration to content embodied in the printed content selection in a layout of a medium independent version of a work embodied in the printed content item, the layout of the medium independent version of the work is independent of printed media.

7. A near-eye, mixed reality display device system configured to automatically augment real printed content with personalized virtual data, the system comprising:

a display positioned by a support structure;

at least one outward facing camera positioned on the support structure for capturing image data of a field of view of real objects disposed beyond the display;

one or more software controlled processors having access to user profile data of a user of the display, the user profile data being stored in a memory and being communicatively coupled to a search engine having access to one or more datastores including content, layout and virtual data for works and printed content items embodying the works;

the one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving image data of the field of view of the camera and being configured for automatically identifying user physical action indicative of the user selecting a printed content portion in a printed content item among the real objects within the field of view of the camera, the identified action being directed to a real printed content providing item that provides real printed content;

the one or more software controlled processors being configured for determining a current state of being of the user, the current state of being of the user including an emotional state of the user, and storing the current state of being in the user profile data of the user;

the one or more software controlled processors being configured for automatically selecting user-personalized virtual data from the one or more datastores based on the user profile data of the user including based on the current state of being and based on the user selected printed content portion of the real printed content providing item; and

the one or more software controlled processors being configured for causing at least one communicatively coupled image generation unit for the display to display the automatically selected virtual data as being automatically registered with a field of view position of the printed content portion.

8. The system of claim 7 wherein the one or more software controlled processors being for identifying physical action user selection of a printed content selection in a printed content item based on the image data of the field of view further comprises identifying one or more gestures identifying selection of the printed content selection from the image data of the field of view.

9. The system of claim 7 further comprising one or more eye tracking assemblies positioned by the support structure for capturing image data of each eye and communicatively coupled to the one or more software controlled processors for sending image data of each eye;

wherein identifying physical action user selection of a printed content selection in a printed content item based on image data of the field of view further comprises

the one or more software controlled processors identifying from the image data of each eye a physical action of gaze duration;

the one or more software controlled processors identifying from the image data of the field a view that a point of gaze is a printed content object in the field of view of the displays; and

responsive to the gaze duration exceeding a threshold, selecting the printed content object as a printed content selection.

10. The system of claim 7 further comprising:

a memory for storing data and software including software for converting text in image data to text in a computer standardized text data storage format; and

wherein the one or more software controlled processors being for identifying a printed content item including the printed content selection and a work including a medium independent version of the printed content selection based on formulating one or more queries based on the image data and sending the one or more queries to the search engine further comprises

the one or more software controlled processors identifying one or more version identifying sections of the printed content item in image data,

the one or more software controlled processors formulating and sending a query based on the one or more version identifying sections to the communicatively coupled search engine, and

the one or more software controlled processors receiving a printed content item version identifier.

11. The system of claim 10 further comprising:

the one or more software controlled processors requesting and receiving from the datastores based on the printed content item version identifier, position data for the printed content selection within a layout of the printed content item, a medium independent work identifier, work position data for the medium independent version of the printed content selection in terms of a medium independent subdivision, and any applicable medium independent work version identifier and work version position data in terms of the medium independent subdivision.

12. The system of claim 11 further comprising:

the one or more software controlled processors searches for virtual data candidates related to the printed content selection based on the work identifier, the work version identifier, and the printed content item version identifier; and

the one or more software controlled processors selects one of the virtual data candidates as virtual data for display based on user profile data.

13. The system of claim 12 further comprising:

the one or more software controlled processors identifying a task to be performed with respect to the printed content selection based on identifying one or more gestures from the image data of the field of view; and

displaying the virtual data selected for display based on user profile data related to the content selection in accordance with the task.

14. One or more processor readable non-volatile storage devices having instructions encoded thereon for causing one or more processors to execute a machine-implemented method for augmenting real printed content with user personalized virtual data for a user wearing and using a near-eye, mixed reality display device, the method comprising:

identifying a real printed content item in a field of view of the near-eye, mixed reality display device, the identified item being real and providing real printed content;

determining a task related to the identified printed content item based on detected physical action of the user directed to a user selected portion of the real printed content; and

performing the task which further includes;

determining a current state of being of the user wearing the near-eye, mixed reality display device, the current state of being of the user including an emotional state of the user, and storing the current state of being of the user,

selecting an output format for personalized virtual data associated with the task based on the current state of being of the user, the output format being from the group consisting of non-literary visual symbols, a language of text, and audio data, and

outputting the personalized virtual data in accordance with the task and placed in registration with the user selected portion of the real printed content.

15. The one or more processor readable non-volatile storage devices of claim 14

wherein performing the task further comprises performing a learning assistance task by

monitoring audio data captured by a microphone of the near-eye mixed reality device for one or more words of the printed content item;

identifying by speech recognition software a question by the user with respect to the one or more words of the printed content item;

formulating a query based on the question;

searching one or more datastores including virtual data for the printed content item based on the query; and

responsive to receiving virtual data based on the query, displaying the virtual data in a position registered to a position of the one or more words in the printed content item.

16. The one or more processor readable non-volatile storage devices of claim 14

wherein the virtual data is formatted on one or more virtual pages appearing to be inserted as one or more adjacent pages to a page including the one or more words.

17. The one or more processor readable non-volatile storage devices of claim 14

wherein performing the task further comprises identifying a word for which a user has previously requested a definition in a section of the printed content item in the image data of the field of view;

and

displaying the definition for the word in a position registered to the word in the section.

18. The one or more processor readable non-volatile storage devices of claim 14

wherein performing the task further comprises identifying a word a user has previously requested a definition for in a section of the printed content item in the image data of the field of view;

and

displaying a synonym for the word in a position registered to the word in the section.