DISPLAYING VIRTUAL DATA AS PRINTED CONTENT

• US 20130147687A1
• Filed: 01/10/2012
• Published: 06/13/2013
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Granted Patent: US 9,183,807 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/14 Electronic books and readers
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The technology provides embodiments for displaying virtual data as printed content by a see-through, near-eye, mixed reality display device system. One or more literary content items registered to a reading object in a field of view of the display device system are displayed with print layout characteristics. Print layout characteristics from a publisher of each literary content item are selected if available. The reading object has a type like a magazine, book, journal or newspaper and may be a real object or a virtual object displayed by the display device system. The reading object type of the virtual object is based on a reading object type associated with a literary content item to be displayed. Virtual augmentation data registered to a literary content item is displayed responsive to detecting user physical action in image data. An example of a physical action is a page flipping gesture.

Claims

1. A method for displaying virtual data as printed content using a see-through, near eye, mixed reality display device system comprising:

receiving a request to display one or more literary content items registered to a reading object in a field of view of the see-through, near-eye, mixed reality display device system;

selecting print layout characteristics for each of the one or more literary content items; and

displaying the one or more literary content items with their respective print layout characteristics registered to the reading object in the field of view.

2. The method of claim 1 further comprising:

determining whether physical action has been detected indicating user selection of a real reading object;

responsive to detecting physical action indicating user selection of a real reading object, selecting the real reading object as the reading object; and

responsive to not detecting physical action indicating user selection of a real reading object, automatically selecting a one or more reading object types for corresponding one or more virtual reading objects based on a reading object type associated with each requested literary content item.

3. The method of claim 1 wherein the reading object comprises at least one of the following:

one or more pages of real paper in the field of view;

a real book;

a real newspaper; and

a real magazine.

4. The method of claim 1 wherein the reading object comprises at least one of the following:

a virtual book;

a virtual newspaper; and

a virtual magazine.

5. The method of claim 1 wherein selecting print layout characteristics for each of the one or more literary content items further comprises

identifying a medium independent work identifier and any applicable medium independent work version identifier for each literary content item;

determining for each literary content item whether one or more publisher rule sets of print layout characteristics are available; and

responsive to one or more publisher rule sets of print layout characteristics not being available for the respective literary content item, assigning print layout characteristics for the respective literary content item.

7. The method of claim 1 wherein displaying the one or more literary content items with their respective print layout characteristics registered to the reading object in the field of view further comprises:

generating one or more page layouts including the one or more literary content items with their print layout characteristics in accordance with stored layout rules;

displaying one or more pages based on the one or more page layouts in the field of view of the see-through, mixed reality display device system; and

changing the one or more pages displayed responsive to user physical action.

8. The method of claim 7 wherein generating one or more page layouts including the one or more literary content items with their print layout characteristics in accordance with stored layout rules further comprises:

assigning each of the requested literary content items to one or more sets based on a reading object type associated with each literary content item;

determining an item order of display of the literary content items based on user preferences;

determining a set order of display of the one or more sets based on a priority indicated in the determined item order; and

generating a page layout set of one or more page layouts for each of the literary content items.

9. The method of claim 8 wherein generating a page layout set of one or more page layouts for each of the literary content items further comprises:

determining an intra-set order of display of the literary content items within the set based on user preferences;

selecting a block template for each page based on the reading object type for the set and the print layout characteristics of the items in the set;

assigning each literary content item in the set to a starting page number within the set based on priority in the intra-set order, size characteristics of the respective item and page dimensions for the reading object;

for each page, assigning each item assigned to the page to a block on the page based on the size characteristics of the respective literary content item; and

responsive to any of the blocks not satisfying visibility criteria, performing adjustment of one or more adjustable print layout characteristics of the literary content items for satisfying visibility criteria.

6. The method of claim 6 further comprising:

responsive to one or more publisher rule sets of print layout characteristics being available for the respective literary content item,

retrieving the available one or more respective publisher print layout characteristics rule sets over a network from one or more datastores storing publisher print layout characteristics rule sets;

determining whether more than one publisher rule set for print layout characteristics is available for the respective literary content item;

responsive to more than one publisher rule set for print layout characteristics being available, selecting one of the retrieved publisher rule sets for print layout characteristics based on page size dimensions of the reading object, visibility criteria and any user layout preferences; and

identifying a printed content item version identifier for the respective literary content item based on the selected publisher print layout characteristics rule set.

10. A see-through, near-eye, mixed reality display device system for displaying virtual data as printed content comprising:

a see-through display positioned by a support structure;

at least one outward facing camera positioned on the support structure for capturing image data of a field of view of the see-through display;

one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving the image data of the field of view;

the one or more software controlled processors having access to one or more datastores including content, print layout characteristics and virtual augmentation data for literary content items and being for selecting print layout characteristics for each of the one or more literary content items from the one or more datastores; and

the one or more software controlled processors causing at least one communicatively coupled image generation unit, which is optically coupled to the see-through display, to display the one or more literary content items with their respective selected print layout characteristics registered to a reading object in the field of view.

11. The system of claim 10 further comprising:

the one or more software controlled processors having access to layout rules for generating one or more virtual pages including the one or more literary content items for display in the field of view registered to the reading object in accordance with visibility criteria.

12. The system of claim 11 further comprising:

one or more eye tracking assemblies positioned by the support structure for capturing image data of each eye and communicatively coupled to the one or more software controlled processors for sending image data of each eye;

the one or more software controlled processors identifying from the image data of each eye one or more physical actions of the eye including gaze on a content object identified in the field of view image data; and

the one or more software controlled processors selecting the content object as a user content selection responsive to the one or more physical actions of the eye.

13. The system of claim 11 further comprising:

the one or more software controlled processors forming a data link between the user generated augmentation data and a medium independent version of the literary content item stored in the accessible one or more datastores; and

the one or more software controlled processors retrieving the user generated augmentation data from an accessible memory based on the data link for display in registration to a printed content item version of the literary content item in the field of view.

14. The system of claim 11 further comprising:

the one or more software controlled processors identifying a user content selection based on identifying one or more gestures from the image data of the field of view.

15. One or more processor readable storage devices having instructions encoded thereon for causing one or more processors to execute a method for displaying virtual data as printed content using a see-through, near-eye, mixed reality display device system, the method comprising:

receiving a request to display one or more literary content items registered to a reading object in a field of view of the see-through, mixed reality display device system;

selecting print layout characteristics for each of the one or more literary content items;

displaying the one or more literary content items with their respective print layout characteristics registered to the reading object in the field of view; and

displaying virtual augmentation data registered to at least one of the one or more literary content items responsive to physical action user input.

16. The one or more processor readable storage devices of claim 15 wherein displaying virtual augmentation data registered to at least one of the one or more literary content items responsive to physical action user input further comprises:

identifying a user content selection within a literary content item based on physical action user input;

determining whether augmentation data is available for the user content selection;

responsive to augmentation data being available for the user content selection,

selecting the augmentation data from the available augmentation data based on user profile data; and

displaying the augmentation data in a position registered to the position of the user content selection.

17. The one or more processor readable storage devices of claim 16 wherein displaying virtual augmentation data registered to at least one of the one or more literary content items responsive to physical action user input further comprises:

determining a task related to the literary content item based on physical action user input;

performing the task; and

displaying augmentation data related to the user content selection in accordance with the task.

18. The one or more processor readable storage devices of claim 17 wherein the reading object is a virtual book;

and

wherein the task is a page flipping task and performing the task further comprises flipping at least one page of a virtual book responsive to identifying from image data one or more thumb actions of a flipping gesture.

19. The one or more processor readable storage devices of claim 18 wherein performing the task further comprises displaying an indicator registered to the at least one page image indicating virtual augmentation data is available for display registered to the at least one page image.

20. The one or more processor readable storage devices of claim 19 wherein the identifier in the margin is a thumbnail image.