Displaying virtual data as printed content

• US 9,183,807 B2
• Filed: 01/10/2012
• Issued: 11/10/2015
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Small, Sheridan Martin, Kipman, Alex Aben-Athar, Vaught, Benjamin I., Perez, Kathryn Stone
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US13/347,576
• Publication Number:
  US 20130147687A1
• Time in Patent Office: 1,400 Days
• Field of Search: 345632-633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/14 Electronic books and readers
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The technology provides embodiments for displaying virtual data as printed content by a see-through, near-eye, mixed reality display device system. One or more literary content items registered to a reading object in a field of view of the display device system are displayed with print layout characteristics. Print layout characteristics from a publisher of each literary content item are selected if available. The reading object has a type like a magazine, book, journal or newspaper and may be a real object or a virtual object displayed by the display device system. The reading object type of the virtual object is based on a reading object type associated with a literary content item to be displayed. Virtual augmentation data registered to a literary content item is displayed responsive to detecting user physical action in image data. An example of a physical action is a page flipping gesture.

Claims

1. A method for displaying virtual data having an appearance of printed content using a near eye, mixed reality display device system comprising:

receiving a request by the near eye, mixed reality display device system to display one or more literary content items having the appearance of printed content on a reading object in a field of view of a near-eye display of the near-eye, mixed reality display device system;

determining by a processor of the near-eye, mixed reality display device system whether user input has been detected by a natural user interface of the near-eye, mixed reality display device system indicating user selection of a real reading object in the field of view as the reading object;

responsive to determining the user input has been detected indicating user selection of a real reading object in the field of view, identifying a reading object type of the real reading object and selecting the real reading object as the reading object; and

responsive to determining by the processor user input has not been detected by a natural user interface of the near-eye, mixed reality display device system indicating user selection of a real reading object, automatically identifying by the processor one or more reading object types associated with each of the requested one or more literary content items, selecting a reading object type from the one or more identified reading object types associated with each of the requested one or more literary content items based on stored layout rules, and selecting a virtual reading object having the selected reading object type as the reading object;

automatically selecting by the processor of the near eye, mixed reality display device system print layout characteristics for each of the one or more literary content items based on the reading object type of the reading object; and

displaying in the near-eye display of the near-eye, mixed reality display device system the one or more literary content items with their respective print layout characteristics for having the appearance of printed content registered to the reading object in the field of view.

2. The method of claim 1 wherein the reading object comprises at least one of the following from the group consisting of:

one or more pages of real paper;

a real book;

a real newspaper;

a real magazines;

one or more pages of virtual paper;

a virtual book;

a virtual newspaper; and

a virtual magazine.

3. The method of claim 1 wherein automatically selecting by the processor of the near eye, mixed reality display device system print layout characteristics for each of the one or more literary content items based on the reading object type of the reading object further comprises:

identifying by the processor a medium independent work identifier and any applicable medium independent work version identifier for each literary content item;

determining by the processor for each literary content item whether one or more publisher rule sets of print layout characteristics for the reading object type of the reading object are available; and

responsive to one or more publisher rule sets of print layout characteristics for the reading object type of the reading object not being available for the respective literary content item, assigning by the processor print layout characteristics for the reading object type of the reading object for the respective literary content item.

4. The method of claim 3 further comprising:

responsive to a plurality of publisher rule sets of print layout characteristics for the reading object type of the reading object being available for the respective literary content item, selecting by the processor at least one of the available one or more respective publisher print layout characteristics rule sets over a network from one or more datastores storing publisher print layout characteristics rule sets based on page size dimensions of the reading object, visibility criteria and any user layout preferences.

5. The method of claim 1 wherein displaying in the near-eye display of the near-eye, mixed reality display device system the one or more literary content items with their respective print layout characteristics for having the appearance of printed content registered to the reading object in the field of view further comprises:

generating by the processor one or more page layouts including the one or more literary content items with respective print layout characteristics for the reading object type of the reading object in accordance with stored layout rules;

displaying by the display of the near-eye, mixed reality display system one or more pages based on the one or more page layouts in the field of view of the mixed reality display device system; and

changing by the processor the one or more pages displayed responsive to user physical action manipulating the one or more pages detected by the natural user interface.

6. The method of claim 5 wherein generating by the processor one or more page layouts including the one or more literary content items with respective print layout characteristics for the reading object type of the reading object in accordance with stored layout rules further comprises:

assigning the one or more literary content items to one or more sets of literary content items based on a reading object type associated with each of the one or more literary content item, each set grouping one or more literary content items associated with a same or compatible reading object type;

determining an item order of display of the one or more literary content items based on user preferences;

determining a set order of display of the one or more sets based on a priority indicated in the determined item order; and

generating a page layout set of one or more page layouts for each set of literary content items.

7. The method of claim 6 wherein generating a page layout set of one or more page layouts for each set of literary content items further comprises:

determining by the processor an intra-set order of display of the one or more literary content items within the set based on user preferences;

selecting by the processor a block template for each page based on the reading object type for the set and the print layout characteristics of the items in the set;

assigning each literary content item in the set to a starting page number within the set based on priority in the intra-set order, size characteristics of the respective item and page dimensions for the reading object;

for each page layout, assigning each item assigned to the page layout to a block on the page layout based on the size characteristics of the respective literary content item; and

responsive to any of the blocks not satisfying visibility criteria, performing adjustment of one or more adjustable print layout characteristics of the literary content items for satisfying visibility criteria.

8. The method of claim 1 wherein the selecting a reading object type from the one or more identified reading object types associated with each of the requested one or more literary content items based on stored layout rules further comprises the stored layout rules include a layout rule selecting as the reading object type a reading object type associated with a highest percentage of literary content in the one or more requested literary content items.

9. The method of claim 1 further comprising determining an item order of display of the literary content items based on user preferences.

10. A near-eye, mixed reality display device system for displaying virtual data as printed content comprising:

a near-eye display positioned by a support structure of the near-eye, mixed reality display device system;

at least one outward facing camera positioned on the support structure for capturing image data of a field of view of the near-eye display;

one or more software controlled processors communicatively coupled to the at least one outward facing camera for receiving the image data of the field of view;

the one or more software controlled processors being operable for implementing a natural user interface based on the image data of the field of view;

the one or more software controlled processors being operable for identifying a user request via the natural user interface to display one or more literary content items having the appearance of printed content on at least one reading object in the field of view of the near-eye display;

the one or more software controlled processors having access to one or more datastores storing literary content items and associated print layout characteristics for the stored literary content items, the one or more databases associating one or more reading object types with each of the literary content items and for their associated print layout characteristics;

the one or more software controlled processors being operable for selecting the at least one reading object upon which to display the requested one or more literary content items by

responsive to the one or more software controlled processors determining the user request indicates user selection of a real reading object in the field of view, the one or more software controlled processors identifying a reading object type of the real reading object and selecting the real reading object as the reading object,

responsive to the one or more software controlled processors determining the user request does not indicate user selection of a real reading object, the one or more software controlled processors selecting at least one virtual reading object as the reading object by

the one or more software controlled processors automatically identifying one or more reading object types associated with each of the requested one or more literary content items in the one or more datastores,

the one or more software controlled processors selecting at least one reading object type from the one or more identified reading object types associated with each of the requested one or more literary content items based on stored layout rules, and

the one or more software controlled processors selecting at least one virtual reading object having the at least one selected reading object type as the reading object;

the one or more software controlled processors being operable for automatically selecting print layout characteristics for each of the one or more literary content items based on the at least one reading object type of the at least one reading object; and

the one or more software controlled processors causing at least one communicatively coupled image generation unit, which is optically coupled to the near-eye display, to display the one or more literary content items with their respective selected print layout characteristics for having the appearance of printed content registered to the reading object in the field of view.

11. The system of claim 10 wherein the stored layout rules include a layout rule selecting as the reading object type a reading object type associated with a highest percentage of literary content in the one or more requested literary content items.

12. The system of claim 10 further comprising:

the one or more software controlled processors being operable for generating one or more page layouts including the one or more literary content items with respective print layout characteristics for the reading object type of the reading object in accordance with stored layout rules further comprising

the one or more software controlled processors being operable for assigning the one or more literary content items to one or more sets of literary content items based on a reading object type associated with each of the one or more literary content item, each set grouping one or more literary content items associated with a same or compatible reading object type;

the one or more software controlled processors being operable for determining an item order of display of the one or more literary content items based on user preferences;

the one or more software controlled processors being operable for determining a set order of display of the one or more sets based on a priority indicated in the determined item order; and

the one or more software controlled processors being operable for generating a page layout set of one or more page layouts each set of literary content items.

13. The system of claim 10 further comprising:

the one or more software controlled processors being operable for identifying via the natural user interface a user content selection within a literary content item of the one or more literary content items, the literary content item being displayed with a first layout based on the reading object type of the reading object;

the one or more software controlled processors forming a data link in one or more accessible datastores between user generated augmentation data for the user content selection and a stored medium independent version of the literary content item;

the one or more software controlled processors being operable for subsequently identifying via the natural user interface the user content selection within the literary content item being displayed with a second, different layout based on a different reading object type;

the one or more software controlled processors retrieving the user generated augmentation data from the one or more accessible datastores based on the data link; and

the one or more software controlled processors being operable for displaying the retrieved augmentation data at a position registered to the user content selection of the literary content item in the second, different layout.

14. The system of claim 12

wherein the one or more software controlled processors being operable for generating a page layout set of one or more page layouts for each set of literary content items further comprises:

the one or more software controlled processors being operable for determining by the processor an intra-set order of display of the one or more literary content items within the set based on user preferences;

the one or more software controlled processors being operable for selecting by the processor a block template for each page based on the reading object type for the set and the print layout characteristics of the items in the set;

the one or more software controlled processors being operable for assigning each literary content item in the set to a starting page number within the set based on priority in the intra-set order, size characteristics of the respective item and page dimensions for the reading object;

the one or more software controlled processors being operable for each page layout, assigning each item assigned to the page layout to a block on the page layout based on the size characteristics of the respective literary content item; and

the one or more software controlled processors being operable for responsive to any of the blocks not satisfying visibility criteria, performing adjustment of one or more adjustable print layout characteristics of the literary content items for satisfying visibility criteria.

15. One or more processor readable non-volatile storage devices having instructions encoded thereon for causing one or more processors to execute a method for displaying virtual data having an appearance of printed content using a near-eye, mixed reality display device system, the method comprising:

receiving a request by the near eye, mixed reality display device system to display one or more literary content items having the appearance of printed content on a reading object in a field of view of a near-eye display of the near-eye, mixed reality display device system;

determining by a processor of the near-eye, mixed reality display device system whether user input has been detected by a natural user interface of the near-eye, mixed reality display device system indicating user selection of a real reading object in the field of view as the reading object;

responsive to determining the user input has been detected indicating user selection of a real reading object in the field of view, identifying a reading object type of the real reading object and selecting the real reading object as the reading object; and

responsive to determining by the processor user input has not been detected by a natural user interface of the near-eye, mixed reality display device system indicating user selection of a real reading object, automatically identifying by the processor one or more reading object types associated with each of the requested one or more literary content items, selecting a reading object type from the one or more identified reading object types associated with each of the requested one or more literary content items based on stored layout rules, and selecting a virtual reading object having the selected reading object type as the reading object;

automatically selecting by the processor of the near eye, mixed reality display device system print layout characteristics for each of the one or more literary content items based on the reading object type of the reading object; and

displaying in the near-eye display of the near-eye, mixed reality display device system the one or more literary content items with their respective print layout characteristics for having the appearance of printed content registered to the reading object in the field of view.

16. The one or more processor readable non-volatile storage devices of claim 15 further comprising displaying virtual augmentation data registered to at least one of the one or more literary content items responsive to physical action user input including:

identifying a user content selection within a literary content item based on physical action user input;

determining whether augmentation data is available for the user content selection;

responsive to augmentation data being available for the user content selection,

selecting the augmentation data from the available augmentation data based on user profile data; and

displaying the augmentation data in a position registered to the position of the user content selection.

17. The one or more processor readable non-volatile storage devices of claim 15

wherein the method further comprises determining an item order of display of the literary content items based on user preferences.

18. The one or more processor readable non-volatile storage devices of claim 15 wherein

displaying in the near-eye display of the near-eye, mixed reality display device system the one or more literary content items with their respective print layout characteristics registered to the reading object in the field of view further comprises:

generating by the processor one or more page layouts including the one or more literary content items with respective print layout characteristics for the reading object type of the reading object in accordance with stored layout rules;

displaying by the display of the near-eye, mixed reality display system one or more pages based on the one or more page layouts in the field of view of the mixed reality display device system; and

changing by the processor the one or more pages displayed responsive to user physical action manipulating the one or more pages detected by the natural user interface.

19. The one or more processor readable non-volatile storage devices of claim 18 wherein generating by the processor one or more page layouts including the one or more literary content items with respective print layout characteristics for the reading object type of the reading object in accordance with stored layout rules further comprises:

assigning the one or more literary content items to one or more sets of literary content items based on a reading object type associated with each of the one or more literary content item, each set grouping one or more literary content items associated with a same or compatible reading object type;

determining an item order of display of the one or more literary content items based on user preferences;

determining a set order of display of the one or more sets based on a priority indicated in the determined item order; and

generating a page layout set of one or more page layouts for each set of literary content items.

20. The one or more processor readable non-volatile storage devices of claim 19 wherein generating a page layout set of one or more page layouts for each set of literary content items further comprises:

determining by the processor an intra-set order of display of the one or more literary content items within the set based on user preferences;

selecting by the processor a block template for each page based on the reading object type for the set and the print layout characteristics of the items in the set;

assigning each literary content item in the set to a starting page number within the set based on priority in the intra-set order, size characteristics of the respective item and page dimensions for the reading object;

for each page layout, assigning each item assigned to the page layout to a block on the page layout based on the size characteristics of the respective literary content item; and

responsive to any of the blocks not satisfying visibility criteria, performing adjustment of one or more adjustable print layout characteristics of the literary content items for satisfying visibility criteria.