Visual space management across information handling system and augmented reality

• US 10,559,133 B2
• Filed: 02/07/2018
• Issued: 02/11/2020
• Est. Priority Date: 02/07/2018
• Status: Active Grant

• Current Assignee: Dell Products LP (Dell Technologies Inc.)
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Knoppert, Michiel Sebastiaan Emanuel Petrus, Aurongzeb, Deeder M.
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/890,800
• Publication Number:
  US 20190244426A1
• Time in Patent Office: 734 Days
• Field of Search: None
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts

Abstract

An information handling system coordinates presentation of a visual item with a flat panel display and head mounted display using tokens detected by one or more sensors, such as a depth camera and ultrasonic sensor. Upon detection of a token by a sensor, gestures detected by at least one sensor manipulate a three dimensional image presented at a location by the head mounted display. A visible surface determination of the view area sorts upright extents of the items presented by the head mounted display and flat panel display so the united view of an end user is uncluttered with image overlap.

Claims

1. An information handling system comprising:

a processor operable to execute instructions that process information;

a memory interfaced with the processor and operable to store the instructions and information;

a flat panel display operable to present visual information as visual images;

a head mounted display operable to present visual information as three dimensional visual images focused at a location in front of the head mounted display;

one or more graphics processors interfaced with one or more of the flat panel display and head mounted display and operable to communicate the visual information to the flat panel display and head mounted display formatted for presentation as visual images;

a plurality of sensors disposed proximate the flat panel display and operable to sense end user hand gestures;

a token having one or more features identifiable by one or more of the plural sensors; and

an environment definition engine interfaced with the plural sensors and the one or more graphics processors, the environment definition engine applying gestures detected by the plural sensors to visual images of the flat panel display absent the token identification of the token by the one or more of the plural sensors and applying gestures detected by the plural sensors to the visual images of the head mounted display in the event of identification of the token.

2. The information handling system of claim 1 wherein:

the plural sensors comprise at least a depth camera and an ultrasonic sensor; and

the environment definition engine applies the ultrasonic sensor to manipulate the head mounted display visual images if the depth camera detects the token.

3. The information handling system of claim 2 wherein the token features include a predetermined geometric shape.

4. The information handling system of claim 2 wherein the token features include a predetermined placement on the flat panel display.

5. The information handling system of claim 1 further comprising a mixed reality engine interfaced with the environment definition engine, the mixed reality engine coordinating presentation of flat panel display and head mounted display visual images by sorting upright extents of the visual images in visibility order.

6. The information handling system of claim 5 wherein the token is associated with color selection and the gestures include at least a hand movement associate with color selection.

7. The information handling system of claim 6 wherein the mixed reality engine applies colors selected for presentation at the head mounted display visual image to the flat panel display visual image.

8. The information handling system of claim 7 wherein the token comprises a stylus.

9. A method for presenting visual information at an information handling system, the method comprising:

executing an application on the information handling system to generate the visual information;

presenting the visual information as a visual image at a flat panel display interfaced with the information handling system;

detecting a predetermined object proximate the flat panel display with at least one of plural sensors;

in response to the detecting, presenting a three dimensional visual image of the image at the flat panel display with a head mounted display interfaced with the information handling system, the three dimensional image presented at a location proximate the visual image presented at the flat panel display;

in response to the detecting, associating one or more of the plural sensors to interact with the location of the three dimensional visual image; and

altering the three dimensional visual image in response to hand movements detected at the location by the one or more of the plural sensors.

10. The method of claim 9 further comprising:

initiating the location at the visual image presented on the flat panel display;

detecting a gesture from the location upwards; and

generating the three dimensional visual image at the location and moving the three dimensional visual image upward with the gesture to have a location raised above the flat panel display.

11. The method of claim 10 further comprising:

detecting overlap of the three dimensional visual image and the flat panel display visual image; and

in response to the detecting, altering the visual image at the flat panel display to remove the overlap.

12. The method of claim 9 wherein:

the plural sensors comprise at least a depth camera and an ultrasonic sensor;

detecting a predetermined object comprises detecting a token on the flat panel display with the depth camera; and

associating one or more of the plural sensors to interact with a location of a three dimensional visual image further comprises associating the ultrasonic sensor to detect gestures as inputs at the three dimensional visual image location.

13. The method of claim 9 further comprising:

detecting a second predetermined object associated with color of the three dimensional visual image;

altering the color at the three dimensional visual image in response to the detecting; and

altering the flat panel display visual image to have the color in response to the detecting.

14. The method of claim 13 wherein the second predetermined object comprises a stylus.

15. The method of claim 9 further comprising:

detecting manipulation of the predetermined object; and

in response to the detecting manipulation, associating the one or more of the plural sensors with inputs to the flat panel display visual image.

16. The method of claim 15 further comprising:

detecting overlap of the three dimensional visual image and the flat panel display visual image; and

in response to the detecting, altering the three dimensional visual image to remove the overlap.

17. A method for presenting visual information at an information handling system, the method comprising:

interfacing a flat panel display and a head mounted display with an information handling system;

monitoring an area proximate the flat panel display with plural sensors;

detecting a first predetermined condition with the plural sensors;

in response to the first predetermined condition, rendering an item as a visual image at only the flat panel display;

detecting a second predetermined condition with the plural sensors;

in response to detecting the second predetermined condition, rendering the item as a visual image at both the flat panel display and the head mounted display; and

in response to detecting the second predetermined condition, assigning gestures detected by one of the plural sensors to manipulation of the visual image presented by the head mounted display.

18. The method of claim 17 wherein the plural sensors comprise a depth camera and an ultrasonic sensor, the one of the plural sensors being the ultrasonic sensor, the ultrasonic sensor detecting gestures at a location of the head mounted display presentation of the visual image as a three dimensional visual image.

19. The method of claim 18 comprising:

comparing the relative view of the three dimensional visual image and the flat panel display visual image; and

sorting the upright extents of the three dimensional visual image and the flat panel display visual image to present the three dimensional visual image without overlap of the flat panel display visual image.

20. The method of claim 18 wherein the gesture comprises altering the three dimensional visual image surface color, the method further comprising rendering the flat panel display visual image to have the surface color selected by the gesture.