Reactive overlays of multiple representations using augmented reality

• US 10,169,899 B2
• Filed: 03/18/2018
• Issued: 01/01/2019
• Est. Priority Date: 09/16/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dey, Prasenjit, Sundararajan, Sharad C.
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/924,263
• Publication Number:
  US 20180211429A1
• Time in Patent Office: 289 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A computer system, method, and computer readable product are provided for conveying multiple representations of a concept to a user via augmented reality, with a mechanism to easily move between these multiple representations along multiple axes. In various embodiments, an augmented reality image is generated and displayed by a computing device. The augmented reality image may then be modified in response to user input along multiple axes, such as complexity, depth, or variety of a concept being conveyed with the augmented reality image.

Claims

1. A method for manipulating multiple dimensions of an augmented reality image, comprising:

capturing an image of a physical scene with a computing device;

identifying a physical object within the physical scene;

generating an overlay image based on the physical object;

rendering, on a display of the computing device, a first augmented reality image comprising the overlay image overlaid on the image of the physical scene;

receiving user input indicative of modifying the first augmented reality image according to a first dimension, the dimension comprising complexity, variety, or depth of a concept;

rendering, on the display of the computing device, a second augmented reality image based on the user input indicative of modifying the first augmented reality image according to the first dimension;

receiving user input indicative of modifying the second augmented reality image according to a second dimension, the second dimension comprising complexity, variety, or depth of the concept, the second dimension differing from the first dimension; and

rendering, on the display of the computing device, a third augmented reality image based on the user input indicative of modifying the second augmented reality image according to the second dimension, and based on the user input indicative of modifying the first augmented reality image according to the first dimension, wherein rendering the second augmented reality image comprises;

modifying the first object; and

determining how to modify the second object based on a relationship between the first object and the second object.

2. The method of claim 1, further comprising:

aligning the overlay image with the image of the physical scene before rendering the first augmented reality image.

3. The method of claim 1, wherein rendering the first augmented reality image comprises:

rendering a first user interface element, wherein receiving the indication of user input at the first user interface element is indicative of modifying the first augmented reality image according to the first dimension; and

rendering a second user interface element, wherein receiving the indication of user input at the second user interface element is indicative of modifying the second augmented reality image according to the second dimension.

4. A computer system, comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is configured to;

render a first augmented reality image;

receive user input indicative of modifying the first augmented reality image according to a first dimension;

render a second augmented reality image based on the user input indicative of modifying the first augmented reality image according to the first dimension;

receive user input indicative of modifying the second augmented reality image according to a second dimension; and

render a third augmented reality image based on the user input indicative of modifying the second augmented reality image according to the second dimension, and based on the user input indicative of modifying the first augmented reality image according to the first dimension, wherein rendering the second augmented reality image comprises;

modifying the first object; and

determining how to modify the second object based on a relationship between the first object and the second object.

5. The computer system of claim 4, wherein the augmented reality image comprises an image of a real-world space overlaid with a computer-generated image, the computer-generated image comprising an animation.

6. The computer system of claim 4, wherein the augmented reality image comprises an image of a real-world space overlaid with a computer-generated image, the computer-generated image comprising a user-interface element that may be manipulated in response to an indication of user input received by the computer system.

7. The computer system of claim 6, wherein the user-interface element comprises a knob or a slider.

8. The computer system of claim 4, wherein:

the second dimension comprises complexity, variety, or depth of the concept, and

the second dimension differs from the first dimension.

9. The computer system of claim 4, wherein the computer system is further configured to:

receive input indicative of a concept graph that expresses the relationship between the first object and the second object of the first augmented reality image in one or more pairs of parameters and elements.

10. The computer system of claim 4, wherein the first dimension comprises one of a complexity, a variety, and a depth of a concept conveyed by the first augmented reality image.

11. A computer program product, comprising:

a computer readable storage medium having programming instructions embodied therewith, the program instructions executable by a computer cause the computer to;

render a first augmented reality image;

receive user input indicative of modifying the first augmented reality image according to a first dimension;

render a second augmented reality image based on the user input indicative of modifying the first augmented reality image according to the first dimension;

receive user input indicative of modifying the second augmented reality image according to a second dimension; and

render a third augmented reality image based on the user input indicative of modifying the second augmented reality image according to the second dimension, and based on the user input indicative of modifying the first augmented reality image according to the first dimension, wherein rendering the second augmented reality image comprises;

modifying the first object; and

determining how to modify the second object based on a relationship between the first object and the second object.

12. The computer program product of claim 11, wherein a relationship between the first object and the second object is maintained in a concept graph comprising at least one pair that identifies a parameter and a corresponding effect.

13. The computer program product of claim 11, wherein the program instructions executable by the computer further cause the computer to:

parse the concept graph to determine that the parameter of the first object affects the parameter of the second object in the second augmented reality image.

14. The computer program product of claim 11, wherein the program instructions executable by the computer further cause the computer to:

align an overlay image with the image of the physical scene before rendering the first augmented reality image.

15. The computer program product of claim 11, wherein the program instructions executable by the computer further cause the computer to:

generate an overlay image; and

overlay the overlay image on an image of the first augmented reality image.

16. The computer program product of claim 11, wherein the first dimension comprises complexity, variety, or depth of the concept.

17. The computer program product of claim 11, wherein the second dimension comprises complexity, variety, or depth of a concept.

18. The computer program product of claim 17, wherein the second dimension differs from the first dimension.