Method and apparatus for using augmented reality for business graphics

• US 9,633,476 B1
• Filed: 10/29/2009
• Issued: 04/25/2017
• Est. Priority Date: 10/29/2009
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Laaser, William T., Huff, Gerald B.
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US12/608,053
• Time in Patent Office: 2,735 Days
• Field of Search: 345632, 345633
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

One embodiment provides a system for using Augmented Reality (AR) to facilitate displaying business graphics. The system receives at a computer system a video stream including an image of a marker, the marker being a physical object that is manipulated by the user and is recognizable by the computer system. The system recognizes the marker and receives from the user a selection of a business graphic. Next, the system renders the business graphic over the image of the marker in the video stream such that the image of the marker is obscured by the business graphic. The system then recognizes a manipulation of the marker indicated by a change in the image of the marker in the video stream. Next, the system performs an action on the business graphic that corresponds to the manipulation of the marker. Finally, the system displays the video stream to the user.

Claims

1. A computer-implemented method for using Augmented Reality (AR) for business graphics, the method comprising:

receiving at a computer system a video stream, wherein the video stream includes an image of a marker, and wherein the marker is a physical object that is manipulated by a user and is recognizable by the computer system;

recognizing the marker at the computer system;

receiving from the user a selection of a business graphic, wherein a business graphic is a visual representation of business data;

rendering the business graphic over the image of the marker in the video stream such that the image of the marker is obscured by the business graphic;

recognizing a physical manipulation of the marker at the computer system, wherein the physical manipulation of the marker is indicated by a change in the image of the marker in the video stream;

performing an action on the business graphic that corresponds to the physical manipulation of the marker, wherein the action is displayed in the video stream;

in response to receiving a command from the user to disregard physical manipulations of the marker, fixing the business graphic to the last known location of the image of the marker in the video stream, thereby allowing the user to physically manipulate the marker without affecting the business graphic;

after fixing the business graphic to the last known location of the image of the marker in the video stream, recognizing a virtual manipulation on the business graphic at the computer system, wherein the virtual manipulation is indicated when the user moves one of the user'"'"'s hands to simulate a rotation of the business graphic;

performing a second action on the business graphic that corresponds to the virtual manipulation, wherein the second action corresponds to rotating the business graphic and the second action is displayed in the video stream; and

displaying the video stream to the user.

2. The computer-implemented method of claim 1, wherein recognizing the marker involves:

determining a location of the image of the marker in the video stream; and

determining an orientation of the marker in the video stream.

3. The computer-implemented method of claim 1, wherein rendering the business graphic over the image of the marker involves rendering the business graphic so that the business graphic has the same location and the same orientation as the image of the marker in the video stream.

4. The computer-implemented method of claim 1, wherein recognizing the marker involves recognizing a depth of the image of the marker in the video stream.

5. The computer-implemented method of claim 1, wherein recognizing the marker involves recognizing a visually unique pattern on the marker.

6. The computer-implemented method of claim 1, wherein recognizing the marker involves recognizing the marker while it is partially obscured.

7. The computer-implemented method of claim 1, further comprising:

recognizing a second marker at the computer system;

recognizing a manipulation of the second marker at the computer system; and

performing a second action on the business graphic that corresponds to the manipulation of the second marker, wherein the second action is displayed in the video stream.

8. The computer-implemented method of claim 1, wherein rendering the business graphic over the image of the marker in the video stream involves animating the business graphic to include a movement, wherein the movement is not related to a change in an orientation or a location of the marker or a camera providing the video stream.

9. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for using Augmented Reality (AR) for business graphics, the method comprising:

receiving at a computer system a video stream, wherein the video stream includes an image of a marker, and wherein the marker is a physical object that is manipulated by a user and is recognizable by the computer system;

recognizing the marker at the computer system;

receiving from the user of the computer system a selection of a business graphic, wherein a business graphic is a visual representation of business data;

rendering the business graphic over the image of the marker in the video stream such that the image of the marker is obscured by the business graphic;

recognizing a physical manipulation of the marker at the computer system, wherein the physical manipulation of the marker is indicated by a change in the image of the marker in the video stream;

performing an action on the business graphic that corresponds to the physical manipulation of the marker, wherein the action is displayed in the video stream;

in response to receiving a command from the user to disregard physical manipulations of the marker, fixing the business graphic to the last known location of the image of the marker in the video stream, thereby allowing the user to physically manipulate the marker without affecting the business graphic;

after fixing the business graphic to the last known location of the image of the marker in the video stream, recognizing a virtual manipulation on the business graphic at the computer system, wherein the virtual manipulation is indicated when the user moves one of the user'"'"'s hands to simulate a rotation of the business graphic;

performing a second action on the business graphic that corresponds to the virtual manipulation, wherein the second action corresponds to rotating the business graphic and the second action is displayed in the video stream; and

displaying the video stream to the user.

10. The non-transitory computer-readable storage medium of claim 9, wherein recognizing the marker involves:

determining a location of the image of the marker in the video stream; and

determining an orientation of the marker in the video stream.

11. The non-transitory computer-readable storage medium of claim 9, wherein rendering the business graphic over the image of the marker involves rendering the business graphic so that it has the same location and the same orientation as the image of the marker in the video stream.

12. The non-transitory computer-readable storage medium of claim 9, wherein recognizing the marker involves recognizing a depth of the image of the marker in the video stream.

13. The non-transitory computer-readable storage medium of claim 9, wherein recognizing the marker involves recognizing a visually unique pattern on the marker.

14. The non-transitory computer-readable storage medium of claim 9, wherein recognizing the marker involves recognizing the marker while it is partially obscured.

15. The non-transitory computer-readable storage medium of claim 9, wherein the method further comprises:

recognizing a second marker at the computer system;

recognizing a manipulation of the second marker at the computer system; and

performing a second action on the business graphic that corresponds to the manipulation of the second marker, wherein the second action is displayed in the video stream.

16. The non-transitory computer-readable storage medium of claim 9, wherein rendering the business graphic over the image of the marker in the video stream involves animating the business graphic to include a movement, wherein the movement is not related to a change in an orientation or a location of the marker or a camera providing the video stream.