Hands-free augmented reality for wireless communication devices

• US 9,171,384 B2
• Filed: 06/28/2012
• Issued: 10/27/2015
• Est. Priority Date: 11/08/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Babak, Velarde, Ruben M., Bednar, David L., Momeyer, Brian Lee
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/536,543
• Publication Number:
  US 20130113827A1
• Time in Patent Office: 1,216 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

This disclosure relates to techniques for providing hands-free augmented reality on a wireless communication device (WCD). According to the techniques, an application processor within the WCD executes an augmented reality (AR) application to receive a plurality of image frames and convert the plurality of image frames into a single picture comprising the plurality of image frames stitched together to represent a scene. The WCD executing the AR application then requests AR content for the scene represented in the single picture from an AR database server, receives AR content for the scene from the AR database server, and processes the AR content to overlay the single picture for display to a user on the WCD. In this way, the user may comfortably look at the single picture with the overlaid AR content on a display of the WCD to learn more about the scene represented in the single picture.

Claims

1. A method of executing an augmented reality application with a wireless communication device (WCD) comprising:

receiving, with a processor of the WCD, a plurality of image frames each representing a portion of a scene;

converting the plurality of image frames into a single picture, wherein the single picture comprises at least portions of the plurality of image frames stitched together to represent the scene;

presenting the single picture to a user of the WCD;

requesting augmented reality (AR) content for the scene represented in the single picture from an AR database server;

progressively receiving the AR content for the scene represented in the single picture from the AR database server;

processing the AR content to overlay the single picture presented to the user of the WCD, wherein processing the AR content to overlay the single picture comprises progressively adding the AR content to the single picture as the AR content is received from the AR database server;

performing a zoom process on the single picture to generate a new view of the scene represented in the single picture; and

updating the AR content to overlay the new view in the single picture, wherein updating the AR content comprises one of adding or removing AR content of the single picture.

2. The method of claim 1, wherein the plurality of image frames comprises one of a plurality of still pictures or a plurality of video frames of video data of the scene.

3. The method of claim 1, wherein receiving the plurality of image frames comprises one or more of receiving image frames captured with a camera included in the WCD, or receiving image frames from a storage device external to the WCD.

4. The method of claim 1, wherein requesting the AR content comprises sending one of the single picture or the portions of the image frames to the AR database server.

5. The method of claim 1, wherein converting the plurality of image frames into a single picture comprises stitching, with the processor of the WCD, the portions of the image frames together to form the single picture.

6. The method of claim 1, wherein converting the image data to a single picture comprises:

sending the plurality of image frames to a remote server;

instructing the remote server to stitch together the portions of the image frames to form the single picture; and

receiving the single picture from the remote server.

7. The method of claim 1, wherein the AR database server comprises a remote server.

8. The method of claim 1, further comprising:

determining AR content settings for the scene represented in the single picture based on user input; and

sending the AR content settings for the scene to the AR database server, wherein progressively receiving AR content comprises progressively receiving the AR content for the scene represented in the single picture from the AR database server according to the AR content settings.

9. The method of claim 8, wherein determining the AR content settings comprises receiving user input indicating one or more types of AR content to receive from the AR database server for the scene.

10. The method of claim 8, wherein determining the AR content settings comprises receiving user input indicating an amount of AR content to receive from the AR database server for the scene.

11. The method of claim 8, wherein determining the AR content settings comprises receiving user input indicating an amount of time in which to receive the AR content for the scene from the AR database server.

12. The method of claim 8, further comprising:

receiving user input indicating a modification to the AR content settings for the scene; and

updating the AR content to overlay the single picture according to the modified AR content settings, wherein updating the AR content comprises one of adding or removing AR content from the single picture.

13. The method of claim 1, further comprising requesting additional AR content for the scene represented in the single picture from the AR database server, and processing the additional AR content to overlay the single picture in addition to the existing AR content for display on the WCD.

14. The method of claim 1, further comprising:

receiving one or more of location information or perspective information from one or more sensors for each of the plurality of image frames that represents a portion of the scene; and

sending the one or more of the location information or the perspective information for at least one portion of the scene to the AR database server,

wherein progressively receiving AR content comprises progressively receiving the AR content for the portion of the scene represented in the single picture from the AR database server at least partially based on the one or more of the location information or the perspective information for the portion of the scene.

15. The method of claim 1, wherein the AR content comprises one or more of text or graphics representing the AR content for the scene represented in the single picture, and position information for the one or more of the text or the graphics corresponding to the single picture.

16. The method of claim 1, wherein the AR content comprises one or more of text or graphics that indicates differences between the scene represented in the single picture and previous pictures of the scene.

17. A wireless communication device (WCD) for executing an augmented reality application, the WCD comprising:

a memory configured to store image data; and

a processor configured to;

receive a plurality of image frames each representing a portion of a scene,

convert the plurality of image frames into a single picture, wherein the single picture comprises at least portions of the plurality of image frames stitched together to represent the scene,

present the single picture to a user of WCD,

request augmented reality (AR) content for the scene represented in the single picture from an AR database server,

progressively receive the AR content for the scene represented in the single picture from the AR database server,

process the AR content to overlay the single picture presented to the user of the WCD, wherein the processor is configured to progressively add the AR content to the single picture as the AR content is received from the AR database server,

perform a zoom process on the single picture to generate a new view of the scene represented in the single picture, and

update the AR content to overlay the new view in the single picture, wherein updating the AR content comprises one of adding or removing AR content of the single picture.

18. The WCD of claim 17, wherein the plurality of image frames comprises one of a plurality of still pictures or a plurality of video frames of video data of the scene.

19. The WCD of claim 17, further comprising a camera included in the WCD to capture image frames, wherein the processor receives the plurality of image frames from one or more of the camera included in the WCD or a storage device external to the WCD.

20. The WCD of claim 17, further comprising a transmitter, wherein the processor sends one of the single picture or the portions of the image frames to the AR database server via the transmitter to request the AR content for the scene.

21. The WCD of claim 17, wherein the processor stitches the portions of the image frames together to form the single picture.

22. The WCD of claim 17, further comprising a transmitter and a receiver, wherein the processor sends the plurality of image frames to a remote server via the transmitter, instructs the remote server to stitch together the portions of the image frames to form the single picture, and receives the single picture from the remote server via the receiver.

23. The WCD of claim 17, wherein the AR database server comprises a remote server.

24. The WCD of claim 17, wherein the processor determines AR content settings for the scene represented in the single picture based on user input, sends the AR content settings for the scene to the AR database server, and progressively receives the AR content for the scene represented in the single picture from the AR database server according to the AR content settings.

25. The WCD of claim 24, wherein the processor receives user input indicating one or more types of AR content to receive from the AR database server for the scene.

26. The WCD of claim 24, wherein the processor receives user input indicating an amount of AR content to receive from the AR database server for the scene.

27. The WCD of claim 24, wherein the processor receives user input indicating an amount of time in which to receive the AR content for the scene from the AR database server.

28. The WCD of claim 24, wherein the processor receives user input indicating a modification to the AR content settings for the scene, and updates the AR content to overlay the single picture according to the modified AR content settings, wherein updating the AR content comprises one of adding or removing AR content from the single picture.

29. The WCD of claim 17, wherein the processor requests additional AR content for the scene represented in the single picture from the AR database server, and processes the additional AR content to overlay the single picture in addition to the existing AR content for display on the WCD.

30. The WCD of claim 17, wherein the processor receives at least one of location information or perspective information from one or more sensors for each of the plurality of image frames that represents a portion of the scene, sends the one or more of the location information or the perspective information for at least one portion of the scene to the AR database server, and progressively receives the AR content for the portion of the scene represented in the single picture from the AR database server at least partially based on the one or more of the location information or the perspective information for the portion of the scene.

31. The WCD of claim 17, wherein the AR content comprises one or more of text or graphics representing the AR content for the scene represented in the single picture, and position information for the one or more of the text or the graphics corresponding to the single picture.

32. The WCD of claim 17, wherein the AR content comprises one or more of text or graphics that indicate differences between the scene represented in the single picture and previous pictures of the scene.

33. A wireless communication device (WCD) for executing an augmented reality application comprising:

means for receiving a plurality of image frames each representing a portion of a scene;

means for converting the plurality of image frames into a single picture, wherein the single picture comprises at least portions of the plurality of image frames stitched together to represent the scene;

means for presenting the single picture to a user of the WCD;

means for requesting augmented reality (AR) content for the scene represented in the single picture from an AR database server;

means for progressively receiving the AR content for the scene represented in the single picture from the AR database server;

means for processing the AR content to overlay the single picture presented to the user of the WCD, wherein the means for processing the AR content to overlay the single picture comprise means for progressively adding the AR content to the single picture as the AR content is received from the AR database server;

means for performing a zoom process on the single picture to generate a new view of the scene represented in the single picture; and

means for updating the AR content to overlay the new view in the single picture, wherein updating the AR content comprises one of adding or removing AR content of the single picture.

34. The WCD of claim 33, wherein the means for converting the plurality of image frames into a single picture comprise means for stitching the portions of the image frames together to form the single picture.

35. The WCD of claim 33, wherein the means for converting the image data to a single picture comprise:

means for sending the plurality of image frames to a remote server;

means for instructing the remote server to stitch the portions of the image frames together to form the single picture; and

means for receiving the single picture from the remote server.

36. The WCD of claim 33, further comprising:

means for determining AR content settings for the scene represented in the single picture based on user input;

means for sending the AR content settings for the scene to the AR database server; and

means for progressively receiving the AR content for the scene represented in the single picture from the AR database server according to the AR content settings.

37. The WCD of claim 36, further comprising:

means for receiving user input indicating a modification to the AR content settings for the scene; and

means for updating the AR content to overlay the single picture according to the modified AR content settings, wherein updating the AR content comprises one of adding or removing AR content from the single picture.

38. A non-transitory computer-readable medium comprising instructions for executing an augmented reality application that when executed in a wireless communication device (WCD) cause one or more programmable processors to:

receive a plurality of image frames each representing a portion of a scene;

convert the plurality of image frames into a single picture, wherein the single picture comprises at least portions of the plurality of image frames stitched together to represent the scene;

present the single picture to a user of the WCD;

request augmented reality (AR) content for the scene represented in the single picture from an AR database server;

progressively receive the AR content for the scene represented in the single picture from the AR database server;

process the AR content to overlay the single picture presented to the user of the WCD, wherein the instructions cause the one or more processors to progressively add the AR content to the single picture as the AR content is received from the AR database server;

perform a zoom process on the single picture to generate a new view of the scene represented in the single picture; and

update the AR content to overlay the new view in the single picture, wherein updating the AR content comprises one of adding or removing AR content of the single picture.

39. The non-transitory computer-readable medium of claim 38, wherein the instructions that cause the processors to convert the plurality of image frames into a single picture further comprise instructions that cause the processors to stitch the portions of the image frames together to form the single picture.

40. The non-transitory computer-readable medium of claim 38, wherein the instructions that cause the processors to convert the plurality of image frames into a single picture further comprise instructions that cause the processors to:

send the plurality of image frames to a remote server;

instruct the remote server to stitch the portions of the image frames together to form the single picture; and

receive the single picture from the remote server.

41. The non-transitory computer-readable medium of claim 38, further comprising instructions that cause the processors to:

determine AR content settings for the scene represented in the single picture based on user input;

send the AR content settings for the scene to the AR database server; and

progressively receive the AR content for the scene represented in the single picture from the AR database server according to the AR content settings.

42. The non-transitory computer-readable medium of claim 41, further comprising instructions that cause the processors to:

receive user input indicating a modification to the AR content settings for the scene; and

update the AR content to overlay the single picture according to the modified AR content settings, wherein updating the AR content comprises one of adding or removing AR content from the single picture.