AUGMENTED REALITY SYSTEM FOR SUPPLEMENTING AND BLENDING DATA

• US 20120120101A1
• Filed: 11/08/2011
• Published: 05/17/2012
• Est. Priority Date: 11/08/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Adhikari, Suranjit, Dunn, Ted, Hsiao, Eric
• Granted Patent: US 9,280,851 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A system, method, and computer program product for automatically combining computer-generated imagery with real-world imagery in a portable electronic device by retrieving, manipulating, and sharing relevant stored videos, preferably in real time. A video is captured with a hand-held device and stored. Metadata including the camera'"'"'s physical location and orientation is appended to a data stream, along with user input. The server analyzes the data stream and further annotates the metadata, producing a searchable library of videos and metadata. Later, when a camera user generates a new data stream, the linked server analyzes it, identifies relevant material from the library, retrieves the material and tagged information, adjusts it for proper orientation, then renders and superimposes it onto the current camera view so the user views an augmented reality.

Claims

1. A computer-implemented method for providing an augmented reality interface, comprising:

acquiring an image of a real-world scene and metadata with a camera;

storing the image and metadata;

retrieving at least one stored image with metadata having selected features;

manipulating the retrieved image; and

combining the manipulated video with at least one historical image and a currently observed real-world scene viewed with a portable electronic device.

2. The method of claim 1, wherein the method places the user in a historically-based reality.

3. The method of claim 1, wherein the method assists in educating the user on historical events.

4. The method of claim 1, wherein the image is at least one of a still photograph, at least one video frame, analog, digital, recorded, live, and communicated in a data stream.

5. The method of claim 1, wherein the metadata describes the physical location and orientation of the camera during the acquiring, and is provided by at least one of a GPS system, a gyroscope, and an accelerometer.

6. The method of claim 1, wherein the metadata is provided by the camera.

7. The method of claim 1, wherein at least one of the currently observed scene, images, and metadata are stored on at least one of a server and the portable electronic device.

8. The method of claim 1, wherein the selected features include the stored physical location and orientation best matching one of (a) a current physical location and orientation of the portable electronic device and (b) a predicted physical location and orientation of the portable electronic device.

9. The method of claim 1, wherein the server searches for the selected features.

10. The method of claim 1, wherein the retrieved image is in a second data stream.

11. The method of claim 1, wherein the portable electronic device is at least one of a smartphone, a hand-held device, the camera, a second camera, a PDA, and a tablet computer.

12. The method of claim 1, wherein the manipulating includes adjusting image orientation.

13. The method of claim 1, wherein the combining includes superimposing the manipulated image on the currently observed scene.

14. The method of claim 1, wherein the combining includes merging the data stream with the second data stream.

15. The method of claim 1, wherein the combining includes displaying the manipulated image with the portable electronic device in one of a display and a viewfinder.

16. The method of claim 1, wherein the method operates continuously and substantially in real time.

17. The method of claim 1, wherein the method operates as the currently observed scene changes as the portable electronic device is moved, such motion including at least one of translating, tilting, panning, and zooming.

18. A system for providing an augmented reality interface, comprising:

a processor; and

a memory containing instructions that, when executed by the processor cause the processor to;

acquire a video of a real-world scene and metadata with a camera;

store the video and metadata;

retrieve at least one stored video with metadata having selected features;

manipulate the retrieved video; and

combine the manipulated video with at least one historical image and a currently observed real-world scene viewed with a portable electronic device.

19. A computer program product for providing an augmented reality interface, comprising a computer readable medium tangibly embodying non-transitory computer-executable program instructions thereon that, when executed, cause a computing device to:

acquire a video of a real-world scene and metadata with a camera;

store the video and metadata;

retrieve at least one stored video with metadata having selected features;

manipulate the retrieved video; and

combine the manipulated video with at least one historical image and a currently observed real-world scene viewed with a portable electronic device.

20. A system for providing an augmented reality interface, comprising:

means for acquiring a video of a real-world scene and metadata with a camera;

means for storing the video and metadata;

means for retrieving at least one stored video with metadata having selected features;

means for manipulating the retrieved video; and

means for combining the manipulated video with at least one historical image and a currently observed real-world scene viewed with a portable electronic device.