AUGMENTED REALITY MAPS

• US 20110199479A1
• Filed: 02/12/2010
• Published: 08/18/2011
• Est. Priority Date: 02/12/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Waldman, Jaron
• Granted Patent: US 9,488,488 B2
• Time in Patent Office: Days
• US Class Current: 348/116
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G01C 21/3682 output of POI information on a road map G01C21/3614, G01C21/3685 take precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A user points a handheld communication device to capture and display a real-time video stream. The handheld communication device detects geographic position, camera direction, and tilt of the image capture device. The user sends a search request to a server for nearby points of interest. The handheld communication device receives search results based on the search request, geographic position, camera direction, and tilt of the handheld communication device. The handheld communication device visually augments the captured video stream with data related to each point of interest. The user then selects a point of interest to visit. The handheld communication device visually augments the captured video stream with a directional map to a selected point of interest in response to the user input.

Claims

1. A method implemented on a handheld communication device having a processor, an image-capture device, and a display, the method comprising:

presenting a captured image on the display;

detecting a geographic position of the handheld communication device and a camera direction of the image capture device;

processing map data describing map objects including streets, buildings, and points of interest near the geographic position of the device, the map data further being processed to generate a route from the geographic position of the device to a selected place of interest; and

overlaying an illustration representing at least a portion of the route on the captured image by correlating the map data describing the route to the geographic position and the camera direction of the image capture device to match objects depicted in the captured image with objects described by the map data.

2. The method of claim 1, further comprising:

requesting, from a server, the map data describing an area surrounding the geographic position of the device.

3. The method of claim 1, further comprising:

searching for places of interest fitting a description that has been input by a user using an input device.

4. The method of claim 1, further comprising:

receiving inputs from a touch screen device effective to select a point of interest and generate and display the route to the point of interest.

5. The method of claim 1, wherein the route is overlaid over streets in the captured image.

6. The method of claim 1, wherein turn arrows are overlaid onto the captured image to indicate upcoming turns in the route as the user travels along the route.

7. A system comprising:

a video capture device configured to capture video images;

a global positioning system device (GPS device) configured to determine GPS coordinates identifying the location of the video capture device;

a digital compass configured to determine a camera direction of the video capture device;

a server for receiving and servicing requests for map data for areas surrounding the GPS location identified by the GPS device, the map data being descriptive of features such as places, roads, buildings, and points of interest; and

a general-purpose processor in digital communication with the video capture device, the GPS device, the digital compass, a graphics processor, and a display, the general-purpose processor for correlating objects represented in the images captured by the video capture device with the same objects as represented by the map data, the correlating being accomplished by using the GPS location and camera direction to match an object as represented in the map data with its position in reality as captured in video;

the graphics processor configured to overlay an illustrated portion of a route from the location identified by the GPS device to a point of interest, the route being determined based on the map data, the graphics processor further configured to update the illustrated portion of the route as the GPS location of the video capture device changes with movement along the route; and

the display configured to present the captured video images with the portion of the route overlaid.

8. The system of claim 7, wherein the display is a touch screen for accepting inputs from the user effective to enter into the general-purpose processor, search terms used to define a request for a point of interest.

9. The system of claim 8, wherein the touch screen display further accepts inputs from the user to select a point of interest, the selection of a point of interest causing the communications interface to request a route to the point of interest from a map database.

10. The system of claim 7, further comprising an accelerometer for determining a tilt orientation of the video capture device.

11. The system of claim 10, wherein the display changes from presentation of the captured video images with the portion of the route overlaid to display of a schematic map with route information when the accelerometer determines that the video capture device is substantially facing the ground.

12. The system of claim 10, wherein the general-purpose processor also uses the tilt orientation to match the location of the object as represented in the map data with its position in reality as captured in video.

13. A product, comprising:

a computer-readable medium, and

computer-executable instructions on the computer-readable medium for causing a computer having a processor and a display to perform the method comprising

presenting a captured image on the display;

detecting a geographic position, camera direction, and tilt orientation of the image capture device;

sending a search request to a server for nearby points of interest based on the detected geographic position, camera direction, and tilt orientation of the capture device;

receiving points of interest in response to the search request;

visually augmenting the captured image with data related to each point of interest by correlating the location of the points of interest to a location relative to the captured image using the geographic position, camera direction and tilt orientation of the image capture device to identify the objects represented in the captured image; and

visually augmenting the captured image with an illustrated route to a selected point of interest in response to a user input.

14. The product of claim 13, wherein the method further comprises filtering the received points of interest before visually augmenting the captured image.

15. The product of claim 13, wherein the method further comprises augmenting the captured image with a turn indication corresponding to turns in the route.

16. The product of claim 13, wherein the method further comprises overlaying the illustrated route onto the displayed streets.

17. A handheld communication device, comprising:

an image-capture device for recording an image, the image including objects and the image capture device having a field of view of a limited angle;

a global positioning system device (GPS device) for determining GPS coordinates for a present location of the device;

a digital compass for detecting a camera direction of the device;

an accelerometer for detecting a tilt orientation of the device;

a communications interface for requesting and receiving map information data for an area surrounding the present location of the device;

the map information data includes search results for a requested category of points of interest;

a processor configured to correlate visible objects in the recorded image with objects represented in the map information data by matching a location on a map with the GPS coordinates;

to determine which map objects are in the field of view by determining at least the camera direction of the device and calculating which map objects are within the captured image; and

to determine which points of interest fall within the field of view when it is extended a certain range beyond the visible objects;

a touch screen display device for displaying the recorded image the map data correlated to the recorded image and determined to be within the field of view, the touch screen display further configured to accept the user input selecting a point of interest; and

the processor configured to respond to the user input to cause the communications device to request and receive directions forming a route from the present GPS coordinates to the selected point of interest, the processor further configured to cause the display device to display a portion of the route overlaid onto the recorded image.

18. The device of claim 17, wherein the processor instructs the display to change from display of the captured video images with the portion of the route overlaid to displaying a schematic map with route information when the tilt information indicates that the device is substantially facing the ground.

19. The device of claim 17, wherein the processor is further configured to cause the display to present indicator arrows indicating that points of interest are outside the field of view.

20. The device of claim 17, wherein the processor is further configured to use the tilt information to determine what objects are in the captured image.