Method and apparatus for providing service using a sensor and image recognition in a portable terminal

• US 10,578,452 B2
• Filed: 01/28/2019
• Issued: 03/03/2020
• Est. Priority Date: 09/24/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jae-Myeon, Hong, Hyun-Su, Jeong, Mi-Jin
• Primary Examiner(s): Goldman, Richard A
• Application Number:
  US16/259,158
• Publication Number:
  US 20190154458A1
• Time in Patent Office: 400 Days
• Field of Search: 701491, 701426, 701428, 701438, 701468, 701532
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3492 employing speed data or traffic data, e.g. real-time or historical traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G03B 13/10 adjusting viewfinders field
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3245 of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
• CPC Class Codes: H04N 2201/3252 Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3273 Display
• CPC Class Codes: H04N 2201/328 Processing of the additional information
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A method and an apparatus to provide a service using a sensor and image recognition in a portable terminal that supports Augmented Reality (AR). The method includes determining whether the portable terminal is parallel to ground using an acceleration sensor. When the portable terminal is parallel to the ground, either a map including nearby Point Of Interest (POI) information or constellation information is displayed.

Claims

1. An electronic device comprising:

a camera;

a display;

a global positioning system (GPS) receiver;

a memory storing instructions; and

a processor configured to execute the instructions to at least;

display, via the display, real-time images obtained from the camera,

identify an object within the obtained real-time images based on location information obtained from the GPS receiver,

identify a distance between a location of the electronic device and a location of the object based on the location information, and

display, via the display, an augmented reality (AR) graphical object including the information on the identified distance while displaying the real-time images.

2. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to:

identify moving speed of the electronic device using at least one sensor,

estimate, based on the identified moving speed, travel time between a location of the electronic device and a location of the object, and

display, via the display, information on the estimated travel time.

3. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to:

based on an input corresponding to a capture event, generate a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display,

store the generated first image in the memory, and

display the generated first image stored in the memory on the display in response to executing an application related to the generated first image including the AR graphical object.

4. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to:

determine whether the AR graphical object is being displayed on the display,

based at least on a determination that the AR graphical object is not displayed on the display, generate a second image by capturing only one of the obtained real-time images without the AR graphical object, and

store the generated second image in the memory.

5. The electronic device of claim 1, wherein the AR graphical object further comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

6. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to obtain information regarding the identified object from an external electronic device.

7. A method performed by an electronic device including a camera, a display, a global positioning system (GPS) receiver, and a memory, the method comprising:

displaying real-time images obtained from the camera;

identifying an object within the obtained real-time images based on location information obtained from the GPS receiver;

identifying a distance between a location of the electronic device and a location of the object based on the location information; and

displaying an augmented reality (AR) graphical object including the information on the identified distance while displaying the real-time images.

8. The method of claim 7, further comprising:

identifying moving speed of the electronic device using at least one sensor;

estimating, based on the identified moving speed, travel time between a location of the electronic device and a location of the object; and

displaying information on the estimated travel time.

9. The method of claim 7, further comprising:

based on an input corresponding to a capture event, generating a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display;

storing the generated first image in the memory; and

displaying the generated first image stored in the memory on the display in response to executing an application related to the generated first image including the AR graphical object.

10. The method of claim 7, further comprising:

determining whether the AR graphical object is being displayed on the display;

based at least on a determination that the AR graphical object is not displayed on the display, generating a second image by capturing only one of the obtained real-time images without the AR graphical object; and

storing the generated second image in the memory.

11. The method of claim 7, wherein the AR graphical object comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

12. The method of claim 7, further comprising:

obtaining information regarding the identified object from an external electronic device.