Method and apparatus for providing service using a sensor and image recognition in a portable terminal

• US 10,190,885 B2
• Filed: 02/05/2018
• Issued: 01/29/2019
• Est. Priority Date: 09/24/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jae-Myeon, Hong, Hyun-Su, Jeong, Mi-Jin
• Primary Examiner(s): Goldman, Richard A
• Application Number:
  US15/888,551
• Publication Number:
  US 20180156628A1
• Time in Patent Office: 358 Days
• Field of Search: 701491, 701426, 701428, 701438, 701468, 701532
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3492 employing speed data or traffic data, e.g. real-time or historical traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G03B 13/10 adjusting viewfinders field
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3245 of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
• CPC Class Codes: H04N 2201/3252 Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3273 Display
• CPC Class Codes: H04N 2201/328 Processing of the additional information
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A method and an apparatus to provide a service using a sensor and image recognition in a portable terminal that supports Augmented Reality (AR). The method includes determining whether the portable terminal is parallel to ground using an acceleration sensor. When the portable terminal is parallel to the ground, either a map including nearby Point Of Interest (POI) information or constellation information is displayed.

Claims

1. An electronic device comprising:

a camera;

a display;

a storage storing instructions; and

a processor configured to execute the instructions to at least;

obtain real-time images from the camera,

identify an object within the obtained real-time images,

display an augmented reality (AR) graphical object related to the identified object on the display while obtaining the real-time images,

based on an input corresponding to a capture event, generate a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display,

store the generated first image in the storage, and

display the generated first image stored in the storage on the display in response to executing an application related to the generated first image including the AR graphical object.

2. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to:

determine whether the AR graphical object is being displayed on the display,

based at least on a determination that the AR graphical object is not displayed on the display, generate a second image by capturing only one of the obtained real-time images without the AR graphical object, and

store the generated second image in the storage.

3. The electronic device of claim 1, wherein the AR graphical object comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

4. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to obtain information regarding the identified object from an external electronic device.

5. A method performed by an electronic device including a camera, a display, and a storage, the method comprising:

obtaining real-time images from the camera;

identifying an object within the obtained real-time images;

displaying an augmented reality (AR) graphical object related to the identified object on the display while obtaining the real-time images;

based on an input corresponding to a capture event, generating a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display;

storing the generated first image in the storage; and

displaying the generated first image stored in the storage on the display in response to executing an application related to the generated first image including the AR graphical object.

6. The method of claim 5, further comprising:

determining whether the AR graphical object is being displayed on the display;

based at least on a determination that the AR graphical object is not displayed on the display, generating a second image by capturing only one of the obtained real-time images without the AR graphical object; and

storing the generated second image in the storage.

7. The method of claim 5, wherein the AR graphical object comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

8. The method of claim 5, further comprising:

obtaining information regarding the identified object from an external electronic device.

9. A method performed by an electronic device including a camera, a display, a storage, and a processor, the method comprising:

controlling, by the processor, to obtain real-time images from the camera;

controlling, by the processor, to identify an object within the obtained real-time images;

controlling, by the processor, to display an augmented reality (AR) graphical object related to the identified object on the display while obtaining the real-time images;

based on an input corresponding to a capture event, controlling, by the processor, to generate a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display;

controlling, by the processor, to store the generated first image in the storage; and

controlling, by the processor, to display the generated first image stored in the storage on the display in response to executing an application related to the generated first image including the AR graphical object.

10. The method of claim 9, further comprising:

controlling to determine whether the AR graphical object is being displayed on the display;

based at least on a determination that the AR graphical object is not displayed on the display, controlling, by the processor, to generate a second image by capturing only one of the obtained real-time images without the AR graphical object; and

controlling to store the generated second image in the storage.

11. The method of claim 9, wherein the AR graphical object comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

12. The method of claim 9, further comprising:

controlling, by the processor, to obtain information regarding the identified object from an external electronic device.

13. A non-transitory computer readable recording medium storing instructions executable by a processor of an electronic device including a camera, a display, and a storage to cause the processor to at least:

control to obtain real-time images from the camera;

control to identify an object within the obtained real-time images;

control to display an augmented reality (AR) graphical object related to the identified object on the display while obtaining the real-time images;

based on an input corresponding to a capture event, control to generate a first image by combining the AR graphical object and one of the obtained real-time images only while displaying the AR graphical object on the display;

control to store the generated first image in the storage; and

control to display the generated first image stored in the storage on the display in response to executing an application related to the generated first image including the AR graphical object.

14. The non-transitory computer readable recording medium of claim 13, wherein the instructions further cause the processor to:

control to determine whether the AR graphical object is being displayed on the display;

based at least on a determination that the AR graphical object is not displayed on the display, control to generate a second image by capturing only one of the obtained real-time images without the AR graphical object; and

control to store the generated second image in the storage.

15. The non-transitory computer readable recording medium of claim 13, wherein the AR graphical object comprises at least one of name information, location information, category information, phone number information, or related image information of the identified object.

16. The non-transitory computer readable recording medium of claim 13, wherein the instructions further cause the processor to:

control to obtain information regarding the identified object from an external electronic device.