Method and apparatus for providing service using a sensor and image recognition in a portable terminal

• US 9,410,810 B2
• Filed: 09/23/2010
• Issued: 08/09/2016
• Est. Priority Date: 09/24/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jae-Myeon, Hong, Hyun-Su, Jeong, Mi-Jin
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US12/924,253
• Publication Number:
  US 20110071757A1
• Time in Patent Office: 2,147 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3492 employing speed data or traffic data, e.g. real-time or historical traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G03B 13/10 adjusting viewfinders field
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3245 of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
• CPC Class Codes: H04N 2201/3252 Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3273 Display
• CPC Class Codes: H04N 2201/328 Processing of the additional information
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A method and an apparatus provides a service using a sensor and image recognition in a portable terminal that supports Augmented Reality (AR). The method includes determining whether the portable terminal is parallel to ground using an acceleration sensor. When the portable terminal is parallel to the ground, either a map including nearby Point Of Interest (POI) information or constellation information is displayed.

Claims

1. A method for providing a service using a sensor and image recognition in a portable terminal, comprising:

displaying, by the portable terminal, a preview image obtained by a camera of the portable terminal;

determining, using an acceleration sensor, whether the portable terminal is moved over a first threshold distance of two or more threshold distances;

in response to determining the portable terminal is moved over the first threshold distance, adjusting a zoom level of the camera based on a distance of movement of the portable terminal;

determining, using the acceleration sensor, whether the portable terminal is moved over a second threshold distance of two or more threshold distances; and

in response to determining that the portable terminal is moved over the second threshold distance, restricting the zoom level of the camera.

2. The method of claim 1, further comprising:

determining, using the acceleration sensor, whether the portable terminal is bounced by the user; and

when the portable terminal is bounced backward by the user, zooming in by a second preset level according to at least one of an amount of the bounce and a number of the bounces, and when the portable terminal is bounced forward by the user, zooming out by a second preset level according to at least one of an amount of the bounce and a number of the bounces.

3. The method of claim 1, further comprising:

determining, using the acceleration sensor, whether the portable terminal is snapped; and

when the portable terminal is snapped backward by the user, zooming in by a third preset level according to at least one of an intensity of the snap and a number of the snaps, and when the portable terminal is snapped forward by the user, zooming out by a third preset level according to at least one of an intensity of the snap and a number of the snaps,

wherein the portable terminal is snapped by a wrist action of a user.

15. The method of claim 1, wherein adjusting the zoom level of the camera comprises:

when the portable terminal is moved backward over the first threshold distance, zooming in by a first preset level based on the distance of movement; and

when the portable terminal is moved forward over the first threshold distance, zooming out by the first preset level based on the distance of movement.

4. A method for providing a service using a sensor and image recognition in a portable terminal, comprising:

displaying, by the portable terminal, a preview image obtained by a camera of the portable terminal;

obtaining current location information and displaying Point Of Interest (POI) information in the preview image;

when an image is captured, synthesizing a graphical object indicating the POI information over the captured image;

tagging the synthesized image as an image of a POI corresponding to the POI information;

executing a related application for guiding an orientation of the portable terminal to the POI information of a corresponding location; and

displaying the synthesized image including the POI information using the related application.

5. The method of claim 4, wherein synthesizing the graphical object comprises:

determining whether the captured image comprises a user'"'"'s face; and

when the user'"'"'s face is detected, synthesizing the graphical object indicating the POI information around the user'"'"'s face.

6. The method of claim 4, further comprising:

when an Internet search function is executed while the POI information is displayed, extracting at least one search keyword from the POI information; and

performing the Internet search with at least one extracted search keyword.

7. The method of claim 4, wherein obtaining the current location information and displaying the POI information in the preview image comprises:

determining and displaying at least one of a travel distance and a travel time by considering an actual path leading to the POI corresponding to the POI information displayed in the preview image,

wherein the travel time is determined based on at least one of a user travel speed of the portable terminal and an average speed of a plurality of users over a certain time.

8. An apparatus for providing a service using a sensor and image recognition in a portable terminal, comprising:

a camera configured to obtain an image;

a display configured to display a preview image obtained through the camera;

an acceleration sensor configured to detect movement of the portable terminal;

a controller configured to;

determine, using of the acceleration sensor, whether the portable terminal is moved over a first threshold distance of two or more threshold distances;

in response to determining that the portable terminal is moved over the first threshold distance, adjust a zoom level of the camera based on a distance of movement of the portable terminal;

determine, using the acceleration sensor, whether the portable terminal is moved over a second threshold distance of the two or more threshold distances; and

in response to determining that the portable terminal is moved over the second threshold distance, restrict the zoom level of the camera.

9. The apparatus of claim 8, wherein the controller determines, using the acceleration sensor, whether the portable terminal is bounced backward by the user, and zooms in by a preset level according to at least one of an amount of the bounce and a number of the bounces, and when the portable terminal is bounced forward by the user, zooms out by a second preset level according to at least one of an amount of the bounce and a number of the bounces.

10. The apparatus of claim 8, wherein the controller determines, using the acceleration sensor, whether the portable terminal is snapped, and when the portable terminal is snapped backward by the user, zooms in by a preset level according to at least one of an intensity of the snap and a number of the snaps, and when the portable terminal is snapped forward by the user, zooms out by a third preset level according to at least one of an intensity of the snap and a number of the snaps,

wherein the portable terminal is snapped by a wrist motion of a user.

16. The apparatus of claim 8, wherein the controller is configured to:

zoom in by a first preset level based on the distance of movement when the portable terminal is moved backward over the first threshold distance; and

zoom out by the first preset level based on the distance of movement when the portable terminal is moved forward over the first threshold distance.

11. An apparatus for providing a service using a sensor and image recognition in a portable terminal, comprising:

a camera configured to capture an image;

a display unit configured to provide a preview of the image by displaying the image from the camera; and

a controller configured to;

obtain Point Of Interest (POI) information in the preview of the image,

synthesize a graphical object indicating the POI information over the captured image when the image is captured, tag the synthesized image as an image of a POI corresponding to the POI information,

zoom in by a first preset level when the portable terminal is moved backward by a user over a first threshold distance,

zoom out by the first preset level when the portable terminal is moved forward by the user over the first threshold distance, and

determine that no zoom operation should be performed when the portable terminal is moved either forward or backward over a second threshold distance, the second threshold distance larger than the first threshold distance.

12. The apparatus of claim 11, wherein the controller determines whether the captured image comprises a user'"'"'s face, and synthesizes the graphical object indicating the POI information around the user'"'"'s face when the user'"'"'s face is detected.

13. The apparatus of claim 11, further comprising:

a communication module configured to transmit and receive data,

wherein when an Internet search function is executed while the POI information is displayed, the controller extracts a at least one search keyword from the POI information and conducts the Internet search through the communication module using at least one extracted search keyword.

14. The apparatus of claim 11, wherein the controller controls to determine and display at least one of a travel distance and a travel time by considering an actual path leading to the POI corresponding to the POI information displayed in the preview image, and determines the travel time based on at least one of a user movement speed of the portable terminal and an average speed of a plurality of users over a certain time.

17. A method for providing a service using a sensor and image recognition in a portable terminal, comprising:

displaying, by the portable terminal, a preview image obtained by a camera of the portable terminal;

determining, using an acceleration sensor, whether the portable terminal is moved over at least one threshold distance of two or more threshold distances;

in response to determining that the portable terminal is moved over a first threshold distance, adjusting a zoom level of the camera based on a distance of movement of the portable terminal; and

in response to determining that the portable terminal is moved over a second threshold distance, restricting the zoom level of the camera,

wherein the first threshold distance is longer than the second threshold distance.

18. An apparatus for providing a service using a sensor and image recognition in a portable terminal, comprising:

a camera configured to obtain an image;

a display configured to display a preview image obtained through the camera;

an acceleration sensor configured to measure movement of the portable terminal; and

a controller configured to;

determine, using an acceleration sensor, whether the portable terminal is moved over at least one threshold distance of two or more threshold distances;

in response to determining that the portable terminal is moved over a first threshold distance, adjust a zoom level of the camera based on a distance of movement; and

in response to determining that the portable terminal is moved over a second threshold distance, restrict the zoom level of the camera, wherein the first threshold distance is longer than the second threshold distance.