Electronic device, control method, and recording medium

• US 10,542,161 B2
• Filed: 02/25/2019
• Issued: 01/21/2020
• Est. Priority Date: 02/26/2018
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Kyocera Corporation
• Inventors: Tano, Masaki
• Primary Examiner(s): Nguyen, Allen H
• Application Number:
  US16/283,816
• Publication Number:
  US 20190268486A1
• Time in Patent Office: 330 Days
• Field of Search: 358 115
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/0044 for image preview or review, e.g. to help the user position a sheet
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2201/3245 of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
• CPC Class Codes: H04N 2201/3273 Display

Abstract

An electronic device includes a camera, a location sensor, a display, and a controller. The controller acquires a database including name information of tangible objects, first location information indicating locations of the tangible objects, and first feature point information related to feature points of the tangible objects. The controller generates image recognition information on the basis of a captured image captured by the camera, the image recognition information including distance information indicating a distance from the electronic device to a particular subject and second feature point information related to feature points of the particular subject. The controller identifies a name of the particular subject on the basis of the image recognition information, the database, and second location information indicating a location of the electronic device acquired by the location sensor. The controller causes an overlay image, yielded by overlaying the name on the captured image, to be displayed on the display.

Claims

1. An electronic device comprising:

a camera;

a location sensor;

a display; and

a controller configured to

acquire a database including name information of tangible objects, first location information indicating locations of the tangible objects, and first feature point information related to feature points of the tangible objects,

generate image recognition information on the basis of a captured image captured by the camera, the image recognition information including distance information indicating a distance from the electronic device to a particular subject and second feature point information related to feature points of the particular subject,

identify a name of the particular subject on the basis of the image recognition information, the database, and second location information indicating a location of the electronic device acquired by the location sensor, and

cause an overlay image, yielded by overlaying the name on the captured image, to be displayed on the display.

2. The electronic device of claim 1, further comprising

a communication interface configured to acquire the database from an external source, wherein

the controller is configured to acquire the database via the communication interface.

3. The electronic device of claim 1, further comprising

a storage in which the database is stored, wherein

the controller is configured to acquire the database stored in the storage.

4. The electronic device of claim 3, wherein the controller is configured to store the overlay image in the storage as a still image when a second operation is performed after the controller causes the overlay image to be displayed on the display.

5. The electronic device of claim 3, wherein the controller is configured to

start processing to store the overlay image in the storage as a moving image when a third operation is performed after the controller causes the overlay image to be displayed on the display, and

end the processing to store the overlay image as the moving image when a fourth operation is performed after the controller starts the processing to store the overlay image as the moving image.

6. The electronic device of claim 1, wherein the particular subject includes a natural object.

7. The electronic device of claim 1, wherein the controller is configured to

identify the tangible objects included in a predetermined range and identify a tangible object indicating the particular subject from among the tangible objects included in the predetermined range, and

cause an overlay image, yielded by overlaying the name representing the tangible object indicating the particular subject on the captured image, to be displayed on the display.

8. The electronic device of claim 7, wherein the controller is configured to

identify the tangible objects included in a first range by identifying the first location information included in the first range, the first range including points included between the location indicated by the second location information and locations away from the location indicated by the second location information by the distance indicated by the distance information,

identify the tangible object indicating the particular subject by identifying the first feature point information matching the second feature point information from among the first feature point information of the tangible objects included in the first range, and

cause an overlay image, yielded by overlaying the name representing the tangible object indicating the particular subject on the captured image, to be displayed on the display.

9. The electronic device of claim 8, further comprising

a direction sensor, wherein

the controller is configured to

identify the tangible objects included in the first range by identifying the first location information included in the first range, the first range including points included between the location indicated by the second location information and locations away from the location indicated by the second location information by the distance indicated by the distance information in a direction indicated by the direction sensor,

identify the tangible object indicating the particular subject by identifying the first feature point information matching the second feature point information from among the first feature point information of the tangible objects included in the first range, and

cause an overlay image, yielded by overlaying the name representing the tangible object indicating the particular subject on the captured image, to be displayed on the display.

10. The electronic device of claim 9, wherein the controller is configured to

identify the tangible objects included in a second range by identifying the first location information included in the second range when no tangible object is included in the first range, the second range including points included between the location indicated by the second location information and locations away from the location indicated by the second location information by the distance indicated by the distance information without regard to the direction,

identify the tangible object indicating the particular subject by identifying the first feature point information matching the second feature point information from among the first feature point information of the tangible objects included in the second range, and

cause an overlay image, yielded by overlaying the name representing the tangible object indicating the particular subject on the captured image, to be displayed on the display.

11. The electronic device of claim 9, wherein the controller is configured to

identify the tangible objects included in a second range by identifying the first location information included in the second range when the first feature point information of the tangible objects included in the first range does not include first feature point information matching the second feature point information, the second range including points included between the location indicated by the second location information and locations away from the location indicated by the second location information by the distance indicated by the distance information without regard to the direction,

identify the tangible object indicating the particular subject by identifying the first feature point information matching the second feature point information from among the first feature point information of the tangible objects included in the second range, and

cause an overlay image, yielded by overlaying the name representing the tangible object indicating the particular subject on the captured image, to be displayed on the display.

12. The electronic device of claim 1, wherein the controller is configured to cause the overlay image to be displayed on the display when a first operation is performed.

13. The electronic device of claim 1, wherein

the overlay image comprises a first overlay image and a second overlay image, and

the controller is configured to

cause the first overlay image to be displayed on the display when names of a plurality of particular subjects included in the captured image are identified, the first overlay image being yielded by overlaying the names of the plurality of particular subjects on the captured image, and

cause the second overlay image to be displayed on the display when a predetermined operation including an operation to select one or more of the particular subjects is performed after the first overlay image is displayed, the second overlay image being yielded by hiding the name of the selected one or more of the particular subjects.

14. The electronic device of claim 1, wherein the controller is configured to

cause the captured image to be displayed on the display, and

cause the overlay image to be displayed on the display when a predetermined operation including an operation to select one or more of particular subjects is performed after the captured image is displayed on the display, the overlay image being yielded by overlaying the name of the selected one or more of the particular subjects on the captured image.

15. The electronic device of claim 1, wherein

the image recognition information includes ratio information indicating a ratio of the particular subject to whole of the captured image, and

the controller does not overlay the name of the particular subject on the captured image when the particular subject has ratio information below a predetermined threshold.

16. The electronic device of claim 1, wherein the image recognition information includes three-dimensional map information of a range included in the captured image.

17. The electronic device of claim 16, wherein the controller is configured to

judge, on the basis of the three-dimensional map information, whether location of the particular subject in the overlay image has moved, and

change a display location of the name overlaid on the captured image when it is judged that the location of the particular subject has moved.

18. The electronic device of claim 17, wherein the controller is configured to move the display location of the name in the overlay image without changing a relative display location of the name relative to the particular subject when the location of the particular subject in the overlay image has moved.

19. The electronic device of claim 18, wherein the name is overlaid near the particular subject.

20. The electronic device of claim 19, wherein the name is not overlaid on the particular subject.

21. A method for controlling an electronic device comprising a camera, a location sensor, a display, a communication interface, and a controller, the method comprising:

causing, using the controller, the communication interface to acquire a database from an external source, the database including name information of tangible objects, first location information indicating locations of the tangible objects, and first feature point information related to feature points of the tangible objects;

generating, using the controller, image recognition information on the basis of a captured image captured by the camera, the image recognition information including distance information indicating a distance from the electronic device to a particular subject and second feature point information related to feature points of the particular subject;

identifying, using the controller, a name of the particular subject on the basis of the image recognition information, the database, and second location information indicating a location of the electronic device acquired by the location sensor; and

causing, using the controller, an overlay image yielded by overlaying the name on the captured image to be displayed on the display.

22. A non-transitory computer-readable recording medium including computer program instructions to be executed by an electronic device comprising a camera, a location sensor, a display, a communication interface, and a controller, the instructions causing the controller to:

cause the communication interface to acquire a database from an external source, the database including name information of tangible objects, first location information indicating locations of the tangible objects, and first feature point information related to feature points of the tangible objects;

generate image recognition information on the basis of a captured image captured by the camera, the image recognition information including distance information indicating a distance from the electronic device to a particular subject and second feature point information related to feature points of the particular subject;

identify a name of the particular subject on the basis of the image recognition information, the database, and second location information indicating a location of the electronic device acquired by the location sensor; and

cause an overlay image, yielded by overlaying the name on the captured image, to be displayed on the display.