Imaging device and information acquisition system in which an acquired image and associated information are held on a display

• US 9,420,251 B2
• Filed: 02/04/2011
• Issued: 08/16/2016
• Est. Priority Date: 02/08/2010
• Status: Active Grant

• Current Assignee: Nikon Corporation
• Original Assignee: Nikon Corporation
• Inventors: Inoue, Hideya
• Primary Examiner(s): Velez, Roberto
• Assistant Examiner(s): Segura, Cynthia
• Application Number:
  US13/021,198
• Publication Number:
  US 20110193985A1
• Time in Patent Office: 2,020 Days
• Field of Search: 348/231.3
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06Q 30/0207 Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/32144 embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3273 Display
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

The imaging device of the invention includes an imaging portion that images a subject; a positional information acquisition portion that acquires positional information of an imaging position; a control portion which acquires information on the subject based on the positional information, and displays image data of the subject and the information on the subject on a display portion; and a hold control portion that outputs a hold control signal, which holds the image data of the subject and the information on the subject, to the control portion.

Claims

1. An imaging device comprising:

a body;

an imager configured to image a subject;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

acquire information on the subject based on the positional information, and to display image data of the subject and the information on the subject on a display of the imaging device, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the image data of the subject and the information on the subject displayed in the AR display mode and to continue displaying the held image data and the held information on the display after completion of the user action, the displayed information on the subject including at least one AR object, the computer being configured such that parameters of (a) contents of the image data of the subject displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the image data of the subject displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

2. The imaging device according to claim 1,

wherein, in the AR display mode, the computer can switch between the display and the non-display of the information on the subject on the display.

3. The imaging device according to claim 1,

wherein, when the hold control signal is received while imaging the subject, the computer displays the held image data of the subject and the held information on the subject on the display.

4. The imaging device according to claim 1,

wherein the computer is configured to transmit the held information on the subject and the held image data of the subject to another terminal on the basis of the hold control signal.

5. An information acquisition system comprising:

an imaging device and an information retrieval system;

wherein the imaging device is the imaging device according to claim 1.

6. The imaging device according to claim 1, wherein the display has a touch panel that is configured to be operative for a touch operation while the image data of the subject and the information on the subject are held and displayed on the display.

7. The imaging device according to claim 1, further comprising an operation portion by which post information about the held image data of the subject and/or the held information on the subject is posted.

8. The imaging device according to claim 1, wherein the display has an input that is used to input post information while the image data of the subject and the information on the subject are held and displayed on the display.

9. The imaging device according to claim 1, wherein the display operates in a mode in which an icon is displayed for posting post information while the image data of the subject and the information on the subject are held and displayed on the display.

10. The imaging device according to claim 1,

wherein the information on the subject includes coupon information,

and wherein the computer is configured to hold the coupon information on the display based on the hold control signal.

11. The imaging device according to claim 1,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

12. The imaging device according to claim 1,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

13. The imaging device according to claim 1,

wherein the computer acquires further information on the subject when the display is touched after the holding.

14. The imaging device according to claim 1,

wherein the information on the subject includes coupon information.

15. An imaging device comprising:

a body;

an imager;

a memory;

a latitude and longitude detector that detects latitude and longitude information of the body;

a azimuthal angle detector which detects an azimuthal angle by which image data is imaged by the imager; and

a hardware computer configured to;

acquire augmented reality (AR) information about a structure imaged by the imager based on the latitude and longitude information and the azimuthal angle, the computer being operable in a AR display mode in which a real-time image acquired by the imager and the augmented reality information are displayed on a display of the imaging device; and

output a hold control signal upon detecting a predetermined user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the real-time image and the augmented reality information displayed in the AR display mode and to continue displaying the held real-time image and the held augmented reality information after completion of the predetermined user action, the AR information including at least one AR object, the computer being configured such that parameters of (a) contents of the real-time image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the real-time image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

16. The imaging device according to claim 15,

wherein, when the hold control signal is output, the computer stores the latitude and longitude information and the azimuthal angle of the body, detected during imaging of the image data, on the memory by adding an image identification information that identifies the image data.

17. The imaging device according to claim 15,

wherein, when the hold control signal is output, the computer stores the image data displayed on the display and the augmented reality information corresponding to the image data on the memory by adding an image identification information that identifies the image data.

18. The imaging device according to claim 15,

wherein the memory stores image identification information that identifies the image data for each image data, structure identification information that identifies a structure in the image data, structure information of a structure shown by the structure identification information, and a user ID added to the image identification information, for each image data.

19. The imaging device according to claim 15,

wherein the computer stores the augmented reality information and the image data in the memory based on the hold control signal, and reads the augmented reality information and the image data from the memory and displays the same on the display.

20. The imaging device according to claim 15,

wherein the computer transmits the augmented reality information added in an imaging state to an information retrieval system.

21. The imaging device according to claim 15,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which a posture of the body is changed, an action by which the body is moved, an action by which the display is touched, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

22. The imaging device according to claim 15,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

23. The imaging device according to claim 15,

wherein the computer acquires further information on the subject when the display is touched after the holding.

24. The imaging device according to claim 15,

wherein the information on the subject includes coupon information.

25. The imaging device according to claim 15,

wherein the AR information includes past AR information or present AR information.

26. The imaging device according to claim 15,

wherein the AR information includes an icon of navigation with respect to the structure.

27. An information acquisition system which includes an imaging device and an information retrieval system configured to extract a structure situated in a range of a direction based on latitude and longitude information and an azimuthal angle transmitted from the imaging device, and to transmit information about the extracted structure to the imaging device,

the imaging device comprising:

a body;

an imager;

a latitude and longitude detector that detects the latitude and longitude information of the body;

an azimuthal angle detector which detects the azimuthal angle by which image data is imaged by the imager and

a hardware computer configured to;

acquire augmented reality (AR) information about the structure imaged by the imager based on the latitude and longitude information and the azimuthal angle, the computer being operable in an AR display mode in which a real-time image acquired by the imager and the augmented reality information are displayed on a display of the imaging device; and

output a hold control signal upon detecting a predetermined user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the real-time image and the augmented reality information displayed in the AR display mode and to continue displaying the held real-time image data and the held augmented reality information on the display after completion of the predetermined user action, the AR information including at least one AR object, the computer being configured such that parameters of (a) contents of the real-time image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the real- time image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action, and

the information retrieval system comprising;

a database which stores (i) a map data, in which a structure identification number of a structure corresponds to latitude and longitude information of the structure, and (ii) a structure table, in which the structure identification number corresponds to the augmented reality information of the structure shown by the structure identification number; and

an information retrieval server which (a) retrieves, from the map data, the structure identification number of the structure situated in the range of the direction based on the latitude and longitude information and the azimuthal angle transmitted from the imaging device, (b) reads the augmented reality information added to the structure shown by the structure identification number from the structure table, and (c) transmits the augmented reality information of the read structure to the imaging device.

28. The information acquisition system according to claim 27,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

29. The information acquisition system according to claim 27,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

30. The information acquisition system according to claim 27,

wherein the computer acquires further information on the subject when the display is touched after the holding.

31. The information acquisition system according to claim 27,

wherein the information on the subject includes coupon information.

32. A non-transitory computer-readable storage medium that stores a program that allows a computer to execute functions of an imaging device so that the computer executes the steps comprising:

inputting positional information of a position, which is detected by a positional information acquirer, of a subject, image data of which is imaged by an imager of the imaging device;

acquiring information on the subject based on the positional information;

AR-displaying a real-time image acquired by the imager and the information on the subject on a display; and

outputting a hold control signal based on a user action during the AR-displaying, the hold control signal causing holding, on the display, of the real-time image and the information on the subject and continued displaying of the held real-time image data and the held information on the subject on the display after completion of the user action, the displayed information on the subject including at least one AR object, the holding being performed such that parameters of (a) contents of the image data of the subject displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the image data of the subject displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

33. The non-transitory computer-readable storage medium according to claim 32,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

34. The non-transitory computer-readable storage medium according to claim 32,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

35. The non-transitory computer-readable storage medium according to claim 32,

wherein the computer acquires further information on the subject when the display is touched after the holding.

36. The non-transitory computer-readable storage medium according to claim 32,

wherein the information on the subject includes coupon information.

37. A non-transitory computer-readable storage medium that stores a program that allows a computer to execute functions of an imaging device so that the computer executes steps comprising:

inputting latitude and longitude information of a body of the imaging device which is detected by a latitude and longitude detector;

inputting an azimuthal angle by which image data is imaged by an imager of the imaging device, the azimuthal angle being detected by an azimuthal angle detector;

acquiring augmented reality information about a structure imaged by the imager based on the latitude and longitude information and the azimuthal angle;

AR-displaying a real-time image acquired by the imager and the augmented reality information on a display; and

outputting a hold control signal to the imaging device upon detecting a predetermined user action during the AR-displaying, the hold control signal causing holding, on the display, of the real-time image and the augmented reality information displayed in the AR-displaying and continued displaying of the held real-time image data and the held augmented reality information on the display after completion of the user action, the AR information including at least one AR object, the holding being configured such that parameters, of (a) contents of the real-time image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the real-time image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

38. The non-transitory computer-readable storage medium according to claim 37,

wherein the information on the subject includes coupon information.

39. An imaging device comprising:

a body;

an imager configured to image a subject;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

display, on a display of the imaging device, a subject image acquired by the imager and subject information acquired based on the positional information, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action while the computer is operating in the AR display mode, the hold control signal causing the computer to keep displaying, on the display, the subject image and the subject information displayed in the AR display mode even if a position and/or a posture of the body is changed after completion of the user action, the displayed subject information including at least one AR object, the computer being configured such that parameters of (a) contents of the subject image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the subject image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

40. The imaging device according to claim 39,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

41. The imaging device according to claim 39,

wherein the computer acquires further information on the subject when the display is touched after the holding.

42. The imaging device according to claim 39,

wherein the information on the subject includes coupon information.

43. An imaging device comprising:

a body;

a sensor provided in the body;

an imager configured to image a subject;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

display, on a display of the imaging device, a subject image acquired by the imager and subject information acquired based on the positional information, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a detection by the sensor of a motion of the body according to a user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the subject image and the subject information displayed in the AR display mode and to continue displaying the held subject image and the held subject information on the display after completion of the user action.

44. The imaging device according to claim 43,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

45. The imaging device according to claim 43,

wherein the computer acquires further information on the subject when the display is touched after the holding.

46. The imaging device according to claim 43,

wherein the information on the subject includes coupon information.

47. An imaging device comprising:

a body;

an imager configured to image a subject; and

a hardware computer configured to;

display, on a display of the imaging device, a subject image acquired by the imager and subject information acquired based on a result of an image recognition for the subject image, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the subject image and the subject information displayed in the AR display mode and to continue displaying the held subject image and the held subject information on the display after completion of the user action, the displayed subject information including at least one AR object, the computer being configured such that parameters of (a) contents of the subject image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the subject image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

48. The imaging device according to claim 47,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

49. The imaging device according to claim 47,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

50. The imaging device according to claim 47,

wherein the computer acquires further information on the subject when the display is touched after the holding.

51. The imaging device according to claim 47,

wherein the information on the subject includes coupon information.

52. An imaging device comprising:

a body;

an imager; and

a hardware computer configured to operate in;

an augmented reality (AR) display mode in which a real-time subject image acquired by the imager and subject information are displayed on a display of the imaging device, the subject information being acquired based on positional information of an imaging position and/or based on a result of an image recognition for the subject image; and

a holding mode, starting based on a user action while in the AR display mode, and in which the subject image and the subject information displayed in the AR display mode keep being displayed on the display even when a position and/or a posture of the body is changed after completion of the user action, the displayed subject information including at least one AR object, the holding mode being operated such that parameters of (a) contents of the subject image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the subject image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

53. The imaging device according to claim 52,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

54. The imaging device according to claim 52,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

55. The imaging device according to claim 52,

wherein the computer acquires further information on the subject when the display is touched after the holding.

56. The imaging device according to claim 52,

wherein the information on the subject includes coupon information.

57. An imaging device comprising:

a body;

an imager; and

a hardware computer configured to operate in;

an augmented reality (AR) display mode in which a real-time subject image of a subject acquired by the imager and subject information are displayed on a display of the imaging device, the subject information being acquired based on positional information of an imaging position and/or based on a result of an image recognition for the subject image; and

a holding mode, starting based on a user action while in the AR display mode, and in which the subject image and the subject information displayed in the AR display mode continue being displayed on the display even when the imager is directed not toward the subject image after the user action, the displayed subject information including at least one AR object, the holding mode being operated such that parameters of (a) contents of the subject image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the subject image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

58. The imaging device according to claim 57,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

59. The imaging device according to claim 57,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

60. The imaging device according to claim 57,

wherein the computer acquires further information on the subject when the display is touched after the holding.

61. The imaging device according to claim 57,

wherein the information on the subject includes coupon information.

62. An imaging device comprising:

a body;

an imager configured to image a subject;

a button provided on the body such that the button is pressed down at least when a subject image is captured via the imager;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

display, on a display of the imaging device, a subject image acquired by the imager and subject information acquired based on the positional information, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action in which the button is pressed down while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the subject image and the subject information displayed in the AR display mode and, also after completion of the user action, to continue displaying the held subject image and the held subject information on the display.

63. The imaging device according to claim 62,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

64. The imaging device according to claim 62,

wherein the computer acquires further information on the subject when the display is touched after the holding.

65. The imaging device according to claim 62,

wherein the information on the subject includes coupon information.

66. An imaging device comprising:

a body;

an imager configured to image a subject;

a touch panel type display provided on the body;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

display, on the touch panel display, a subject image acquired by the imager and subject information acquired based on the positional information, the being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action in which the touch panel display is touched while the computer operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the subject image and the subject information displayed in the AR display mode and, also after completion of the user action, to continue displaying the held subject image and the held subject information on the display, the displayed subject information including at least one AR object, the computer being configured such that parameters of (a) contents of the subject image displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the subject image displayed on the display, immediately before the user action are respectively the same as those after completion of the user action.

67. The imaging device according to claim 66,

wherein the touch panel display has an icon related to the touch operation for the holding.

68. The imaging device according to claim 66,

wherein the computer transmits the held information on the subject to an external device when the hold control signal is detected.

69. imaging device according to claim 66,

wherein the computer acquires further information on the subject when the display is touched after the holding.

70. The imaging device according to claim 66,

wherein the information on the subject includes coupon information.

71. An imaging device comprising:

a body;

an imager configured to image a subject;

a positional information acquirer configured to acquire positional information of an imaging position of the body; and

a hardware computer configured to;

acquire information on the subject based on the positional information, and to display image data of the subject and the information on the subject on a display of the imaging device, the computer being operable in an augmented reality (AR) display mode; and

output a hold control signal based on a user action while the computer is operating in the AR display mode, the hold control signal causing the computer to hold, on the display, the image data of the subject and the information on the subject displayed in the AR display mode, the displayed information of the subject including at least one AR object, the computer being configured such that parameters of (a) contents of the image data of the subject displayed on the display, (b) the number of the at least one AR object displayed on the display, and (c) an arrangement of the at least one AR object with respect to the image data of the subject displayed on the display, immediately before the user action are respectively the same as those after completion of the user action, wherein the computer transmits the held information on the subject and the held image data of the subject to another terminal when the computer detects the hold control signal.

72. The imaging device according to claim 71,

wherein the user action includes at least one of an action by which a button provided on the body is pressed-down, an action by which the display is touched, an action by which a posture of the body is changed, an action by which the body is moved, an action by which a detected characteristic point of an image is moved in a preset distance in a time of a preset range, an action by which a detected contrast of an image is changed beyond a preset difference, and an action by which a change in an image pattern, which is equal to or greater than a predetermined threshold value, is to be detected.

73. The imaging device according to claim 71,

wherein the computer acquires further information on the subject when the display is touched after the holding.

74. The imaging device according to claim 71,

wherein the information on the subject includes coupon information.