Smartphone-based methods and systems

• US 9,354,778 B2
• Filed: 08/05/2014
• Issued: 05/31/2016
• Est. Priority Date: 12/06/2013
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Cornaby, Colin P., Weaver, Matthew M.
• Primary Examiner(s): AMIN, JWALANT B
• Application Number:
  US14/452,239
• Publication Number:
  US 20150163345A1
• Time in Patent Office: 665 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

The present disclosure generally concerns sensor-equipped consumer electronic devices, such as smartphones and tablet computers. One implementation is directed toward a smartphone configured to operate in an image discovery mode. Results from the image discovery mode may be displayed on a smartphones display screen. The smartphone may concurrently operate an audio discovery mode. Results from the audio discovery mode are added to a discovery history, but are not displayed on the smartphone'"'"'s display screen while operating in the image discovery mode. Of course, many other combinations and implementations are provided as well.

Claims

1. A method utilizing a portable device comprising a camera, a microphone, a movement sensor and a touchscreen display, said method comprising the acts of:

receiving image data captured by the camera of the portable device;

communicating data associated with the image data to a remotely located server;

in response to said communicating, receiving identifying information associated with an object depicted in the image data, the identifying information comprising an image or graphic;

causing the touchscreen display to simultaneously display;

i) a first tile comprising an image or graphic, the first tile associated with a first image-recognized object, and a second tile comprising an image or graphic, the second tile associated with a second image-recognized object; and

ii) live image data captured by the camera while the portable device remains in an image discovery mode, in which the image discovery mode operates to identify objects depicted in captured image data; and

operating the portable device in an image discovery mode rather than an audio discovery mode based on data obtained from the movement sensor, in which the audio discovery mode operates to identify audio captured with the microphone.

2. The method of claim 1 in which the movement sensor comprises an accelerometer or gyroscope.

3. The method of claim 1 in which the first tile, second tile and live image data are displayed on the same plane of the touchscreen display.

4. The method of claim 1 in which, in response to input received through the touchscreen display, displaying a third tile in a stacked arrangement with the first tile and the second tile.

5. The method of claim 4 further comprising, causing the stacked arrangement to grow in tiles until a predetermined height is reached, and then tiles resulting from content discovery cause a bottom tile in the stacked arrangement to sink out of sight on the touchscreen display.

6. The method of claim 1 in which the live image data is displayed in a viewfinder window.

7. The method of claim 6 in which upon the viewfinder window being occluded, said method further comprises deactivating the camera.

8. The method of claim 6 further comprising causing the portable device to vibrate to confirm recognition of an object depicted in the viewfinder window.

9. A smartphone comprising:

a touchscreen display;

a movement sensor;

a camera for capturing image data;

a microphone for capturing audio data;

memory for storing software instructions;

one or more processors programmed with said software instructions for;

processing image data captured by said camera;

communicating data associated with the processed image data to a remotely located server;

in response to said communicating, receiving identifying information associated with an object depicted in the image data, the identifying information comprising an image or graphic, the identifying information associated with fingerprint data extracted from data representing the image data;

causing the touchscreen display to simultaneously display;

i) a first tile including an image or graphic, the first tile associated with a first image-recognized object, and a second tile including an image or graphic, the second tile associated with a second image-recognized object; and

ii) a viewfinder area for displaying live image data captured by said camera while the smartphone remains in an image discovery mode, in which the image discovery mode operates to identify objects depicted in captured image data from fingerprint analysis; and

upon the viewfinder area being occluded, deactivating said camera.

10. The smartphone of claim 9 in which said one or more processors are programmed for making an inference from data obtained from said movement sensor of a preference for operating said smartphone in the image discovery mode rather than an audio discovery mode, in which the audio discovery mode operates to identify audio captured with said microphone.

11. The smartphone of claim 10 in which said movement sensor comprises an accelerometer or gyroscope.

12. The smartphone of claim 9 in which the first tile, second tile and viewfinder area are displayed on the same plane of said touchscreen display.

13. The smartphone of claim 9 in which said one or more processors are programmed for, in response to input received through said touchscreen display, displaying a third tile in a stacked arrangement with the first tile and the second tile.

14. The smartphone of claim 13 in which said one or more processors are programmed for causing the stacked arrangement to grow in tiles until a predetermined height is reached, and then tiles resulting from further content discovery cause a bottom tile in the stack to sink out of sight on the touchscreen display.

15. The smartphone of claim 9 in which said one or more processors are programmed for controlling a vibration to confirm recognition of an object depicted in the viewfinder area.

16. The smartphone of claim 9 in which said one or more processors programmed with said software instructions include programming for:

determining whether to disable an audio discovery mode based on information from said movement sensor, in which the audio discovery mode operates to identify audio captured with said microphone.

17. The smartphone of claim 9 in which said one or more processors programmed with said software instructions include programming for:

switching between an audio discovery and the image discovery mode based on information obtained from said movement sensor, in which the audio discovery mode operates to identify audio captured with said microphone.

18. A system comprising:

A. a smartphone comprising;

a touchscreen display;

a camera for capturing image data;

a microphone for capturing audio data;

one or more processors configured for;

communicating data associated with captured image data to a remotely located server;

in response to said communicating, receiving payoff information associated with an object depicted in the image data, the payoff information comprising an image or graphic;

causing the touchscreen display to simultaneously display;

i) stacked tiles comprising a plurality of tiles including a first tile including an image or graphic, the first tile associated with a first image-recognized object, and a second tile including an image or graphic, the second tile associated with a second image-recognized object; and

ii) a viewfinder window where live image data captured by said camera is displayed while said smartphone remains in an image discovery mode, in which the image discovery mode operates to identify objects depicted in captured image data; and

while operating in the image discovery mode, concurrently operating an audio discovery mode, and including a third tile obtained from the audio discovery mode in a tile history but not in the stacked tiles, in which the audio discovery mode operates to identify audio captured with said microphone;

B. said remotely located server comprising one or more processors configured for;

processing data associated with remotely captured imagery to extract identifying information therefrom;

indexing the identifying information in a data repository to find payoff information associated with an object depicted in the captured imagery; and

control communication of the payoff information to said smartphone.

19. The system of claim 18 in which said smartphone further comprises a movement sensor, and in which said smartphone'"'"'s one or more processors are configured for making an inference from data obtained from said movement sensor of a preference for operating said smartphone in the image discovery mode rather than the audio discovery mode.

20. The system of claim 19 in which said movement sensor comprises an accelerometer or gyroscope.

21. The system of claim 18 in which the first tile, second tile and viewfinder window are displayed on the same plane of said touchscreen display.

22. The system of claim 18 in which said smartphone'"'"'s one or more processors are configured for, in response to input received through said touchscreen display, displaying a forth tile in the stacked tiles.

23. The system of claim 18 in which upon the viewfinder window being occluded, said smartphone'"'"'s one or more processors are configured for deactivating said camera.

24. The system of claim 18 in which said smartphone'"'"'s one or more processors are configured for controlling a vibration to confirm recognition of an object depicted in the viewfinder window.

25. The system of claim 18 in which said smartphone'"'"'s one or more processors are programmed for causing the stacked tiles to grow in tiles until a predetermined height is reached, and then tiles resulting from image or audio discovery cause a bottom tile in the stack to sink out of sight on said touchscreen display.

26. A smartphone comprising:

means for displaying imagery and graphics;

means for capturing image data;

means for capturing audio data;

means for processing image data captured by said means for capturing image data;

means for communicating data associated with processed image data to a remotely located server;

means for, in response to communicating data associated with processed image data, receiving identifying information associated with an object depicted in the image data, the identifying information comprising an image or graphic;

means for simultaneously displaying on said means for displaying imagery and graphics;

i) stacked tiles comprising a first tile including an image or graphic, the first tile associated with a first image-recognized object, and a second tile including an image or graphic, the second tile associated with a second image-recognized object; and

ii) a viewfinder window where live image data captured by said means for capturing imagery is displayed while said smartphone remains in an image discovery mode, in which the image discovery mode operates to identify objects depicted in captured image data; and

means for concurrently operating an audio discovery mode while operating in the image discovery mode, and means for including a third tile obtained from the audio discovery mode in a tile history but not in the stacked tiles, in which the audio discovery mode controls said smartphone to identify audio captured with said microphone.

27. The smartphone of claim 26 in which said smartphone comprises means for displaying a tile in the stacked tiles, in response to input received through said means for simultaneously displaying.

28. The smartphone of claim 26 further comprising means for deactivating said means for capturing image data, which operates upon the viewfinder window being occluded.

29. The smartphone of claim 26 further comprising means for controlling a vibration to confirm recognition of an object depicted in the viewfinder window.

30. The smartphone of claim 26 in which the first tile, second tile and viewfinder window are displayed on the same plane of said means displaying imagery and graphics.