SMARTPHONE-BASED METHODS AND SYSTEMS

• US 20140357312A1
• Filed: 08/15/2014
• Published: 12/04/2014
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Davis, Bruce L., Knudson, Edward B., Rhoads, Geoffrey B., Rodriguez, Tony F., Rogers, Eliot, Cornaby, Colin P., Sinclair, Eoin C.
• Granted Patent: US 9,484,046 B2
• Time in Patent Office: Days
• US Class Current: 455/550.1
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 2201/0052 Embedding of the watermark in the frequency domain
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72469 for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
• CPC Class Codes: H04W 4/50 Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Arrangements involving portable devices (e.g., smartphones and tablet computers) are disclosed. One arrangement enables a content creator to select software with which that creator'"'"'s content should be rendered—assuring continuity between artistic intention and delivery. Another utilizes a device camera to identify nearby subjects, and take actions based thereon. Others rely on near field chip (RFID) identification of objects, or on identification of audio streams (e.g., music, voice). Some technologies concern improvements to the user interfaces associated with such devices. For example, some arrangements enable discovery of both audio and visual content, without any user requirement to switch modes. Other technologies involve use of these devices in connection with shopping, text entry, and vision-based discovery. Still other improvements are architectural in nature, e.g., relating to evidence-based state machines, and blackboard systems. Yet other technologies concern computational photography. A great variety of other features and arrangements are also detailed.

Claims

1. A method comprising the acts:

receiving audio data using a microphone of a user'"'"'s portable device;

receiving image data using a camera of said device;

recognition-processing both the received audio and image data—

without a user requirement to switch modes therebetween;

recognizing an item of audio content, and displaying a visual indicia related thereto, the indicia being selectable by the user to launch an online payoff related to the recognized audio content, said recognizing being performed by a hardware processor configured to perform such act; and

recognizing an item of visual content, and launching an online payoff related to the recognized visual content without waiting for user action.

2. The method of claim 1 that includes storing entries in a history data structure, said entries identifying both recognized audio content and recognized visual content.

3. A method comprising the acts:

receiving sensor data, the sensor data including audio data sensed using a microphone of a user'"'"'s portable device, and image data sensed using a camera of said device;

applying three or more different recognition-processing techniques to the received sensor data in cyclical serial fashion or in parallel fashion, including applying at least one recognition-processing technique to audio data, and applying at least one recognition-processing technique to image data; and

disabling one or more of said recognition-processing techniques based on context information indicating that the received sensor data is unsuitable for said one or more techniques, said context information being sensor-derived based on information from one or more sensors other than the camera.

4. The method of claim 3 that includes disabling said one or more of the recognition-processing techniques based on sensor-derived context information produced by a classifier.

5. The method of claim 3 that includes disabling said one or more of the recognition-processing techniques based on sensor-derived context information involving plural different types of sensor data.

6. A method comprising the acts:

receiving ambient content information based on data from a sensor in a user'"'"'s portable system, at first and second times;

invoking a recognition-processing technique on the content information at the first time; and

not invoking said recognition-processing technique on the content information at the second time;

wherein a processor in the portable system determines not to invoke the recognition-processing technique at the second time at least in part because of system context, the system context comprising battery state, or on-going processes from which resources should not be diverted.

7. A method comprising the acts:

receiving audio data using a microphone of a user'"'"'s portable device;

receiving image data using a camera of said device;

recognition-processing both the received audio and image data—

without a user requirement to switch modes therebetween;

inferring, based on input data, that the user has a preference for discovering visual content over audio content, said inferring being performed by a processor configured to apply rules to said input data to reach an inference;

notifying the user about recognition relating to the image data; and

not notifying the user about recognition relating to the audio data.

8. The method of claim 7 in which the input data comprises data from a sensor in the device.

9. The method of claim 7 in which the input data comprises data indicating the device is being held horizontally.

10. An extended discovery method comprising the acts:

capturing imagery depicting a subject of interest to a user, yielding image data, the capturing employing a camera-equipped device conveyed by the user;

performing recognition processing on the image data, said processing yielding data identifying the subject;

based on the identifying data, conducting an online investigation—

employing a variety of online resources—

into the subject, to develop a collection of diverse results corresponding to said subject; and

presenting said results using a display of a second device, and displaying said captured imagery in association with said presented results, the second device not being carried by the user.

11. The method of claim 10 that includes capturing imagery depicting plural subjects of interest to the user, and presenting results for said subjects in a user interface display of the second device, said user interface display presenting imagery of a first subject adjacent results corresponding to the first subject, and presenting imagery of a second subject adjacent results corresponding to the second subject.

12. The method of claim 10 in which said collection of diverse results includes at least two results drawn from the list:

imagery of the subject different than said captured imagery, a nutrition facts panel, a data table, a news article, and a graph.

13. The method of claim 12 in which said collection of diverse results includes at least three results drawn from said list.

14. A seeing and hearing discovery method comprising the acts:

receiving sensor data corresponding to both audio and imagery, from sensors in a portable apparatus;

based on the sensor data, obtaining audio content identification data identifying a source of the audio, and presenting a first graphic that visually alerts a user to the obtained audio content identification data;

based on the sensor data, obtaining visual content identification data identifying an object depicted in the imagery, and presenting a second graphic that visually alerts the user to the obtained visual content identification data, the second graphic being different than the first graphic;

storing the obtained audio content identification data and the visual content identification data;

receiving user input requesting recall of information about historical discovery activity; and

retrieving both the audio content identification data the visual content identification data as a consequence of said user input, and presenting third and fourth graphics respectively related thereto for user selection;

wherein a single user input results in retrieval of both audio and visual content identification data; and

wherein the fourth graphic is different than the second graphic.

15. A method comprising the acts:

determining, by context, whether a user is interested in audio or visual content; and

controlling notifications about content recognition, based on a result of said determining;

wherein at least one of said acts is performed by a hardware processor configured to perform said act(s).

16. A non-transitory computer readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

presenting on a touch screen of the device a graphical user interface including first, second and third buttons;

in connection with a user tap of the first button, launching an audio discovery, wherein said audio discovery comprises processing audio captured by a microphone of the device, said audio processing yielding data that is sent to a remote computer, which returns first identification data, said first identification data identifying a source of the audio;

in connection with a user tap of the second button, launching an image discovery, wherein said image discovery comprises processing imagery captured by a camera of the device, said image processing yielding data that is sent to a remote computer, which returns second identification data, said second identification data identifying an object depicted in the imagery; and

responsive to a user tap of the third button, recalling the first and second data about previous audio and image discoveries, and presenting associated audio and image discovery information in a scrollable list on said display.

17. The medium of claim 16 in which said operations further include presenting time information in said scrollable list, said time information indicating a time of different discovery events.

18. The medium of claim 17 in which said time information is relative to a present time.

19. The medium of claim 16 in which said scrollable list also presents information about previous discovery of near field or RFID chips.

20. The medium of claim 16 in which the audio processing includes applying a speech classifier to the audio captured by the microphone, and determining whether to send data to a remote music identification computer based on information from the speech classifier.

21. The medium of claim 16 in which the image processing includes determining whether to perform image processing based on data from an accelerometer, gyroscope, or magnetometer in the device.

22. A method comprising:

obtaining data indicating a user'"'"'s location;

by reference to said data, identifying a collection of images that depict retail products near the user;

sending certain of said images to a user device for presentation to the user; and

associating at least one of said images or products together with the user, by data posted to a social network service.