Methods and Systems for Content Processing

• US 20110034176A1
• Filed: 07/13/2010
• Published: 02/10/2011
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: MacIntosh, Brian T., Rhoads, Geoffrey B., Lord, John D., Rodriquez, Tony F.
• Granted Patent: US 8,886,206 B2
• Time in Patent Office: Days
• US Class Current: 455/450
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/72457 according to geographic location
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04W 28/26 Resource reservation

Abstract

Many of the detailed technologies are useful in enabling a smart phone to respond to a user'"'"'s environment, e.g., so it can serve as an intuitive hearing and seeing device. A few of the detailed arrangements involve using radio base station SDR equipment (e.g., at a cell tower) to perform image recognition operations for phones; forecasting service needs from remote processors, and reserving such capacity in advance of use; delegating a remote execution task to a service provider chosen in a competitive process, such as by a reverse auction; using nearby processors, e.g., in an automobile, another phone, or set-top box, for remote execution tasks; phones with separable camera and/or illumination components; phone camera illumination using different colors of light; using search tree methods with image frames captured at different focuses; using a phone'"'"'s microprojector to aid in object identification; correcting lens aberrations by texture mapping captured imagery onto a corrective polygon surface using a phone GPU; etc. A great variety of other features and arrangements are also detailed.

Claims

1. A method of using a portable electronic device that includes a wireless interface and reconfigurable hardware, the method including the acts:

in response to user action, initializing the device to prepare it for use;

said initializing including downloading latest hardware configuration instructions for the hardware from a remote source, through the wireless interface; and

configuring hardware of the device in accordance with the downloaded hardware configuration instructions;

wherein the method dynamically reconfigures the hardware each time the device is prepared for use, said reconfiguration assuring that the device is not initialized with superseded configuration instructions for its programmable hardware.

2. A method comprising:

employing a hardware processor component of a radio system base station to process data related to radio signals exchanged between the base station and plural associated remote wireless devices; and

also using said hardware processor component of the radio system base station to process image-related data offloaded to the radio base station for processing by a consumer camera device.

3. The method of claim 2, wherein the hardware processing component comprises a field programmable object array, and the remote wireless devices comprise cell phones.

4. A method comprising:

characterizing an optical distortion function associated with a lens;

defining geometry of a correction surface corresponding to the optical distortion function; and

texture-mapping an image onto the correction surface using a GPU, after receiving the image through the lens, or before projecting the image through the lens;

wherein tilts and/or elevations in the correction surface serve to compensate for the optical distortion function associated with the lens.

5. A method of correcting lens distortion in an image, characterized by projecting the image onto a virtual surface whose topology is shaped to counteract the lens distortion.

6. A method comprising:

at a radio base station that communicates with plural portable devices, receiving a services reservation message from a first portable device, the services reservation message including one or more parameters of a future service that the first portable device requests be made available to it not immediately, but at a future time; and

making a decision about a service provided to a second portable device, based in part on the services reservation message received from the first portable device;

wherein allocation of processing and/or bandwidth resources of the radio base station is improved due to advance information about anticipated services to be provided to the first portable device.

7. The method of claim 6, wherein the one or more parameters include a parameter indicating a time in the future when the future service is requested to be made available.

8. The method of claim 6, wherein the one or more parameters include a parameter indicating an expected duration of the requested future service.

9. A cell phone including an image sensor with a thermoelectric cooling device coupled thereto, a processor, and a memory, instructions in the memory configuring the processor to control a camera function of the cell phone as follows:

capture image data;

assess noise in the captured image data; and

if the noise exceeds a threshold, activate the thermoelectric cooling device and capture further image data after the sensor has cooled.

10. A cell phone apparatus comprising:

a first portion comprising an optical sensor and lens assembly, the first portion being adapted for carrying by a user in a first location relative to a body of the user; and

a second portion comprising a display, a user interface, and a cellular radio transceiver, the second portion being adapted for carrying by the user in a second, different, location relative to the body of the user; and

a wireless communication system linking the first and second portions.

11. The apparatus of claim 10, wherein the second portion is adapted to detachably receive the first portion.

12. A cell phone apparatus comprising:

a first portion comprising an LED illumination assembly, detachably coupled to;

a second portion comprising a display, a user interface, a cellular radio transceiver, an optical sensor, and a lens; and

a wireless communication system linking the first and second portions;

wherein the first portion can be detached from the second portion and positioned to illuminate a subject being imaged by the optical sensor of the second portion.

13. A method of operating a mobile phone, comprising:

sensing visual stimulus using a sensor in said phone;

transmitting first data, including data related to the sensed stimulus, from the mobile phone for remote signal processing;

responsive to said transmitting, receiving processed second data back at the mobile phone; and

taking an action, at the phone, based on the received second data;

wherein said transmitting comprises transmitting the first data to one of a set-top box, automobile, thermostat, or an HVAC system, for signal processing by a processor therein.

14. A method of operating a first mobile phone, comprising:

offloading a first image processing task from the first mobile phone to a second mobile phone, and collecting results therefrom, the first and second mobile phones being directly coupled across a radio link such as Bluetooth; and

offloading a second image processing task to a third mobile phone, the third mobile phone being different than the first and second mobile phones, and collecting results therefrom; and

at the first mobile phone, taking an action based on said collected results;

wherein the first and third mobile phones do not directly exchange radio signals with each other, but rather the first phone directly exchanges radio signals with the second mobile phone, and the second phone directly exchanges radio signals with the third mobile phone.

15. A method comprising:

(a) using a camera sub-system of a mobile phone, capturing and buffering plural frames of image data, while the camera sub-system is changing a parameter that varies a camera focal distance from a first focal distance to a second focal distance, thereby yielding frames of image data at a variety of focal distance settings;

(b) performing a first analysis on one or more of said frames of image data, to assess a focus parameter;

(c) based on said first analysis, selecting a frame of image data, not analyzed in act (b), as having a desired focus parameter; and

(d) taking a further action with the selected frame of image data.

16. A method comprising:

using a camera sub-system of a mobile phone, capturing a first frame of image data;

assessing the captured image data;

setting a parameter of the camera sub-system based on said assessment; and

capturing a second frame of image data using said set parameter;

wherein the assessing includes assessing the content of the first frame of image data in a particular color channel, and the setting includes setting said parameter to maximize a dynamic range of the second frame of image data in said particular color channel, wherein the particular color channel is selected from blue, yellow, red, and green.

17. A method comprising:

receiving one or more data packets, the packet(s) including first data relating to imagery captured by a mobile phone, and indicating an operation to be performed using said imagery-related data;

after said imagery has been captured by the mobile phone, conducting an automated evaluation of plural different computer-based service providers, to identify which of them is to provide results of said indicated operation on the first data for use by the mobile phone; and

based on said evaluation, sending the first data to a selected one of said computer-based service providers, for it to perform the indicated operation thereon.

18. The method of claim 17 that includes sending results of said indicated operation performed by the selected one of said computer-based service providers to the mobile phone.

19. The method of claim 17 that includes conducting said automated evaluation after receiving said one or more data packets.

20. A method comprising:

receiving one or more packets of data, the packet(s) including first data relating to image content captured by a mobile phone, and also indicating an operation to be performed using the first data;

contacting plural computer-based service providers, to solicit interest in performing the operation, and providing results of said operation for use by the mobile phone on a competitive basis;

receiving responses from one or more of said service providers; and

deciding a winner from among said plural computer-based service providers;

wherein the method further includes providing at least some of the first data to at least some of said plural service providers before deciding the winner.

21. A method of processing image data captured using an image sensor in a cell phone, comprising:

performing a first image processing operation on the captured image data using a processing module in the cell phone, yielding internally-processed image data;

evaluating a competition between two or more external service providers, and sending the internally-processed image data to a winner of said competition for further processing, the winning service provider delivering externally-processed image data; and

taking an action based on the externally-processed image data.

22. The method of claim 21 wherein a module in the cell phone generates plural sets of visual keyvector data, and wherein the method further includes:

routing one of said sets of keyvector data to the winning service provider; and

routing another of said sets of keyvector data to a different external service provider, identified without reference to a competition.

23. The method of claim 21 wherein the competition comprises a reverse auction.

24. A method of operating a camera-equipped mobile phone to image a printed subject, the method comprising the acts:

activating only a first LED light source of a first visible light color, for a first interval, while capturing image data from a printed subject using said mobile phone;

activating only a second LED light source of a second visible light color different than the first, for a second interval different than the first, while capturing image data from said printed subject using said mobile phone; and

taking an action using said captured image data.

25. The method of claim 24 wherein both said activating acts occur during a single frame exposure.

26. The method of claim 24 wherein said activating acts occur during different frame exposures.

27. The method of claim 24 wherein said action comprises decoding machine readable data from the captured image data.

28. A method of operating a mobile phone equipped with a camera and a projector, the method comprising the acts:

capturing first and second frames of image data from a subject using the camera, and illuminating the subject—

during one of said frames—

with a first pattern of collimated light from the projector;

by reference to a difference between said frames, identifying said pattern; and

adjusting a camera parameter by reference to said identified pattern.

29. The method of claim 28 wherein the camera parameter comprises a parameter relating to focal depth.

30. A method comprising the acts:

in a mobile phone, which has an RF output power state that varies with proximity to a base station with which the phone wirelessly communicates, assessing variables including variables relating to (a) said RF transmitter output power state, (b) mobile phone battery state, and (c) network bandwidth state, to determine whether to have a processing operation related to imagery captured by the mobile phone performed by a processor in the mobile phone, or by a processing node at the base station;

transmitting data to the processing node at the base station, if said determination is to have said processing operation performed by said processing node at said base station; and

performing the image processing operation using the processor in the mobile phone, if said determination is to have the image processing operation performed by said mobile device processor.

31. The method of claim 30 in which the base station is at a cell tower site.

32. A method comprising the acts:

in a mobile phone, which has an RF output power state that varies with proximity to a base station with which the phone wirelessly communicates, assessing variables including variables relating to (a) said RF transmitter output power state, (b) mobile phone battery state, and (c) network bandwidth state, in connection with deciding (1) where, and (2) when, to have processing related to imagery captured by the mobile phone performed;

based on said assessing, deciding to have said processing performed (1) by a processor remote from the mobile phone, (2) at a later time, after one of said variables has changed; and

at said later time, after one of said variables has changed, transmitting data relating to said imagery to said remote processor for processing.

33. A method performed by a portable device, and involving a remote processing system to which the portable device refers one or more tasks for remote execution, the method including the acts:

transmitting information from the portable device to the remote processing system, requesting reservation of processing capacity for the remote processing system to meet a forthcoming processing need for the portable device, the information including information characterizing the processing need, and also indicating when—

in the future—

first data for said processing need is expected to be available;

receiving, at the portable device, information confirming reservation of said processing capacity by the remote processing system; and

at a later time, transmitting the first data from the portable device to the remote processing system for processing, in accordance with said reservation.

34. The method of claim 33 wherein the remote processing system is associated with a radio base station with which the portable device wirelessly communicates.

35. The method of claim 33 wherein the remote processing system comprises a cloud of data processing systems.

36. A method performed by a remote processing system, and involving a portable device, the method including the acts:

receiving information from the portable device, requesting reservation of processing capacity for the remote processing system to meet a forthcoming processing need for the portable device, the information including information characterizing the processing need, and also indicating when—

in the future—

first data for said processing need is expected to be available;

sending, to the portable device, information confirming reservation of said processing capacity; and

at a later time, receiving the first data from the portable device, in accordance with said reservation;

wherein advance knowledge of processing to be referred to it by the portable device allows the remote processing system to better allocate its resources.

37. A method performed by a cellular base station that includes a computer system, the method comprising the acts:

for a group of first cellular subscribers, who do not transmit information to the base station about their future service usage intentions, modeling their future service usage by statistics;

from one or more second cellular subscribers, receiving transmitted information about their future service usage intentions; and

allocating services to plural cellular subscribers in accordance with said statistics and the received information.

38. The method of claim 37 that includes the base station providing an improved level of service to the first cellular subscribers, than if the second cellular subscribers also did not transmit information to the base station about their future service usage intentions, due to reduced uncertainty about aggregate usage.

39. A method comprising the acts:

receiving information, including imagery depicting one or more faces, sent from a mobile phone associated with a user;

determining, from the received information, data not included in the received information, said determined data including data enabling access to a first photo repository account, and data enabling access to a second photo repository account;

submitting the imagery to the first photo repository account, using said determined data, and receiving therefrom names associated with any face(s) recognized from the imagery by the first account;

submitting the imagery to the second photo repository account, using said determined data, and receiving therefrom names associated with any faces recognized from the imagery by the second photo repository account—

if any; and

sending data including said received name(s to the mobile phone.

40. The method of claim 39 wherein at least one of said accounts is associated with the user.

41. The method of claim 39 wherein at least the first of said accounts is associated with a social network friend of the user.

42. The method of claim 41 that includes receiving information from the first account identifying a name of a person whose face is recognized by the first account, and also about a relationship between the friend and said person.

43. A computer readable medium containing non-transitory first software causing a processing system in a mobile phone that is programmed thereby to perform the following acts:

provide a layered stack interface between (a) mobile phone hardware, including at least a camera, and (b) application software running on said mobile phone processing system that relies on such hardware, including OCR software and/or barcode reading software; and

provide a layered stack interface between (a) said mobile phone hardware, and (b) application software running on one or more processors remote from the mobile phone that relies on such hardware;

wherein execution of said first software establishes a platform enabling local and remote application software to interact with mobile phone hardware, without resort to specialized software stacks enabling particular application software to interact with said mobile phone hardware.

44. A method practiced using a mobile phone, comprising the acts:

projecting at least one pattern onto a subject from a projector portion included in the mobile phone;

using a camera portion in the mobile phone, capturing image data from the subject including said pattern(s);

by reference to the captured image data including the pattern(s), discerning 3D information about the subject; and

using said 3D information in connection with a recognition operation involving said subject.

45. The method of claim 44 wherein the mobile phone projects two patterns onto the subject, from two projector portions of said mobile phone.

46. The method of claim 44 wherein the mobile phone captures image data from the subject using two camera portions of said mobile phone.