Methods and apparatus for capturing images using multiple camera modules in an efficient manner

• US 10,075,651 B2
• Filed: 04/22/2015
• Issued: 09/11/2018
• Est. Priority Date: 04/17/2015
• Status: Active Grant

• Current Assignee: Blue River Technology Inc. (Deere & Company)
• Original Assignee: Light Labs Inc.
• Inventors: Laroia, Rajiv, Shroff, Sapna A.
• Primary Examiner(s): Adams, Eileen M
• Application Number:
  US14/693,351
• Publication Number:
  US 20160309095A1
• Time in Patent Office: 1,238 Days
• Field of Search: None
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors

Abstract

Methods and apparatus for capturing images using a camera device including multiple camera modules, e.g., multiple optical chains, in an efficient manner are described. Different camera devices in the camera device may capture images corresponding to different size fields of view of a scene area. User input is used to identify an object of interest. A first type camera module having a large field of view is used to track the identified object of interest. Sets of second type camera modules having a smaller field of view are used to capture images of the object of interest. Different sets of second type camera modules may be used at different times as a function of object of interest location. In some embodiments, at least some second type camera modules include the capability to adjust, e.g., move, image capture area, e.g., by changing a mirror angle inclination and/or a mirror rotation.

Claims

1. A method of operating a camera device comprising:

receiving user input indicating an object of interest, said object of interest being at a first position when user input indicating the object of interest is received;

identifying, based on the received user input, a first set of one or more first type camera modules which should be used to capture an image of the object of interest, said identified first set of one or more first type camera modules including a first camera module including a first movable mirror, said step of identifying a first set of one or more first type camera modules including identifying said first camera module based on the field of view of said first camera module, said first camera module having a field of view which includes said first position or which can be adjusted to include said first position;

tracking the position of the object of interest;

controlling a position of said first moveable mirror of said first camera module to change based on the tracked position of the object of interest, controlling the position of said first moveable mirror including changing an angle of inclination of said first moveable mirror and rotating said first moveable mirror; and

operating the first camera module to capture an image of the object of interest; and

wherein said first set of one or more first type camera modules includes a plurality of first type camera modules, each camera module of said plurality of first type camera modules being a camera module with a single moveable mirror that can be rotated and tilted.

2. The method of claim 1, further comprising:

receiving user input indicating a size of an area surrounding the object of interest to be captured; and

wherein identifying, based on the received user input, said first set of one or more first type camera modules which should be used to capture an image of the object of interest is further based on the size of the area surrounding the object of interest to be captured.

3. A method of operating a camera device comprising:

receiving user input indicating an object of interest;

identifying, based on the received user input, a first set of one or more first type camera modules which should be used to capture an image of the object of interest, said identified first set of one or more first type camera modules including a first camera module including a first movable mirror;

tracking the position of the object of interest;

controlling a position of said first moveable mirror of said first camera module to change based on the tracked position of the object of interest, controlling the position of said first moveable mirror including changing an angle of inclination of said first moveable mirror and rotating said first moveable mirror; and

operating the first camera module to capture an image of the object of interest;

operating a second type camera module having a first field of view which is larger than a field of view of said first type camera modules to capture images of a scene area;

detecting, based on images captured by the second type camera module a move of the object of interest from the first position to a new position; and

based on the new position, identifying a second set of one or more first type camera modules to be used to capture an image of the object of interest.

4. The method of claim 3, wherein identifying said second set of one or more first type camera modules includes identifying one or more first type camera modules having a field of view including the new position of the object of interest said second set of one or more first type camera modules including a second camera module.

5. The method of claim 4, further comprising:

controlling a position of a second moveable mirror, said second moveable mirror being a moveable mirror in said second camera module, controlling the position of the second moveable mirror including moving the second moveable mirror based on the tracked change in the position of the object of interest to keep the object of interest in the field of view of said second camera module for a period of time.

6. The method of claim 5, wherein controlling the position of the second moveable mirror includes changing an angle of inclination of said second moveable mirror.

7. The method of claim 5, wherein controlling the position of the second moveable mirror includes changing an angle of inclination of said second moveable mirror and rotating said second moveable mirror to point the second moveable mirror in a different direction than a direction used to capture an image of the object of interest at the start of said period of time.

8. A camera device comprising:

a plurality of camera modules;

a module configured to receive user input indicating an object of interest, said object of interest being at a first position when user input indicating the object of interest is received;

a module configured to identify, based on the received user input, a first set of one or more first type camera modules which should be used to capture an image of the object of interest, said identified first set of one or more first type camera modules including a first camera module including a first moveable mirror, said module configured to identify said first set of one or more first type camera modules including a module configured to identify said first camera module based on the field of view of the first camera module, said first camera module having a field of view which includes said first position or which can be adjusted to include said first position;

a module configured to track the position of the object of interest;

a module configured to control a position of said first moveable mirror of said first camera module to change based on the tracked position of the object of interest, controlling the position of the first moveable mirror including changing an angle of inclination of said first moveable mirror and rotating said first moveable mirror; and

a module configured to control the first camera module to capture an image of the object of interest; and

wherein said first set of one or more first type camera modules includes a plurality of first type camera modules, each camera module of said plurality of first type camera modules being a camera module with a single moveable mirror that can be rotated and tilted.

9. The camera device of claim 8, further comprising:

a module configured to receive user input indicating a size of an area surrounding the object of interest to be captured; and

wherein said module configured to identify said first set of one or more first type camera modules which should be used to capture an image of the object of interest is further configured to base the identification on the size of the area surrounding the object of interest to be captured.

10. A camera device comprising:

a plurality of camera modules;

a module configured to receive user input indicating an object of interest, said object of interest being at a first position when user input indicating the object of interest is received;

a module configured to identify, based on the received user input, a first set of one or more first type camera modules which should be used to capture an image of the object of interest, said identified first set of one or more first type camera modules including a first camera module including a first moveable mirror, said module configured to identify said first set of one or more first type camera modules including a module configured to identify said first camera module based on the field of view of the first camera module, said first camera module having a field of view which includes said first position or which can be adjusted to include said first position;

a module configured to track the position of the object of interest;

a module configured to control a position of said first moveable mirror of said first camera module to change based on the tracked position of the object of interest, controlling the position of the first moveable mirror including changing an angle of inclination of said first moveable mirror and rotating said first moveable mirror; and

a module configured to control the first camera module to capture an image of the object of interest;

a module configured to control a second type camera module, having a first field of view which is larger than a field of view of said first type camera modules, to capture images of a scene area;

a module configured to detect, based on images captured by the second type camera module a move of the object of interest from the first position to a new position; and

a module configured to identify, based on the new position, a second set of one or more first type camera modules to be used to capture an image of the object of interest.

11. The camera device of claim 10, wherein said module configured to identify said second set of one or more first type camera modules includes:

a module configured to identify one or more first type camera modules having a field of view including the new position of the object of interest, said identified one or more first type camera modules having a field of view including the new position of the object of interest including a second camera module; and

a module configured to include said second camera module in said second set of one or more first type camera modules.

12. The camera device of claim 11, further comprising:

a module configured control a position of a second moveable mirror, said second moveable mirror being a moveable mirror in said second camera module, to move based on the tracked change in the position of the object of interest to keep the object of interest in the field of view of said second camera module for a period of time.

13. The camera device of claim 12, wherein said module configured to control the position of the second moveable mirror includes:

a module configured to change an angle of inclination of said second moveable mirror.

14. The camera device of claim 12, wherein said module configured to control the position of the second moveable mirror includes:

a module configured to change an angle of inclination of said second moveable mirror and a rotation of said second moveable mirror to point the second moveable mirror in a different direction than a direction used to capture an image of the object of interest at the start of the period of time.

15. A non-transitory machine readable medium including processor executable instructions which when executed by a processor of a camera device, control the camera device to perform the steps of:

receiving user input indicating an object of interest;

identifying, based on the received user input, a first set of one or more first type camera modules which should be used to capture an image of the object of interest, said identified first set of one or more first type camera modules including a first camera module including a first movable mirror;

tracking the position of the object of interest;

controlling a position of said first moveable mirror of said first camera module to change based on the tracked position of the object of interest, controlling the position of said first moveable mirror including changing an angle of inclination of said first moveable mirror and rotating said first moveable mirror; and

operating the first camera module to capture an image of the object of interest;

operating a second type camera module having a first field of view which is larger than a field of view of said first type camera modules to capture images of a scene area;

detecting, based on images captured by the second type camera module a move of the object of interest from the first position to a new position; and

identifying, based on the new position, a second set of one or more first type camera modules to be used to capture an image of the object of interest.