Image-based motion sensor and related multi-purpose camera system

• US 9,609,217 B2
• Filed: 09/12/2012
• Issued: 03/28/2017
• Est. Priority Date: 11/02/2011
• Status: Active Grant

• Current Assignee: Xueshan Technologies Inc. (Wi-LAN Inc.)
• Original Assignee: MediaTek, Inc.
• Inventors: Ju, Chi-Cheng, Ho, Cheng-Tsai, Chen, Ding-Yun
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US13/610,890
• Publication Number:
  US 20130147974A1
• Time in Patent Office: 1,658 Days
• Field of Search: 348/208.1
• US Class Current: 1/1
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/95 Computational photography systems, e.g. light-field imaging systems

Abstract

An image-based motion sensor includes a camera system and a processing system. The camera system is arranged for generating an image output including a plurality of captured images. The processing system is arranged for obtaining a motion sensor output by processing the image output, wherein the motion sensor output includes information indicative of at least one of a motion status and an orientation status of the image-based motion sensor. For example, information indicative of the orientation status of the image-based motion sensor may includes roll, pitch, and yaw related motion information, and information indicative of the motion status of the image-based motion sensor may includes speed, acceleration and displacement information.

Claims

1. An image-based motion sensor, comprising:

a camera system, arranged for generating an image output including a plurality of captured images; and

a processing system, arranged for obtaining a motion sensor output by processing the image output, and identifying a user input as one of a plurality of pre-defined user actions according to the motion sensor output, wherein different functions of at least one application performed by one electronic device are controlled by the pre-defined user actions; and

the motion sensor output includes information indicative of at least one of a motion status and an orientation status of the image-based motion sensor;

wherein the camera system performs an RGB-to-YUV color transform upon one color image to generate one grayscale image, and only luminance values derived from the RGB-to-YUV color transform are involved in obtaining the motion sensor result.

2. The image-based motion sensor of claim 1, wherein the motion sensor output includes information indicative of the orientation status of the image-based motion sensor, and the information indicative of the orientation status includes roll, pitch, and yaw related motion information.

3. The image-based motion sensor of claim 1, wherein the motion sensor output includes information indicative of the motion status of the image-based motion sensor, and the information indicative of the motion status includes speed, acceleration and displacement information.

4. The image-based motion sensor of claim 1, wherein the processing system comprises:

a global motion estimation block, arranged for generating global motion information according to the image output; and

a processor, arranged for determining the motion sensor output according to the global motion information.

5. The image-based motion sensor of claim 4, wherein the global motion estimation block includes an electronic image stabilization (EIS) module arranged to perform global motion estimation.

6. The image-based motion sensor of claim 4, wherein the global motion estimation block comprises:

a motion estimation unit, arranged for performing motion estimation upon each of a plurality of pairs of co-located sub-images in a first captured image and a second captured image, and generating a plurality of motion vectors of the pairs of co-located sub-images, wherein the first captured image and the second captured image are consecutive captured images; and

a motion vector analysis unit, arranged for generating the global motion information by analyzing the motion vectors.

7. The image-based motion sensor of claim 4, wherein the global motion estimation block comprises:

an image down-scaling unit, arranged for down-scaling a first captured image and a second captured image to generate a first down-scaled image and a second down-scaled image, respectively, wherein the first captured image and the second captured image are consecutive captured images; and

a global motion estimation unit, arranged for generating the global motion information by performing global motion estimation upon the first down-scaled image and the second down-scaled image.

8. The image-based motion sensor of claim 4, wherein the global motion estimation block comprises:

an image down-scaling unit, arranged for generating a first down-scaled image of a first captured image and a second down-scaled image of a second captured image, wherein the first captured image and the second captured image are consecutive captured images;

a transform unit, arranged for performing a specific transform upon the first down-scaled image and the second down-scaled image to generate a first transformed image and a second transformed image, respectively; and

a global motion estimation unit, arranged for generating the global motion information by performing global motion estimation according to the first transformed image and the second transformed image.

9. The image-based motion sensor of claim 8, wherein the specific transform is a linear polar transform or a Log-polar transform.

10. The image-based motion sensor of claim 9, wherein the global motion estimation unit is further arranged for generating another global motion information by performing global motion estimation upon a third down-scaled image and a fourth down-scaled image;

and the image down-scaling unit comprises;

an image down-scaling circuit, arranged for down-scaling the first captured image and the second captured image to generate the third down-scaled image and the fourth down-scaled image, respectively; and

an image alignment circuit, arranged for generating the first down-scaled image and the second down-scaled image by referring to the another global motion information to perform image alignment upon the third down-scaled image and the fourth down-scaled image.

11. The image-based motion sensor of claim 8, wherein the global motion estimation unit performs the global motion estimation in one direction only.

12. The image-based motion sensor of claim 1, wherein the processing system comprises:

an edge detection block, arranged for generating global motion information by performing edge detection according to the image output; and

a processor, arranged for determining the motion sensor output according to the global motion information.

13. The image-based motion sensor of claim 12, wherein the edge detection block comprises:

an image down-scaling unit, arranged for down-scaling a first captured image and a second captured image to generate a first down-scaled image and a second down-scaled image, respectively, wherein the first captured image and the second captured image are consecutive captured images;

an edge detection unit, arranged for performing edge detection upon the first down-scaled image to generate first edge information, and performing edge detection upon the second down-scaled image to generate second edge information; and

an edge histogram unit, arranged for deriving a first edge histogram from the first edge information; and

deriving a second edge histogram from the second edge information; and

an edge histogram matching unit, arranged for generating the global motion information by performing edge histogram matching upon the first edge histogram and the second edge histogram.

14. The image-based motion sensor of claim 13, wherein the edge histogram unit is further arranged for performing a filtering operation upon the first edge histogram and the second edge histogram before transmitting the first edge histogram and the second edge histogram to the edge histogram matching unit.

15. The image-based motion sensor of claim 1, wherein the image-based motion sensor is disposed in a first electronic device external to a second electronic device, the processing system is further arranged to generate a control signal to the second electronic device according to the motion sensor output.

16. A multi-purpose camera system, comprising:

an image capture block, arranged for generating an image signal; and

an image signal processing block, arranged for processing the image signal;

wherein when the multi-purpose camera system is operated in a first operation mode, the multi-purpose camera system acts as a camera for generating a captured image output; and

when the multi-purpose camera system is operated in a second operation mode, the multi-purpose camera system acts as part of a motion sensor for generating a motion sensor output, wherein the motion sensor output includes information indicative of at least one of a motion status and an orientation status of the multi-purpose camera system, and is transmitted to a processing system arranged to identify a user input as one of a plurality of pre-defined user actions according to the motion sensor output;

different functions of at least one application performed by one electronic device are controlled by the pre-defined user actions;

wherein the image signal processing block performs an RGB-to-YUV color transform upon one color image to generate one grayscale image in the second operation mode, and only luminance values derived from the RGB-to-YUV color transform are involved in generating the motion sensor output.

17. The multi-purpose camera system of claim 16, wherein the image capture block comprises an image sensor, an optical system, and a control circuit, and an overall configuration of the control circuit and the image signal processing block in the first operation mode is different from an overall configuration of the control circuit and the image signal processing block in the second operation mode.

18. The multi-purpose camera system of claim 17, wherein power consumption of the image signal processing block in the second operation mode is lower than power consumption of the image signal processing block in the first operation mode.

19. The multi-purpose camera system of claim 17, wherein one signal processing module included in the image signal processing block is enabled in the first operation mode, and is disabled in the second operation mode.

20. The multi-purpose camera system of claim 17, wherein one signal processing module included in the image signal processing block employs a first algorithm in the first operation mode, and employs a second algorithm different from the first algorithm in the second operation mode.

21. The multi-purpose camera system of claim 17, wherein power consumption of the control circuit in the second operation mode is lower than power consumption of the control circuit in the first operation mode.

22. The multi-purpose camera system of claim 17, wherein one control module included in the control circuit employs a first algorithm in the first operation mode, and employs a second algorithm different from the first algorithm in the second operation mode.

23. The multi-purpose camera system of claim 16, wherein when the multi-purpose camera system acts as part of the motion sensor, the image capture block employs a pre-defined and fixed focus parameter setting.

24. The multi-purpose camera system of claim 16, wherein when the multi-purpose camera system acts as part of the user input apparatus, the image capture block employs a pre-defined and fixed exposure parameter setting.

25. The multi-purpose camera system of claim 16, wherein a white balance module included in the image signal processing block is enabled in the first operation mode, and is disabled in the second operation mode.

26. The multi-purpose camera system of claim 16, wherein the image capture block operates at a first capture rate in the first operation mode, and operates at a second capture rate higher than the first capture rate in the second operation mode.

27. The multi-purpose camera system of claim 16, wherein the image capture block operates at a first capture resolution in the first operation mode, and operates at a second capture resolution lower than the first capture resolution in the second operation mode.

28. An electronic device, comprising:

a multi-purpose camera system, comprising;

an image capture block, arranged for generating an image signal; and

an image signal processing block, arranged for processing the image signal, wherein when the multi-purpose camera system is operated in a first operation mode, the multi-purpose camera system acts as a camera for generating a captured image output, and when the multi-purpose camera system is operated in a second operation mode, the multi-purpose camera system acts as part of a motion sensor; and

a processing system, acting as another part of the motion sensor and arranged for generating a motion sensor output by processing the image output, and identifying a user input as one of a plurality of pre-defined user actions according to the motion sensor output, wherein the motion sensor output includes information indicative of at least one of a motion status and an orientation status of the electronic device;

different functions of at least one application performed by one electronic device are controlled by the pre-defined user actions;

wherein the image signal processing block performs an RGB-to-YUV color transform upon one color image to generate one grayscale image in the second operation mode, and only luminance values derived from the RGB-to-YUV color transform are involved in generating the motion sensor output.

29. The electronic device of claim 28, being a portable electronic device.