Image movement based device control method, program, and apparatus

• US 8,094,204 B2
• Filed: 08/23/2007
• Issued: 01/10/2012
• Est. Priority Date: 08/28/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tsurumi, Shingo
• Primary Examiner(s): SELBY, GEVELL V
• Application Number:
  US11/844,191
• Publication Number:
  US 20080050035A1
• Time in Patent Office: 1,601 Days
• Field of Search: 348/207.1, 348/208.1, 348/208.3, 348/222.1, 348/333.01
• US Class Current: 348/222.1
• CPC Class Codes: G06F 3/0486 Drag-and-drop

Abstract

An information processing apparatus of the present invention includes a shot image obtaining portion, an image processing portion, a data conversion portion and a control portion. The shot image obtaining portion includes an imaging element and obtains a shot image in a time-series from the imaging element. The image processing portion performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the information processing apparatus. The data conversion portion converts the analyzed movement of the shot image into control data, and the control portion controls functions based on the control data.

Claims

1. An information processing apparatus comprising:

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the information processing apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data; and

a control portion that controls functions based on the control data,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein the image processing portion includes;

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine matrix selection portion that computes the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein the movement of the shot image in the time-series is analyzed based on the affine matrix that is selected by the affine matrix selection portion.

2. The information processing apparatus according to claim 1, wherein

the information processing portion includes a movement classification portion that classifies the movement of the shot image into at least one classification type including movement in a direction orthogonal to an optical axis of an optical system that images a target object on the imaging element, rotation around the optical axis, and enlargement/contraction, and

the data conversion portion converts the movement of the analyzed shot image into the control data such that the control data differs depending on the classification type of the movement of the shot image.

3. The information processing apparatus according to claim 1, further comprising:

a switch that is operable by a user,

wherein the data conversion portion variably changes the control data based on an operating state of the switch.

4. The information processing apparatus according to claim 1, further comprising:

a display portion that displays information,

wherein the control portion controls operation of a pointer displayed on the display portion based on the control data.

5. The information processing apparatus according to claim 1, further comprising:

a display portion that displays information,

wherein the control portion controls scrolling of a screen displayed on the display portion based on the control data.

6. The information processing apparatus according to claim 1, wherein

the control portion executes a direct command including starting up and shutting down the information processing apparatus based on the control data.

18. The information processing apparatus according to claim 1, wherein deriving a time-series of movement of the feature points is performed by analyzing the shot image in a plurality of increasing resolutions, by first deriving the time-series of movement in a lower resolution and further refining the time-series of movement in a higher resolution.

21. The information processing apparatus of claim 1, wherein the affine matrix selection portion computes a plurality of affine matrix scores corresponding to the plurality of affine matrixes, and further selects an affine matrix of the plurality of affine matrixes for which the corresponding affine matrix score is the highest.

7. An information processing apparatus comprising:

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the information processing apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data;

a control portion that controls functions based on the control data; and

a display portion that displays information, wherein a painting tool that indicates a painting position is displayed at a determined position on the display portion, and

the control portion uses the control data as a basis for moving one of the painting tool and a background screen displayed on the display portion such that paint is applied to the background screen at the position of the painting tool.

8. An information processing apparatus comprising:

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the information processing apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data;

a control portion that controls functions based on the control data;

a movement information attachment portion that attaches movement information to each one of a plurality of the shot images obtained in the time-series, the movement information being related to the movement of the shot images that is analyzed by the image processing portion; and

an image synthesis portion that synthesizes the plurality of shot images based on the movement information related to the movement of the shot images.

9. An information processing apparatus that is connected to an imaging apparatus by one of a wired and a wireless connection, comprising:

a shot image receiving portion that receives a shot image obtained in time-series from the imaging apparatus;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the imaging apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data; and

a control portion that controls functions based on the control data,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein the image processing portion includes;

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine matrix selection portion that computes the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein the movement of the shot image in the time-series is analyzed based on the affine matrix that is selected by the affine matrix selection portion.

10. The information processing apparatus according to claim 9, further comprising:

a display portion that displays information,

wherein the control portion controls operation of a pointer displayed on the display portion based on the control data.

11. The information processing apparatus according to claim 9, further comprising:

a display portion that displays information,

wherein the control portion controls scrolling of a screen displayed on the display portion based on the control data.

12. An imaging apparatus that is connected to an information processing apparatus by one of a wired and a wireless connection, comprising:

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the imaging apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data; and

a transmission portion that transmits the control data to the information processing apparatus for use in controlling functions of the information processing apparatus,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein the image processing portion includes;

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine matrix selection portion that computes the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein the movement of the shot image in the time-series is analyzed based on the affine matrix that is selected by the affine matrix selection portion.

13. An information processing system comprising an imaging apparatus and an information processing apparatus, the imaging apparatus connected to the information processing apparatus by one of a wired and a wireless connection, wherein

the imaging apparatus includes

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element,

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the imaging apparatus,

a data conversion portion that converts the analyzed movement of the shot image into control data, and

a transmission portion that transmits the control data to the information processing apparatus for use in controlling functions of the information processing apparatus, and

the information processing apparatus includes

a receiving portion that receives the control data from the imaging apparatus, and

a control portion that controls functions based on the control data,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein the image processing portion further includes:

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine matrix selection portion that computes the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein the movement of the shot image in the time-series is analyzed based on the affine matrix that is selected by the affine matrix selection portion.

14. An information processing system comprising an imaging apparatus and an information processing apparatus, the imaging apparatus connected to the information processing apparatus by one of a wired and a wireless connection, wherein

the imaging apparatus includes

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element, and

a shot image transmission portion that transmits the shot image to the information processing apparatus, and

the information processing apparatus includes

a shot image receiving portion that receives the shot image,

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the imaging apparatus,

a data conversion portion that converts the analyzed movement of the shot image into control data, and

a control portion that controls functions based on the control data,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein the image processing portion further includes:

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine matrix selection portion that computes the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein the movement of the shot image in the time-series is analyzed based on the affine matrix that is selected by the affine matrix selection portion.

15. An apparatus control method comprising the steps of:

obtaining a shot image in a time-series from an imaging element;

performing image processing on the shot image and analyzing movement of the shot image resulting from movement of an apparatus that shot the shot image;

converting the analyzed movement of the shot image into control data; and

controlling at least one of functions of the apparatus and functions of another apparatus connected to the apparatus based on the control data,

wherein analyzing the movement of the time-series of the shot image includes setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein performing images processing further includes;

computing a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

computing the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein analyzing the movement of the time-series of the shot image is based on the affine matrix that is selected by the affine matrix selection portion.

16. A non-transitory computer readable storage medium having encoded thereon a computer program comprising a set of instructions when executed by a computer to implement a method comprising instructions that command a computer to function as:

an obtaining portion that obtains a shot image in a time-series from an imaging element;

a processing-analysis portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of an apparatus including the imaging element;

a conversion portion that converts the analyzed movement of the shot image into control data; and

a control portion that controls functions based on the control data,

wherein analyzing the movement of the time-series of the shot image includes setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein performing images processing further includes;

computing a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

computing the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the affine matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein analyzing the movement of the time-series of the shot image is based on the affine matrix that is selected by the affine matrix selection portion.

17. A non-transitory computer readable storage medium having encoded thereon a computer program comprising a set of instructions when executed by a computer to implement a method comprising instructions that command a computer to function as:

a receiving portion that receives a shot image obtained in a time-series from an imaging apparatus;

a processing-analysis portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the imaging apparatus;

a conversion portion that converts the analyzed movement of the shot image into control data; and

a control portion that controls functions based on the control data,

wherein analyzing the movement of the time-series of the shot image includes setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing,

wherein performing images processing further includes;

computing a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

computing the number of feature points which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value, and selects the ante matrix that has the most feature points for which the difference is equal to or less than the determined threshold value,

and wherein analyzing the movement of the time-series of the shot image is based on the affine matrix that is selected by the affine matrix selection portion.

19. An information processing apparatus comprising:

a shot image obtaining portion, including an imaging element, that obtains a shot image in a time-series from the imaging element;

an image processing portion that performs image processing on the shot image and analyzes movement of the shot image resulting from movement of the information processing apparatus;

a data conversion portion that converts the analyzed movement of the shot image into control data;

a control portion that controls functions based on the control data;

a display portion that displays a displayed object; and

a drag-and-drop switch that is operable by a user for enabling a drag and drop function,

wherein the control portion uses the control data as a basis for implementing the drag and drop function on the displayed object, such that an object is picked up when the drag-and-drop switch is pressed, the picked up object is moved while the drag-and-drop switch is held pressed, and the moved object is dropped when the drag-and-drop switch is released,

wherein the image processing portion analyzes the movement of the time-series of the shot image by setting a plurality of feature points in the shot image, and deriving a time-series of movement of the feature points using optical flow processing, and wherein the image processing portion includes;

an affine matrix computation portion that computes a plurality of affine matrixes that represent the relationship of positions of feature points in a present frame and positions of feature points in a frame one before using feature points selected at random, and

an affine, matrix selection portion that computes a, plurality of affine matrix scores corresponding to the plurality of affine matrixes, and further selects an affine matrix of the plurality of affine matrixes for which the corresponding affine matrix score is the highest,

and wherein the image processing portion analyzes the movement of the shot image in the time-series based on the affine matrix that is selected by the affine matrix selection portion.

20. The information processing apparatus of claim 19, wherein the control portion enables the drag and drop function solely by operating the drag-and-drop switch without operating any other switch.

22. The information processing apparatus of claim 19, wherein computing the plurality of affine matrix scores includes computing the number of feature points for which the difference in the positions of the feature points in the present frame and the feature points obtained by multiplication of the positions of feature points in the frame one before and the affine matrix is equal to or less than a determined threshold value.