Method and apparatus for displaying screen in response to event related to motion of external object

• US 9,983,681 B2
• Filed: 05/22/2015
• Issued: 05/29/2018
• Est. Priority Date: 10/21/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Park, Keun Joo, Ryu, Eric Hyunsurk, Lee, Kyoobin, Lee, Jun Haeng
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): TRAN, KIM THANH THI
• Application Number:
  US14/719,607
• Publication Number:
  US 20160109955A1
• Time in Patent Office: 1,103 Days
• Field of Search: 345633, 345668, 345670, 345671
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A method and an apparatus for displaying a screen in response to an event related to a motion of an external object, are provided. The method includes generating an event signal in response a motion of an external object being sensed, sensing a movement of the external object relative to an apparatus based on the event signal, and displaying a screen based on the movement of the external object.

Claims

1. A method for displaying a screen, the method being performed by an apparatus, and the method comprising:

sensing, by an event sensor, a motion of an external object that is external to the apparatus, based on a change of an intensity of light incident on the event sensor;

generating, by the event sensor, an event signal, in response to the motion of the external object being sensed;

sensing, by a processor, a moving velocity of the external object relative to the apparatus, based on the event signal;

displaying, by a display, the screen;

controlling, by the processor, the display to move the screen that is displayed, in response to the moving velocity of the external object being sensed to be less than or equal to a threshold velocity; and

controlling, by the processor, the display to magnify a region of the screen, in response to the moving velocity of the external object being sensed to greater than the threshold velocity.

2. The method of claim 1, further comprising determining, by the processor, any one or any combination of a first moving direction, a first moving distance, and a moving velocity of the screen, based on any one or any combination of a second moving direction, a second moving distance, and the moving velocity of the external object.

3. The method of claim 1, further comprising controlling, by the processor, the display to display the screen, based on a number of times that the external object is shaken during a period of time.

4. The method of claim 1, wherein the generating comprises generating a first event signal and a second event signal, in response to the motion of the external object being sensed, and

wherein the sensing comprises:

generating an event map comprising a first map element that comprises a first time at which the first event signal is generated by a first sensing element of the event sensor, and comprising a second map element that comprises a second time at which the second event signal is generated by a second sensing element of the event sensor;

determining a time difference between the first time and the second time and a distance difference between a first position of the first map element and a second position of the second map element;

determining an optical flow comprising a velocity vector, based on the time difference and the distance difference; and

sensing the moving velocity of the external object, based on the optical flow.

5. The method of claim 1, wherein the sensing comprises:

generating an event frame by storing the event signal during a time window;

extracting a region of interest (ROI) from the event frame;

determining an optical flow corresponding to the ROI; and

sensing the moving velocity of the external object, based on the optical flow.

6. The method of claim 1, wherein the sensing comprises:

generating an event frame by storing the event signal during a time window;

extracting a region of interest (ROI) from the event frame; and

sensing the moving velocity of the external object, based on a movement of the ROI.

7. The method of claim 1, further comprising:

sensing, by an acceleration sensor, an acceleration of the apparatus;

sensing, by a tilt sensor, a tilt of the apparatus; and

controlling, by the processor, the display to display the screen, based on the acceleration, the tilt, and the moving velocity of the external object.

8. A non-transitory computer-readable recording medium that stores a program that is executable by a computer to perform the method of claim 1.

9. An apparatus for displaying a screen, the apparatus comprising:

an event sensor configured to;

sense a motion of an external object that is external to the apparatus, based on a change of an intensity of light incident on the event sensor; and

generate an event signal, in response to the motion of the external object being sensed;

a processor configured to sense a moving velocity of the external object relative to the apparatus, based on the event signal; and

a display configured to display the screen,

wherein the processor is further configured to;

control the display to move the screen that is displayed, in response to the moving velocity of the external object being sensed to be less than or equal to a threshold velocity; and

control the display to magnify a region of the screen, in response to the moving velocity of the external object being sensed to greater than the threshold velocity.

10. The apparatus of claim 9, wherein the processor is further configured to control the display to display the screen, based on a number of times that the external object is shaken during a period of time.

11. The apparatus of claim 9, wherein the event sensor comprises a first sensing element configured to generate a first event signal, in response to the motion of the external object being sensed, and comprises a second sensing element configured to generate a second event signal, in response to the motion of the external object being sensed, and

wherein the processor is further configured to:

generate an event map comprising a first map element that comprises a first time at which the first event signal is generated by the first sensing element, and comprising a second map element that comprises a second time at which the second event signal is generated by the second sensing element;

determine a time difference between the first time and the second time and a distance difference between a first position of the first map element and a second position of the second map element;

determine an optical flow comprising a velocity vector, based on the time difference and the distance difference; and

sense the moving velocity of the external object, based on the optical flow.

12. The apparatus of claim 9, wherein the processor is further configured to:

generate an event frame by storing the event signal during a time window;

extract a region of interest (ROI) from the event frame;

determine an optical flow corresponding to the ROI; and

sense the moving velocity of the external object, based on the optical flow.

13. The apparatus of claim 9, wherein the processor is further configured to:

generate an event frame by storing the event signal during a time window;

extract a region of interest (ROI) from the event frame; and

sense the moving velocity of the external object, based on a movement of the ROI.

14. The apparatus of claim 9, further comprising:

an acceleration sensor configured to sense an acceleration of the apparatus; and

a tilt sensor configured to sense a tilt of the apparatus,

wherein the processor is further configured to control the display to display the screen, based on the acceleration, the tilt, and the moving velocity of the external object.

15. The apparatus of claim 9, wherein the display is a head mounted display configured to be mounted on a head of a user to display the screen.

16. A method for displaying a screen, the method being performed by an apparatus, and the method comprising:

generating a first event signal and a second event signal, in response to a motion of an external object being sensed;

generating an event map comprising a first map element that comprises a first time at which the first event signal is generated by a first sensing element of an event sensor, and comprising a second map element that comprises a second time at which the second event signal is generated by a second sensing element of the event sensor;

determining a time difference between the first time and the second time and a distance difference between a first position of the first map element and a second position of the second map element;

determining an optical flow comprising a velocity vector, based on the time difference and the distance difference;

sensing a moving velocity of the external object relative to the apparatus, based on the optical flow;

displaying, by a display, the screen; and

controlling the display to move the screen that is displayed, in response to the moving velocity of the external object being sensed to be less than or equal to a threshold velocity.