Mobile augmented reality video editing system

• US 10,643,662 B1
• Filed: 01/16/2019
• Issued: 05/05/2020
• Est. Priority Date: 12/12/2018
• Status: Active Grant

• Current Assignee: HY3D Co., Ltd.
• Original Assignee: HY3D Co., Ltd.
• Inventors: Lee, Jun Ho, Ko, Hong Sik, Lee, Jeong Uk, Yang, Yoon Young, Park, Sang Yeon, Lee, Se Yoon
• Primary Examiner(s): Topgyal, Gelek W
• Application Number:
  US16/249,227
• Time in Patent Office: 475 Days
• Field of Search: None
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Provided is a mobile augmented reality video editing system including a real image input section configured to receive a real image through an image sensor, a spatial recognizer configured to separately recognize the ground surface and a space within the real image, an object extraction and display section configured to extract and display objects in the real image, an editing tool display section configured to display video editing tools to a user, a video processor configured to delete an object touched for deletion by the user among the extracted and displayed objects in the real image and correct horizontal lines constituting a region of the deleted object by substituting each of the horizontal lines with a pixel value of the same height in an adjacent space, and a screen display section configured to display the real image corrected by the video processor.

Claims

1. A mobile augmented reality video editing system comprising:

a real image input section receiving a real image through an image sensor;

a spatial recognizer separately recognizing a ground surface and a space within the real image;

an object extraction and display section extracting and displaying objects in the real image;

an editing tool display section displaying video editing tools to a user;

a video processor deleting an object touched for deletion by the user among the extracted and displayed objects in the real image and correct horizontal lines constituting a region of the deleted object by substituting each of the horizontal lines with a pixel value of a height identical to that of the horizontal line in an adjacent space; and

a screen display section displaying the real image corrected by the video processor,

wherein the spatial recognizer detects edges in the real image, generates relative coordinates between feature points constituting an edge on the basis of the location of a mobile device, generates planes using a reference point and two of the generated relative coordinates and recognizes a plane which is parallel to the ground surface among the generated planes.

2. The mobile augmented reality video editing system according to claim 1,

wherein the video processor deletes an object touched for movement by the user among the extracted and displayed objects from the real image, corrects horizontal lines constituting a region of the deleted object by substituting each of the horizontal lines with a pixel value of a height identical to a pixel value of the horizontal line in an adjacent space, adjusts a scale of the object touched for movement using three-dimensional (3D) coordinate values of a pixel corresponding to a position designated for object movement, and thereby generates a corrected video in which the object has been moved.

3. The mobile augmented reality video editing system according to claim 1,

further comprising an object storage matching an object, which is touched for storage by the user, among the extracted and displayed objects to an input filename and store the object in a storage,

wherein the video processor adjusts a scale of an object selected from the storage by the user with 3D coordinate values of a pixel corresponding to a position designated for object composition, composites the selected object into the real image, and thereby generates a corrected video into which the object has been composited.

4. The mobile augmented reality video editing system according to claim 3,

wherein, in addition to the real image input section, the spatial recognizer, the object extraction and display section, the editing tool display section, the video processor, and the screen display section, the object storage is a set of program code data constituting an app which is able to be downloaded to a memory of a user mobile device and executed.

5. The mobile augmented reality video editing system according to claim 1, further comprising:

an object transmitter transmitting an object touched for augmented reality by the user among the extracted and displayed objects to an interoperable external device via a communication network; and

a service server comparing feature points of the object touched for augmented reality with a machine learning object database (DB), read a representative 3D object having a highest concordance rate with the object from the machine learning object DB, and transmit the representative 3D object to the video processor.

6. The mobile augmented reality video editing system according to claim 5,

wherein the video processor generates a corrected video, whose scale has been adjusted according to a scale value of the object touched for augmented reality with respect to a screen, by compositing the representative 3D object into the real image.

7. The mobile augmented reality video editing system according to claim 6,

wherein the video processor generates a corrected video in which the representative 3D object has been adjusted in size, position, and color according to a user command from the video editing tools.

8. The mobile augmented reality video editing system according to claim 5,

wherein, in addition to the real image input section, the spatial recognizer, the object extraction and display section, the editing tool display section, the video processor, and the screen display section, the object transmitter is a set of program code data constituting an app which is able to be downloaded to a memory of a user mobile device and executed.

9. The mobile augmented reality video editing system according to any one of claim 1,

wherein each of the real image input section, the spatial recognizer, the object extraction and display section, the editing tool display section, the video processor, and the screen display section is a set of program code data constituting an application (app) which is able to be downloaded to a memory of a user mobile device and executed.