Refrigerator with camera and control method for the same

• US 9,915,471 B2
• Filed: 04/22/2014
• Issued: 03/13/2018
• Est. Priority Date: 04/23/2013
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Kim, Kwanhyung, Hong, Samnyol, Kwon, Sungdu, Lee, Ikkyu, Lee, Myoungju, Kim, Jeonghwan
• Primary Examiner(s): Bradford, Jonathan
• Application Number:
  US14/786,719
• Publication Number:
  US 20160123657A1
• Time in Patent Office: 1,421 Days
• CPC Class Codes: F25D 11/00 Self-contained movable devices, e.g. domestic refrigerators
• CPC Class Codes: F25D 2201/10 with respect to heat
• CPC Class Codes: F25D 23/00 General constructional features F25D21/00 takes precedence
• CPC Class Codes: F25D 23/02 Doors; Covers F25D23/08 takes precedence locks or fastenings E05B65/0042
• CPC Class Codes: F25D 23/028 Details
• CPC Class Codes: F25D 23/04 with special compartments, e.g. butter conditioners
• CPC Class Codes: F25D 23/062 defining a cabinet
• CPC Class Codes: F25D 23/067 Supporting elements
• CPC Class Codes: F25D 23/12 Arrangements of compartments additional to cooling compartments; Combinations of refrigerators with other equipment, e.g. stove
• CPC Class Codes: F25D 2300/00 Special arrangements or features for refrigerators; cold rooms; ice-boxes; Cooling or freezing apparatus not covered by any other subclass
• CPC Class Codes: F25D 2323/021 French doors
• CPC Class Codes: F25D 2400/02 Refrigerators including a heater
• CPC Class Codes: F25D 2400/361 Interactive visual displays
• CPC Class Codes: F25D 25/025 Drawers
• CPC Class Codes: F25D 2500/06 Stock management
• CPC Class Codes: F25D 2600/00 Control issues
• CPC Class Codes: F25D 27/00 Lighting arrangements in general F21
• CPC Class Codes: F25D 2700/02 Sensors detecting door opening
• CPC Class Codes: F25D 2700/06 Sensors detecting the presence of a product
• CPC Class Codes: F25D 29/00 Arrangement or mounting of control or safety devices
• CPC Class Codes: F25D 29/005 : Mounting of control devices
• CPC Class Codes: G02B 27/0006 : with means to keep optical surfaces clean, e.g. by preventing or removing dirt, stains, contamination, condensation G02B1/18 takes precedence; cleaning in general B08B
• CPC Class Codes: G03B 29/00 : Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects combinations with flash apparatus G03B15/03
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 : involving subtraction of images
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/52 : Elements optimising image sensor operation, e.g. for electromagnetic interference [EMI] protection or temperature control by heat transfer or cooling elements
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/57 : Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/633 : for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 5/2621 : Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/2624 : for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A refrigerator includes a storage compartment formed in a cabinet of the refrigerator by a fixed insulation wall, the storage compartment provided with an access opening, a door rotatably provided to the cabinet to open and close the access opening, a drawer provided in the storage compartment, a camera fixed to a ceiling of the storage compartment to photograph both a first region for storing of food arranged in an external space of the drawer in the storage compartment and a second region for storing of food arranged in an internal space of the drawer in the storage compartment, and a controller to separate, through a capturing time of a picture containing both the first region and the second region, a first region picture and a second region picture from the picture to individually divide and store the first region picture and the second region picture.

Claims

1. A refrigerator comprising:

a storage compartment;

a drawer movably provided in the storage compartment and provided with a marker;

a camera arranged to photograph inside the drawer from outside the drawer; and

a controller arranged to sense a position of the marker in a plurality of photographs taken through the camera at time intervals and to determine state information about the drawer based on a change in position of the marker in the plurality of photographs.

2. The refrigerator according to claim 1, wherein the state information about the drawer includes at least one selected from the group comprising:

a degree of withdrawal of the drawer;

whether the drawer is withdrawn;

a movement direction of the drawer; and

a stopped state or moving state of the drawer.

3. The refrigerator according to claim 1, wherein the controller is arranged to process the marker in a single photo within a predetermined range allowing the marker to be positioned.

4. The refrigerator according to claim 1, wherein the controller is arranged to determine a coordinate of the marker within a taken photograph with respect to a reference coordinate in order to determine a path of movement of the marker, and wherein the reference coordinate is a one-dimensional coordinate in a direction parallel with the movement direction of the drawer.

5. The refrigerator according to claim 1, wherein the controller is arranged to compare two photographs of the plurality of photographs to determine movement of the drawer and movement direction of the drawer according to change in the position of the marker.

6. The refrigerator according to claim 1, wherein the controller is arranged to:

determine the position of the marker in one photograph in a first section of the moveable distance of the drawer, wherein the first section is a section from an introduced position of the drawer to a position reached by the drawer withdrawn by a predetermined distance, and

determine a change in the position of the marker by comparing two photographs of the plurality of photographs in a second section of the moveable distance of the drawer, wherein the second section is a section from the position reached by the drawer withdrawn by the predetermined distance to the position reached by the drawer when the drawer is fully withdrawn.

7. The refrigerator according to claim 6, wherein the length of the second section is less than or equal to the first section, and wherein a capturing time interval of the camera is longer than or equal to a time taken for the controller to compare the positions of the marker located in the second section in the two photographs.

8. The refrigerator according to claim 1, wherein the capturing time interval of the camera is longer than or equal to a time taken for the controller to determine the position of the marker.

9. The refrigerator according to claim 1, wherein the marker is disposed on a top surface of a front wall of the drawer.

10. The refrigerator according to claim 9, wherein the drawer comprises a drawer disposed on a left side of a lateral center of the storage compartment and a drawer disposed on a right side of the lateral center, and the marker is installed at each of the drawers,

wherein the camera is fixed to a ceiling over the drawers at a position of the lateral center of the storage compartment, and the marker installed at each of the drawers is arranged close to the lateral center of the storage compartment.

11. The refrigerator according to claim 1, wherein the marker comprises a colour different from a colour of the drawer positioned at a boundary of the marker.

12. The refrigerator according to claim 11, wherein

the marker comprises a width side defined to be parallel with the movement direction of the drawer and a length side defined to be perpendicular to the movement direction of the drawer, the length side is longer than the width side;

or,

the marker comprises a first color portion and a second color portion having a different color than the first color portion, wherein the first color portion and the second color portion are alternately disposed.

13. The refrigerator according to claim 1, wherein the marker comprises at least one shape having a different color and repeatedly disposed.

14. A method of controlling a refrigerator provided with a drawer and a camera fixedly provided to a ceiling of a storage compartment of the refrigerator, the method comprising:

recognizing closing of the drawer, wherein the recognizing comprises;

sensing start of opening of the drawer;

acquiring an image of the interior of drawer using the camera;

processing the acquired image;

recognizing a predetermined marker from the acquired image as a result of the processing;

tracing movement of the recognized marker;

acquiring a final image of an interior of the drawer using the camera at a moment the closing the drawer begins; and

displaying the final image.

15. The method according to claim 14, wherein the tracing comprises:

sensing termination of the opening of the drawer using the recognized marker; and

sensing the closing of the drawer using the marker.

16. The method according to claim 14, further comprising displaying, when the predetermined marker is not recognized from the acquired image as a result of the processing, failure of recognition of the marker.

17. The method according to claim 15, further comprising:

determining, when the termination of the closing of the drawer is sensed, whether acquisition of the final image through recognition of the marker has been successful; and

displaying, when acquisition of the final image through recognition of the marker the marker has been failed.

18. The method according to claim 15, wherein the predetermined marker is provided with a pattern of repeated identical shapes having different colors.