RoboCrop

Revision as of 16:37, 23 April 2020

Description

RoboCrop is an automatic cropping solution to crop black borders from video clips, loosely based on (but using no code from) AutoCrop by Glenn Bussell.

For the most part, you can just call it with RoboCrop(clip) and not bother with any other arguments, it is intended to be pretty much autonomous in its decisions. You might however want to alter eg WMod/HMod if your encoder has special requirements, and perhaps Laced if your source is likely to be fully progressive.

Borders are detected by sampling at Samples frames, at scanlines (h/v) using AverageLuma (RGB is converted to Luma Y at either TV or PC levels, See Matrix). This sampling is done on all 4 sides of the frame. If a scanline Average luma is below or equal to Thresh, then is considered possible black border, above considered possible image, if Baffle [default=4] adjacent scanlines above Thresh, then it IS image.

Requirements

• [x86]: AviSynth+ or AviSynth 2.6
• [x64]: AviSynth+
• Supported color formats: RGB32, RGB24, YUY2, Y8, YV12, YV16, YV24
• (Must be Seekable ie best not DivX or XVid). AUDIO: No change.

• Microsoft Visual C++ 2008 Redistributable Package (x86 / x64)

*** vcredist_x86.exe is required for RoboCrop-x86

*** vcredist_x64.exe is required for RoboCrop-x64

Syntax and Parameters

RoboCrop (clip, int "Samples", float "Thresh", bool "Laced", int "WMod", int "HMod", int "RLBT", bool "Debug", float "Ignore", int "Matrix", int "Baffle", bool "ScaleAutoThreshRGB", bool "ScaleAutoThreshYUV", int "CropMode", bool "Blank", bool "BlankPC", bool "Align", bool "Show", string "LogFn", bool "LogAppend", float "ATM", int "Start", int "End", int "LeftAdd", int "TopAdd", int "RightAdd", int "BotAdd", int "LeftSkip", int "TopSkip", int "RightSkip", int "BotSkip")

   Samples=40, Number of frames to sample from source clip c. v1.06 changed default from 32 to 40.
   As an observation, most clips have good border recogition within the 1st 2 or 3 sampled frames using the default -40.0 AUTOTHRESH
   although have noticed some dark clips that required ~8 samples (maybe more) for full recognition @ default Thresh = -40.0
   We use a default Samples=40, because we are intrinsically paranoid.
     If number of frames between frame at 5% of framecount and frame at 90% of framecount is greater than 250 and is also greater
   than Samples, will ignore the first 5% and last 10% of frames when both auto-thresholding and crop sampling to try to negate
   effects of artificial black in titles and end credits. Can override the Auto Credits skipping by setting Start and/or End frame
   of the range to sample.
     Samples = 0, will be converted to Samples = FrameCount - 1, ie auto credits skipping disabled  and ALL FRAMES SAMPLED,
   of use for very short scenes, not for general full movie clips.
   Set eg Samples = Int(FrameCount * 2.0 / 100.0 + 0.5) to sample 2.0% of frames in clip.

   Thresh: Default= -40.0 (==DEFAULT AUTOTHRESH, any -ve (or 0) Thresh is AUTOTHRESH where Thresh adjustments are automatic).
   v1.06, Changed DEFAULT AUTOTHRESH from -32.0 to -40.0.

   Thresh > 0: (Explicit Threshold)
     A user supplied +ve Thresh should be at correct TV/PC levels for the the clip being worked on ie 16->235 for TV levels and
     0->255 for PC Levels (RGB, as appropriate for matrix being used).

   Thresh <= 0: (AUTOTHRESH)
     When Thresh <= 0, the clip will be sampled over Samples frames to find the minimum YPlaneMin (using Matrix if RGB) which
     we will call MINLUMA and an Explicit Threshold calculated by adding MINLUMA to abs(Thresh), after that it is processed
     as for Thresh > 0: (Explicit Threshold) as noted above, but, before adding MINLUMA, some AUTOTHRESH massaging and scaling occurs.

     Here AUTOTHRESH Thresh massaging and scaling occurs in sequence:-
     1 ) if (Thresh == DEFAULT AUTOTHRESH && ATM < 40.0) (DEFAULT AUTOTHRESH = exactly -40.0, defaulted OR user supplied):
           Let sampstart and sampend, be starting and ending frames numbers after any Auto Credits skipping and/or user set Start or End.
           Let Samples be limited to sampend-sampstart+1.
           Let SampRange (Sample Range) = SampEnd-SampStart+1.
           samples_massage =(Samples>=20)           ? 1.0 : (Samples-1)   * (1.0/(20-1))           # No massaging if Samples >= 20
           range_massage   =(SampRange >= (250*20)) ? 1.0 : (SampRange-1) * (1.0/((250*20)-1))     # No massaging if SampRange >= 5000
            Both samples_massage and range_massage will be in range 0.0 to 1.0.
            Thresh = -(((samples_massage * range_massage) * (40.0 - ATM)) + ATM)
             This adjustment to Auto Thresh is to reduce the possibility of overcropping on eg a dark low 'Samples' clip, or where
             source SampRange (ie temporal frame set) is too small to take a reliable sample from.
             Resulting massaged Thresh will range between -ATM and -40.0.
             Although massaging is intended to reduce overcropping, it could result in not cropping enough border (less disastrous),
             its a bit of a balancing act really. See also ATM.
     2 ) If RGB AND PC matrix(default) AND ScaleAutoThreshRGB==True(default) then
           Thresh= Thresh*(255.0/(235-16))
     3 ) If YUV AND ScaleAutoThreshYUV==True(default=false) then
           Thresh= Thresh*(255.0/(235-16))

     Steps 2) and 3) above, by default treat a -ve AUTOTHRESH as being @ TV Levels and so Scale RGB Thresh to PC levels but not YUV.
     If you want to supply a PC levels AUTOTHRESH Thresh for RGB, then simply set ScaleAutoThreshRGB=false to inhibit scaling.
     Note, if a TV levels Matrix is supplied for RGB, then scaling will always be inhibited.
      If your clip is YUV at PC levels and you want to use eg DEFAULT AUTOTHRESH (-40.0, which is considered to be @ TV levels),
     then set ScaleAutoThreshYUV=True to enable Thresh scaling.
      If your clip is YUV at PC levels and you want to use a PC levels AUTOTHRESH (-ve) then leave ScaleAutoThreshYUV at default false
     which does not scale Thresh.
     After any scaling, MINLUMA is then added to abs(Thresh) and processed as for +ve Explicit Threshold as noted above.
     NOTE, Above RoboCrop step 1) 'massages' DEFAULT AUTOTHRESH (exactly -40.0) if low samples count or if short clip. Reason being to
      avoid overcropping when insufficient data available for reliable cropping. It is considered better to not crop enough or at all,
      than to overcrop. You can override by simply setting an explicit threshold (+ve) of eg 40.0, or setting a NON-DEFAULT auto thresh
      (-ve) eg -16.0 or -40.1, where YPlaneMin is established for the sampled frames and then abs(thresh) is added to that value which
      is then used as an explicit thresh.

  Laced:, Default=true, true for Interlaced.
         RoboCrop automatically deduces colorspace cropping restrictions and sets internal XMod and YMod,
         eg XMod and YMod would both be set to 2 for YV12, for YUY2 Xmod=2, YMod=1, etc.
         If Laced==true(default), then internal YMod is doubled so as not to destroy interlaced chroma.
         Below HMod is defaulted to internal YMod after the Laced hint is applied to YMod.
         You can set Laced=False if your source is known Progressive, if your dont then you may lose a few scanlines.

  WMod:, Default=Max(XMOD,4), Where XMOD is the natural chroma cropping restriction of the colorspace concerned, eg 2 for YV12.
      WMod MUST be a multiple of internal XMOD as described above, or it will throw an error.
        v1.06, Changed default from XMOD to Max(XMOD,4), this is for several reasons, some encoders, media players and VDubMod
      demand WMOD==4 ie width an exact multiple of 4 (some Mpeg encoders may demand WMOD=16 which you would have to set yourself).
      In addition to above, Avisynth v2.58 kicks up a fuss if any YUV colorspace clip with Width not an exact multiple of 4 is
      returned as final result clip (intermediate clips only have to comply with colorspace XMOD requirment). It is not really
      the job of any plugin to guess whether it's result will be the final return clip and so is not really for RoboCrop to
      enforce WMOD=4 for eg YV12, BUT, combined with the other reasons above, it was decided to change default WMOD to Max(XMOD,4),
      you will have to set eg WMOD=2 manually if required for eg YV12 or YUY2, or WMOD=1 for eg RGB32 if that is what you want.
      Avisynth has no problem returning eg RGB32 with an odd width, but at least one player had problems when RGB32 was XMOD=2
      about 12 months prior to writing this (now fixed). WMOD set to 4 [as Max(XMOD,4) will likely be] will be least problematic
      choice but you know what you require and can set WMOD to suit.
        
   HMod:, Default=The natural chroma cropping restriction of the colorspace concerned, BUT, doubled if laced=true.
     HMod MUST be a multiple of internal YMod as described under Laced above, or it will throw an error. If eg colorspace is
     YV12 then YMod would be set to 2, and if Laced, then YMod would be doubled to 4, so HMod MUST be a multiple of 4.
     Some encoders may require an WMod/HMod of eg 8 or 16, and if set thus, would crop off more or less depending upon
     which CropMode is set, if later resizing will be done, then encoder requirements can be satisfied during the resize.
     NOTE, Some players and VirtualDubMod (Helix YV12 decoder) dont like WMOD less than 4 (Vdub latest, OK, internal decoder).
       If eg VDMod shows blank frame, OR eg player halts saying eg "No combination of filters cound be found to render frame"
       then WMod needs to be a multiple of 4.
     NOTE, RT_Stats has a function RT_GetProcessName() which could be used to set WMod=4 only for specific programs eg,
         RoboCrop(WMod=(RT_GetProcessName=="VirtualDubMod.exe") ? 4 : Last.RT_ColorspaceXMod) # Where VDubMod opened Avisynth

   RLBT:=15=All Borders, Bitflags of borders to detect, 15 ($0F) crops all four. Each letter in the name 'RLBT' represents an edge and bit position
     starting with 'R' in bit 3 representing the value 8 (2^3=8). 'L' = bit 2=4 (2^2=4), 'B' = bit 1=2 (2^1=2), 'T' = bit 0=1 (2^0=1).
     To calculate the RLBT bitflags, for 'R'(Right) add 8, for 'L'(Left) add 4, for 'B'(Bottom) add 2, and for 'T'(Top) add 1.
     Add all of the bit flags together 8+4+2+1 (=15) crops all four edges, 8+4 crops only Right & Left, and 2+1 crops only Bottom & Top.
     RLBT affects border detection.
     User Skips eg LeftSkip are applied irrespective of what RLBT is set to.
     User adjustments eg LeftAdd are applied irrespective of what RLBT is set to.

   DEBUG:=False=No Debug. Set True for debugging info, need DebugView: https://docs.microsoft.com/en-us/sysinternals/downloads/
     The debug info output shows eg range limiting of Samples and sample info and resultant auto set Thresh. You are encouraged
     to use debug to see the eg the auto Thresh massaging in action, it may help to understand usage of the plugin. MS DebugView
     can also be used to view output from other plugins and programs that can also be useful.

   Ignore:=0.4, Percentage of darkest pixels to ignore during AutoThresh scan to detect minimum luma pixel of all sampled frames.
    (ignores a few extreme pixel values ie noise, as for Threshold arg in YPlaneMin).
    v1.05, changed default from 0.2% to 0.4%.

   Matrix:, RGB ONLY. For conversion of RGB to YUV-Y, 0 = Rec601, 1 = Rec709, 2 = PC601, 3 = PC709
     Default for RGB is:- If clip Width > 1100 OR clip Height > 600 Then 3(PC709) , else 2(PC601) : YUV not used
     The defaults are for PC601 & PC709 range 0-255.
     So as to not require different AutoThresh for RGB, if clip c is RGB AND matrix is PC Levels AND Thresh < 0.0 and ScaleAutoThreshRGB=true,
     then Thresh will be scaled to RGB full range ie Thresh = Thresh * (255.0/(235.0-16.0)) ONLY when AutoThresh (ie Thresh < 0.0,
     YPlaneMin relative).
     When +ve Thresh is explicitly supplied (Thresh > 0.0) it is not scaled and assumed to be already correct range TV or PC levels.

   Baffle:=4, Number of scanlines (h/v) that must break threshold to be considered image (can avoid noise around frame edges).
     Does not usually need changing but might be of use to avoid some kind of eg teletext data at top of frame in broadcast video.

   ScaleAutoThreshRGB: bool default True. If true and RGB and Matrix at PC levels, and Thresh -ve (autothresh) then thresh will (after any
     auto Thresh massaging) be scaled to PC levels. By default, -ve Auto Thresh is considered to be at TV levels, and so will be scaled to
     to PC levels to match the matrix setting. If ScaleAutoThreshRGB is False Then autothresh is considered to be at PC levels and not scaled.

   ScaleAutoThreshYUV: bool default False. If true and YUV and Thresh -ve (autothresh) then Thresh will (after any auto Thresh massaging),
     be scaled to PC levels. By default, -ve Auto Thresh is considered to be at TV levels, this allows you to change that assumption when
     source clips are at PC levels. If supplying PC levels clip and PC levels autothresh then leave this setting at false (no scaling).
     The above seemingly awkward  settings are due to having to deal with YUV @ TV levels and RGB @ PC levels with the possibility of
     YUV @ PC levels.

   Cropmode: default=2 == CropMore. 1 = CropLess. Range 1 -> 2.
     1 (CropLess) crops a little less border if the exact cropping coords do not comply with XMod/YMod/WMod/HMod as
       described earlier. May leave a little border edge.
     2 Default, (CropMore) crops a little extra where exact coords do not comply XMod/YMod/WMod/HMod requirements.
       You may lose a little of the image edges.
     CropMode is applied AFTER any user edge adjustments via LeftAdd, TopAdd, RightAdd and BotAdd.

   Blank: Default False. If true, then Blanks the borders, ie sets border to RGB(0,0,0) or YUV(16,128,128) instead of cropping.
     Is overridden if Show == true.

   BlankPC: default false. If Blanking then blanks to YUV(0,128,128) instead of default YUV(16,128,128).

   Align: Default false. If true, then aligns frames in memory when cropping, for eg SSE2 16-byte alignment required by some
     filters, particularly smoothing filters (filters following RoboCrop). See Docs for Crop().
     Only Applies if Cropping, ie Blank==false AND Show==false.

   Show: Default false. If true, then does no cropping, just shows a little info on frame. Overrides both Blank and Align.

   LogFn: Default "" (unset). Filename of Log file.
     Will output crop coords, crop coords will be output whether cropping or not (Blank/Show), format as within quotes below.

     '28 92 668 392 2 4 2 4 720 576 27 89 696 484 2 27 89 696 484 '

     Where in order:
       CropX CropY CropW CropH     :  Crop coords mod XMod, YMod, WMod, HMod.
       XMod YMod WMod HMod         :  As used for above crop coords
       Width Height                :  Original frame width and height
       ORG_X1 ORG_Y1 ORG_X2 ORG_Y2 :  Exact coords of found image, not rounded according to x/y/w/h mod (Before User Adjust)
       CropMode                    :  1 = CropLess, 2 = CropMore(default)
       X1 Y1 X2 Y2                 :  Coords after any User adjustments eg ORG_X1+LeftAdd etc and before applying XMod etc.
     All values output are single SPACE separated.

   LogAppend: Default false. If true then appends log to any existing log file.

   ATM: Float default 4.0 (0.5 -> 40.0). Silently limited to valid range.
     ATM allows user to set the DEFAULT AUTOTHRESH massaging minimum. When eg samples = 1, then auto thresholding is of course quite
     unreliable and so auto Thresh would be 'massaged' to -ve(ATM), other values of Samples below 20 will likewise have auto thresh
     massaged but to a lesser degree, linearly between -ve(ATM) for Samples == 1 and -ve(40.0) for Samples == 20.
       Auto Thresh massaging also takes into account the frame range of samples (first sampled frame to last sampled frame inclusive)
     and this is mixed together with any samples massaging then applied to auto Thresh.
     Previous (Fixed) default for ATM was 0.5, for maximum safety so that a single Samples scan would NOT overcrop. The new ATM arg default
     of 4.0 is a less paranoid safety setting which should in most cases work well but with a very short clip or eg single image then
     user might be better off giving the minimum ATM of 0.5. An ATM of 40.0 will switch OFF default auto thresh massaging.
     So long as sample range is about 5000+ frames  and Samples at least 20(default 40), then there will be no auto thresh massaging and
     current default is unlikely to need changing, with very short clips or reduced Samples count, then you might want to reduce ATM, for
     maximum paranoia, set ATM=0.5 to 1.0, especially in an app that processes single images. See Thresh.

   Start: Default Undefined. Start frame of scan area. Overrides Auto Intro credits skipping.
     For Auto Intro and End Credits Skipping to be set to 5% (of FrameCount for Intro Skipping) and 90% (for End Skipping), the number of
     frames between them must be greater or equal to 250 frames, and MUST also be greater than Samples, otherwise Auto skipping ignored and
     the Start and End frame numbers are set to 0 and Framecount - 1. If a user supplies eg a Start frame number ONLY, then End Skipping
     has to comply with the same conditions, range between End Skip frame and user supplied Start has to be at least 250 frames and greater
     than Samples, otherwise End frame set to FrameCount - 1.
     After either via Auto Credits skipping, or user supplied Start/End, or defaulted to 0 & FrameCount -1, we have a sample scan range.

   End: Default Undefined. End frame for scan area. Overrides Auto End credits skipping. 0 (or less) will be converted to Framecount - 1.
     See previous Start setting.

   Int LeftAdd, TopAdd, RightAdd, BotAdd, All default 0. Valid range depends upon clip dimensions, Skip and Baffle settings. 
     User adjustments to Skip + Autocrop exact found coords. CropMode applied after user Add adjustments.

   Int LeftSkip, TopSkip, RightSkip, BotSkip, All default 0. TopSkip/BotSkip range 0 -> Height/4. LeftSkip/RightSkip range 0 -> Width/4. 
   Minimum edge croppings, detection starts with these edges cropped already, and are applied whether RLBT flags (ie edge crop scanning)
   are applied or not. Avoid noise eg teletext at top of frame of VHS gunk at bottom.
   NOTE, The Skip and Add args eg LeftSkip and LeftAdd are processed irrespective of the relevant RLBT flags so if you just wanted to
   autocrop top and botton then set RLBT=$03 and can set either LeftSkip = 16 OR LeftAdd=16 to manually crop 16 pixels from Left hand
   Side, DO NOT SET BOTH are this would crop off 32 pixels. LeftSkip is applied before autocrop, and LeftAdd after autocrop, and then
   results are rounded to WMOD:HMOD, dependent upon cropmode (Default CROPMORE).

Additional Information

Cropping will likely fail on END CREDITS ONLY clips, where on black background, and will probably crop off the sides that are no different in average luma to any letterbox borders, if you cannot see the borders, then neither can RoboCrop(), even setting the auto Thresh to eg -1.0 or 0.0 is quite likely to fail. (See RLBT edge bitflags).

If cropping too much border, then increase Samples or reduce Thresh or lower ATM if short clip. If not cropping enough border then border is being seen as image, increase Thresh (for -ve Thresh , make more -ve).

To speed up, you may want to leave Auto-Thresh alone and reduce samples, but there is danger that it will not detect all borders (overcrop). Suggest you do not go below Samples=20, although early default for samples was 12, we use a paranoid setting of 40 by default. However, RoboCrop is quite sprightly and you will probably not need to reduce Samples, but if you are interested in how long it takes to do it's auto thresh scanning in the filter constructor, the time taken is output via the debug arg to DebugView. Clips that have no letter boxing are quickest dealt with and those with largest borders take most time as it has to scan the borders of all Samples frames, looking for bigger image. Doing a short test as I write this on a 10 mins 12 secs PAL DVD vob of Cabaret, it took 0.875secs where borders = Crop(30,90,-24,-92) with default args except for Debug=true (RoboCrop v1.03, YV12, Core Duo, Duel Core 2.14Ghz, Sata 2). NOTE, scans over entire range of clip and so clip must be seekable, ie not eg DIVX with only a single keyframe, would take nearly forever.

NOTE, The plugin AutoCrop() uses a Thresh of 40.0 and samples == 5 by default (No AutoThresh), but the base logic is not too dissimilar.

Final NOTE, use arg Laced=False if you always process progressive or deinterlaced, And ATM set between 0.5 and 4.0 if very short clips/single-image.

Changelog

Version       Date            Changes

v1.06         04/22/2015      - Changed Default WMod to 4, avoid possible problems in Avisynth v2.58, some media players and VDubMod.             
                              - Changed Default AutoThresh to -40.0, been wanting to do this for some time.
                              - Changed default Samples to 40, paranoia.

v1.05         02/25/2015      - Addef LeftSkip, RightSkip, TopSkip, BotSkip (also shown when Show).
                              - Changed default Ignore from 0.2 to 0.4.

v1.04                         - Added User adjustments to detected border edges.

v1.03         12/26/2013      - Minor mods.

v1.00         11/11/2013      - Beta status removed (ATM arg added).

v0.21         07/12/2013      - Changed default samples to 32, modified Thresh Massaging, generally a bit more paranoia.

v0.20         06/19/2013      - Added args, BlankPC, LogFn, LogAppend.
                              - Changed, range massage switch on frames to below 1500 (from 1000).
 
v0.10         06/18/2013      - Small mod for Auto Thresh Massaging.

v0.00         06/12/2013      - Initial release.

External Links

Back to External Filters ←