View source for Dup

{{FilterCat5|External_filters|Plugins|Plugins_x64|Adjustment_filters|Duplicate_Frame_Detectors}}
{{Filter3
|1={{Author/Donald Graft}}
|2=v2.32a
|3=[https://forum.doom9.org/attachment.php?attachmentid=15132&d=1447966003 dup232a_src.7z]
|4=Duplicate frame detectors
|5=[http://www.gnu.org/licenses/gpl-2.0.html GPLv2]
|6=[http://forum.doom9.org/showthread.php?t=41850 Doom9 Forum] (original), [https://forum.doom9.org/showthread.php?t=153037 Doom9 Forum] (update)}}
<br>
== Description ==
This plugin for AviSynth implements a robust duplicate frame detector. It was initially created as a test bed for a new frame differencing algorithm, but to make it a useful filter in its own right, copy and blend features have been implemented, whereby a string of frames that are all within threshold difference of the first frame of the string will be replaced with a string of frames all identical to the last frame of the source string. The last frame is used instead of the first because often the first frame after a scene change has more blocking artifacts, etc. If the blend option is enabled, the replacement frame will be generated from a blending of all the duplicates in the string (useful for noise reduction).

This is intended for use in clips that have a significant number of duplicate content frames, but which differ due to noise. Typically anime has many such duplicates. By replacing noisy duplicates with exact duplicates, a bitrate reduction can be achieved. It's a free lunch! ('Marc FD' first implemented this copying functionality.) Additionally, if the blend option is enabled, a significant noise reduction can be achieved for duplicated frames.

Some duplicate detectors are incapable of adapting to changes in the luminance levels of the various scenes in a clip, and they are badly affected by noise. They have delicate absolute thresholds that must be set by the user. This filter attempts to solve all of these problems.

Dup uses a small window that it scans over the frame, so it is capable of seeing very small changes. The effect of noise is reduced by means of automatic cancellation through a summation process. The frame difference is normalized to the range 0-100% to make setting the threshold more intuitive. 

<br>

== Requirements ==
* [x86] [[AviSynth+]] or [http://sourceforge.net/projects/avisynth2/ AviSynth 2.60]
* [x64] [[AviSynth+]]
*Supported color formats: [[YUY2]], [[YV12]]
<br>
* [https://www.microsoft.com/en-us/download/details.aspx?id=48145 Microsoft Visual C++ 2015 Redistributable Package (x86)]
:<span style="color:red">***</span> <tt>vcredist_x86.exe</tt> is required for 32-bit <tt>Dup</tt>
<br>

== [[Script variables|Syntax and Parameters]] ==
:{{Template:FuncDef|Dup1 (clip, float "threshold", bool chroma", bool "show", bool "copy", int "maxcopies, bool "blend, bool "debug", string log", int "blksize")}}
<br>
::{{Par2| |clip| }}
:::Input clip.
<br>
::{{Par2|threshold|float|3.0}}
:::This parameter defines the percentage change in the most different 32x32-pixel window that is enough to declare a frame a duplicate. The way to think about it is that if the threshold is (say) 7%, then if any 32x32 pixel area changes by 7% or more, it is not a duplicate, otherwise it is. Use the show option to display frame difference values to help in setting the threshold if you prefer something other than the default.

:::You should always set your threshold above the noise level of the clip. The amount above will depend on how many duplicates you want to generate, i.e., the desired bitrate reduction. To determine the noise level, examine the frames and look for those having duplicate content, albeit with noise differences. You may find that frames that are content duplicates show a metric of as high as a few percent (due to noise). Determine the highest metric that you get for the content duplicates. Then set your threshold above that level with as much margin as you want to achieve the desired bitrate reduction. 
<br>
::{{Par2|chroma|bool|true}}
:::Set chroma=false to exclude chroma from the frame differencing. 
<br>
::{{Par2|show|bool|false}}
:::When set to true, information is overlayed onto the upper left of the frame. When copy=false, the 32x32 pixel area that is most different from the same area in the next frame is delineated with a white box, and the difference metric is displayed. If the box has an X through it, the frame was declared a duplicate according to the configured difference percentage (threshold). When copy=true, the box is not displayed. The text information will tell you which frames are duplicated and with which source frame. 
<br>
::{{Par2|copy|bool|true}}
:::If this option is set to true, declared duplicates will be replaced with a single copy. If there is a string of duplicates, the last frame in the string is used for all copies. If blend=true, the copy frame will be generated by blending together all the duplicates in the string. 
<br>
::{{Par2|maxcopies|int|20}}
:::Determines the maximum number of copies that can be emitted due to duplicate declaration. However, if the input stream has more than the maximum number of duplicates in a string, then, of course, they will all be output. This limit just forces the current duplicate sequence to end; the comparisons restart with the nextframe frame.
<br>
::{{Par2|blend|bool|false}}
:::If this option is set to true, the copy behavior will be modified such that instead of using the last frame in the string of duplicates, a frame will be created and used from a blend of all the duplicates in the string. This is useful for noise reduction. This option requires copy=true. Also, note that the blend frame is generated when the first frame in the string is requested. So random access into the middle of a string will not invoke blending. Play the clip straight through from a starting point for correct operation.
<br>
::{{Par2|debug|bool|false}}
:::This parameter enables debug output to the DebugView utility. This utility can be downloaded from my web site as given below. 
<br>
::{{Par2|log|string|NULL}}
:::If debug=true and this parameter is set to a file name (or path/name), then the debug information will be logged to the file. The file is closed when the filter instance is destroyed, e.g., when VirtualDub is closed.
<br>
::{{Par2|blocksize|int|32}}
::: Determines the blocks size for analysis. The lesser block size, the greater sensitivity to small details change (and to noise).
<br>
== Examples ==
Dup with default settings:
 [[AviSource]]("Blah.avi")
 #TODO
<br>
== Changelog ==
 Version      Date            Changes<br>
 v2.32a       2015/11/19      - Author: tartak
                              - Fixed access violation bugs (use of uninitialized cache_count and highest_sum2)
                              - Updated to FilterSDK api of AviSynth 2.6<br>
 v2.32        2014/11/11      - Author: Max73
                              - Show metric (cur-prev) when reading the first frame outside a set of duplicates
                              - Skip border blocks in worst block metric calculation, they are more noisy
                              - Added a simple motion metric to denoise decisions
                                old_metric       = SumAbsDiff[]         = Sum(Abs(Diff))         : more sensitive to high contrast any motion, less sensitive in low contrast motion, more noisy
                                lc_motion_metric = abs(SumSigDiff[])    = Abs(Sum(Diff))         : more sensitive to large motion in low contrast, less noisy (pos and neg noise cancels out)
                                new metric       = 32/BLKSIZE * (old_metric/2 + lc_motion_metric): highest_Sum refers to this now
                              - Show Largest sum block shown in overlay, for new metric (much brighter), and old one (little brighter)
                                settings (personal preference):  Dup(threshold=1.50, chroma=false, show=true, copy=true, maxcopies=20, blend=true, blksize=32, debug=false)
<br>
<br>

== External Links ==
[http://rationalqm.us/dup/dupnew.html rationalqm.us] - Official Dup homepage.
<br>
<br>
-----------------------------------------------
'''Back to [[External_filters#Duplicate_Frame_Detectors|External Filters]] &larr;'''