Dup

Abstract
Author	Donald Graft
Version	v2.32a
Download	[x86]: dup232a_src.7z [x64]: Dup1_x64.7z
Category	Duplicate frame detectors
License	GPLv2
Discussion	Doom9 Forum (original), Doom9 Forum (update)

Description

This plugin for AviSynth implements a robust duplicate frame detector. It was initially created as a test bed for a new frame differencing algorithm, but to make it a useful filter in its own right, copy and blend features have been implemented, whereby a string of frames that are all within threshold difference of the first frame of the string will be replaced with a string of frames all identical to the last frame of the source string. The last frame is used instead of the first because often the first frame after a scene change has more blocking artifacts, etc. If the blend option is enabled, the replacement frame will be generated from a blending of all the duplicates in the string (useful for noise reduction).

This is intended for use in clips that have a significant number of duplicate content frames, but which differ due to noise. Typically anime has many such duplicates. By replacing noisy duplicates with exact duplicates, a bitrate reduction can be achieved. It's a free lunch! ('Marc FD' first implemented this copying functionality.) Additionally, if the blend option is enabled, a significant noise reduction can be achieved for duplicated frames.

Some duplicate detectors are incapable of adapting to changes in the luminance levels of the various scenes in a clip, and they are badly affected by noise. They have delicate absolute thresholds that must be set by the user. This filter attempts to solve all of these problems.

Dup uses a small window that it scans over the frame, so it is capable of seeing very small changes. The effect of noise is reduced by means of automatic cancellation through a summation process. The frame difference is normalized to the range 0-100% to make setting the threshold more intuitive.

• Note: blend=true does not work in Dup1_x64.

Requirements

• [x86] AviSynth+ or AviSynth 2.60
• [x64] AviSynth+
• Supported color formats: YUY2, YV12

• x86 only: Microsoft Visual C++ 2015 Redistributable Package (x86)

*** vcredist_x86.exe is required for 32-bit Dup1-x86

Syntax and Parameters

Dup1 (clip, float "threshold", bool chroma", bool "show", bool "copy", int "maxcopies, bool "blend, bool "debug", string log", int "blksize")

clip   =

Input clip.

float  threshold = 3.0

This parameter defines the percentage change in the most different 32x32-pixel window that is enough to declare a frame a duplicate. The way to think about it is that if the threshold is (say) 7%, then if any 32x32 pixel area changes by 7% or more, it is not a duplicate, otherwise it is. Use the show option to display frame difference values to help in setting the threshold if you prefer something other than the default.

You should always set your threshold above the noise level of the clip. The amount above will depend on how many duplicates you want to generate, i.e., the desired bitrate reduction. To determine the noise level, examine the frames and look for those having duplicate content, albeit with noise differences. You may find that frames that are content duplicates show a metric of as high as a few percent (due to noise). Determine the highest metric that you get for the content duplicates. Then set your threshold above that level with as much margin as you want to achieve the desired bitrate reduction.

bool  chroma = true

Set chroma=false to exclude chroma from the frame differencing.

bool  show = false

When set to true, information is overlayed onto the upper left of the frame. When copy=false, the 32x32 pixel area that is most different from the same area in the next frame is delineated with a white box, and the difference metric is displayed. If the box has an X through it, the frame was declared a duplicate according to the configured difference percentage (threshold). When copy=true, the box is not displayed. The text information will tell you which frames are duplicated and with which source frame.

bool  copy = true

If this option is set to true, declared duplicates will be replaced with a single copy. If there is a string of duplicates, the last frame in the string is used for all copies. If blend=true, the copy frame will be generated by blending together all the duplicates in the string.

int  maxcopies = 20

Determines the maximum number of copies that can be emitted due to duplicate declaration. However, if the input stream has more than the maximum number of duplicates in a string, then, of course, they will all be output. This limit just forces the current duplicate sequence to end; the comparisons restart with the nextframe frame.

bool  blend = false

If this option is set to true, the copy behavior will be modified such that instead of using the last frame in the string of duplicates, a frame will be created and used from a blend of all the duplicates in the string. This is useful for noise reduction. This option requires copy=true. Also, note that the blend frame is generated when the first frame in the string is requested. So random access into the middle of a string will not invoke blending. Play the clip straight through from a starting point for correct operation.

bool  debug = false

This parameter enables debug output to the DebugView utility. This utility can be downloaded from my web site as given below.

string  log = NULL

If debug=true and this parameter is set to a file name (or path/name), then the debug information will be logged to the file. The file is closed when the filter instance is destroyed, e.g., when VirtualDub is closed.

int  blocksize = 32

Determines the blocks size for analysis. The lesser block size, the greater sensitivity to small details change (and to noise).

Examples

Dup with default settings:

AviSource("Blah.avi")
#TODO

Changelog

Version      Date            Changes

v2.32a       2015/11/19      - Author: tartak
                             - Fixed access violation bugs (use of uninitialized cache_count and highest_sum2)
                             - Updated to FilterSDK api of AviSynth 2.6

v2.32        2014/11/11      - Author: Max73
                             - Show metric (cur-prev) when reading the first frame outside a set of duplicates
                             - Skip border blocks in worst block metric calculation, they are more noisy
                             - Added a simple motion metric to denoise decisions
                               old_metric       = SumAbsDiff[]         = Sum(Abs(Diff))         : more sensitive to high contrast any motion, less sensitive in low contrast motion, more noisy
                               lc_motion_metric = abs(SumSigDiff[])    = Abs(Sum(Diff))         : more sensitive to large motion in low contrast, less noisy (pos and neg noise cancels out)
                               new metric       = 32/BLKSIZE * (old_metric/2 + lc_motion_metric): highest_Sum refers to this now
                             - Show Largest sum block shown in overlay, for new metric (much brighter), and old one (little brighter)
                               settings (personal preference):  Dup(threshold=1.50, chroma=false, show=true, copy=true, maxcopies=20, blend=true, blksize=32, debug=false)

Archived Downloads

Version	Download	Mirror
v2.32a	[x86]: dup232a_src.7z [x64]: Dup1_x64.7z	[x86]: dup232a_src.7z [x64]: N/A
v2.30	dup230.zip	dup230.zip

• Note: blend=true does not work in Dup1_x64.

External Links

• rationalqm.us - Official Dup homepage.

---

Back to External Filters ←

---