AvsInpaint

Latest revision as of 11:30, 28 October 2021

[edit] Description

Image Inpainting is the art of restoring destroied parts of an image by using information of valid parts of the image in a way, so that the human eye does not recognize the damaged areas (at least not at a first sight). In video processing image inpainting is often applied to videos in order to remove TV station logos. This plugin comes with the intention to provide a suit for the removal of logos, whether opaque or transparent. It provides algorithms for these tasks:

• Logo detection: An obstacle is analyzed closely after being masked roughly by the user. The exact color map of the logo is determined as well as its alpha mask.
• Deblending: Given the color image and alpha mask of an overlayed logo, the logo is removed where it is not opaque.
• Logo inpainting: Opaque parts of the logo are interpolated. A fast method is used which has been published recently by Folkmar Bornemann and Tom März and which is roughly described below.

Finally, a function for calculating the distance function of a mask is provided, which can be used to expand or inflate masks as well as to provide smooth blending masks.

[edit] Requirements

• [x86]: AviSynth+ or AviSynth 2.6
• [x64]: AviSynth+
• Supported color formats: RGB32, RGB24, YUY2, Y8, YV12, YV16, YV24

[edit] Syntax and Parameters

[edit] Inpaint Function

InpaintLogo restores masked parts of video frames by interpolation with non-masked parts of the frame. Each masked pixel will be inpainted by averaging surrounding pixels. The weights of this averaging are determined by calculating a special structure tensor that contains information about the directions of isophotes near that pixel.

InpaintLogo (clip Clip, clip "Mask", float "Radius", float "Sharpness", float "PreBlur", float "PostBlur", float "ChromaWeight", float "PreBlurSize", float "PostBlurSize", bool "ChromaTensor", float "PixelAspect", int "Steps")

clip   =

Input clip; the damaged source clip. All color spaces are allowed. If no Mask is provided, the source clip must be RGB32 and the alpha channel is used as mask (see Mask for details). Otherwise all color channels of Clip will be inpainted. The resulting clip will have the same properties as the source clip.

clip  Mask =

Mask defines the parts of the video which will be treated by the plugin. This clip may either have a YUV colorspace (the Y channel is used) or RGB32 (the alpha channel is used). All pixels of the source clip where this mask has values greater than 127 will be inpainted. The mask clip must have the same (spatial) dimensions as the source clip. Moving logos are allowed: If the mask changes in time, the inpainted region in the source clip will change accordingly. If the logo is fixed (hence all mask frames are equal), the mask clip should be trimmed to one single frame indicating a static logo to the function. Since each movement of the inpainting region causes a lot new calculations, inpainting static logos is much faster.

float  Radius = 5.0

Radius describes the neighborhood of a damaged pixel from where values are taken when the pixel is inpainted. Bigger values prevent isophotes being inpainted in the wrong direction, but also create more blur. (default: 5.0 Px)

float  sharpness = 30.0

Sharpness describes how faithfull the algorithm follows directional information contained in the structure tensor. Higher values can prevent blurring caused by high Radius values. (default: 30.0)

float  PreBlur = 0.5

PreBlur is the standard deviation of the blur which is applied to the image before the structure tensor is computed. Higher values help connecting isophotes which have been cut by the inpainting region, but also increase CPU usage. PreBlur=0.0 disables pre-blurring. (default: 0.5 Px)

float  PostBlur = 4.0

PostBlur is the standard deviation of the blur which is applied to the structure tensors before they are used to determine the inpainting direction. Higher values help gathering enough directional information when there are only few valid pixels available, but also need lots of CPU cycles. (default: 4.0 Px)

float  ChromaWeight = 0.0

ChromaWeight describes how chroma channels are taken into account when the structure tensor is build. After tensors for each color channel are set up, tensors are added, where luma is weighted with 1.0-ChromaWeight, and all chroma tensors together with ChromaWeight. Tensors not needed are not computed. This parameter is ignored for RGB color spaces: Each channel is weighted equally then. (default: 0.0)

float  PreBlurSize = 2×PreBlur

float  PostBlurSize = 2×PostBlur

PreBlurSize and PostBlurSize describe the sizes of the blurring kernels used for pre blurring and post blurring, respectively. (default: 2×PreBlur and 2×PostBlur, resp.)

bool  ChromaTensor = false

When ChromaTensor is set to true, all structure tensors will be computed again before a chroma channel pixel is inpainted (after the corresponding luma pixel has been inpainted). Chroma values are always inpainted after a luma value has been inpainted in the same position. This can be especially useful when chroma channels have a lower resolution than luma channels (e.g. YUY2, YV12, YV16), i.e. the chroma pixel is not in exactly the same position in which the just inpainted luma pixel is. This argument is ignored for RGB color spaces. (default: false)

float  PixelAspect = 1.0

PixelAspect is the ratio between width and height of a pixel. E.g. for 16:9 DVD movies with a resolution of 720x576, one would use PixelAspect=(16.0/9.0)*(576.0/720.0)≈1.422. This value distorts the circle defined by Radius as well as the blurring kernels and kernel extents and modifies the inpainting order of pixels. (default: 1.0)

int  Steps = -1

If Steps is non-negative, only the first Steps pixels (in the inpainting order which is determined by the distance of the pixels to the border of the inpainting region) are inpainted, all remaining pixels will be untouched. This is mainly usefull for visualizing how the function works and for debugging purposes. If Steps is less then -1, a copyright message will be displayed (along with an error message). (default: -1)

If the mask is provided as a separate clip (Mask argument), all color channels available in the source Clip will be inpainted, including the alpha channel, if the source is RGB32. In most cases this is not desired since it needs additional CPU time, and information from the alpha channel is also used for inpainting the color channels which might tamper the results. To avoid this, use ConvertToRGB24 first when working with RGB32 clips.

The following table shows which combinations of color spaces are allowed and which color channels will be inpainted and returned.

YUV color spaces (YUY2, YV12, YV16, YV24) are usually better considering speed and quality (the latter because RGB colors easily become distorted when different channels are treated differently). Yet, color spaces with subsampled chroma channels (which applies to YUY2,YV12 and YV16) may expose difficulties when the inpainting mask does not respect the subsampling: If some of the chroma pixels are partly masked (i.e. some of the corresponding luma pixels are masked for inpainting, some are not), the respective chroma pixel will be masked for inpainting, although it might be possible there are not enough non-masked or yet inpainted chroma pixels around when it's its turn to be inpainted. In such chase, the chroma pixel will simply left untouched which will generate a reasonable result in most cases. However, to avoid such problems make sure each partly masked chroma pixel has a neighboring non-masked chroma pixel (e.g. by forcing your mask on the chroma grid, or by inflating and then deflating your mask by at least three pixels).

There is no explicit mechanism that deals with parts of the picture which are destroied but which should not or cannot be recovered (like black bars on the top and the bottom of the movie). If you cannot Crop away these parts, mask them for inpaint. Otherwise the function would consider them as valid data and use them for inpainting (black bars usually create a big spurious edge!).

It is in general a hard (or even impossible) task to find parameters for good inpainting a fixed mask for a long movie. The best choice of parameters depends on several factors, like

• the shape of the mask
• the amount of “good” pixels available near the mask border
• the complexity of the underlying pictures
• ...
• your personal flavor

In most cases it is a good idea to pick some frames of the movie, stack them together with Stack*al and watch the changes with ScriptClip as you change the parameters.

Beware: This algorithm has originally been developed for inpainting pictures, not movies. No temporal information is used for inpainting. You might observe strong discontinuities in time (jumping of shapes created by the algorithm). See included documentation for some details on how the algorithm works.

[edit] Deblend Function

DeblendLogo is roughly the inverse function of Layer: If a clip (the logo) is blended onto another clip (by using a given alpha mask), DeblendLogo tries to undo this and to restore the original clip.

DeblendLogo (clip Clip, clip Logo, clip "Alpha")

clip   =

Input clip; Clip is the source clip with the logo which you want to deblend. It can have any color space. The resulting clip will have the same properties.

clip   =

Logo contains the logo you want to remove from the source clip. The logo may be moving, but a static logo (Logo and Mask having only one frame) results in faster processing since some values can be pre-computed. If Mask is not provided, Logo must be RGB32, Clip RGB24, and the alpha channel is used as mask. Otherwise, Logo must have the same color space as Clip. painting region causes a lot new calculations, inpainting static logos is much faster.

clip  Alpha =

Mask contains the transparency mask of the logo you want to remove. This clip may either have a YUV colorspace (the Y channel is used) or RGB32 (the alpha channel is used). Pixel values of 0 mean full transparency (the source clip is returned unchanged here), values of 255 means full opacity (the logo cannot be deblended in these regions, the resulting pixels will have undefined values). The interval [0,255] also applies to YUV masks, so do not forget to add Matrix="PC.601" to the converting function when converting RGB masks to YUV.

The following table shows which combinations of color spaces are allowed and which color channels will be deblended and returned.

[edit] Analyze Function

AnalyzeLogo tries to determine the color image and the alpha mask of a static logo which is part of a given source clip. The result of this function can be used to remove transparent parts of the logo with DeblendLogo and opaque parts with InpaintLogo.
DeblendLogo is roughly the inverse function of Layer: If a clip (the logo) is blended onto another clip (by using a given alpha mask), DeblendLogo tries to undo this and to restore the original clip.

AnalyzeLogo (clip Clip, clip "Mask", bool "ComputeAlpha", float "DeviationWeight", float "SubsamplingWeight")

clip   =

Input clip; Clip is the source clip containing the logo to be analyzed. This clip may have any color space. If Mask is not provided, Clip must be RGB32 and the alpha channel is used as mask. In general the resulting clip will have the same properties of the source clip except being twice as high: The upper part contains the color image of the logo, the lower part the alpha mask. If no separate mask is provided (the mask being the alpha channel of the source clip), the resulting clip will also be a RGB32 clip with the same height and the logo alpha mask being placed in the alpha channel.

clip  Mask =

Mask defines the parts of the video where the logo is suspected to be. This clip may either have a YUV colorspace (the Y channel of the first frame is used) or RGB32 (the alpha channel of the first frame is used). All pixels of the source clip where this mask has values greater than 127 will be treated as “possibly damaged by the logo”, all other pixels are used as reference (they are not ignored). If ComputeAlpha is set to false, Mask is expected to contain the alpha mask of the logo (obeying the same conventions as described above for DeblendLogo), and the function will only determine the color part.

bool  ComputeAlpha = true

When ComputeAlpha is set to true, the analyzer tries to determine both the alpha mask and the color part of the logo, based on where the Mask clip has values greater than 127. If it is set to false, Mask must already contain the alpha mask of the logo, and only the color part is computed. In both cases, the mask will be returned. (default: true)

float  DeviationWeight = 0.5

float  SubsamplingWeight = 0.5

DeviationWeight and SubsamplingWeight define the way the alpha mask of the logo is estimated. Since video clips contain multiple color channels, the alpha mask is first computed for each channel separately, the final mask then is a weighted mean of those. The weights are a product of the alpha values (each alpha value is weighted with itself) and of the degree of subsampling (0.5 for YUY2, 0.25 for YV12) of each channel. These arguments define the exponents of those factors — 0.0 means the weight is not used at all. A negative value for one of these arguments will show a copyright message (along with an error message). (default: 0.5 for both)

If the mask is provided as separate clip (Mask argument), all color channels available in the source clip will be analyzed, including the alpha channel of an RGB32 clip. In most cases this is not desired since it needs additional CPU time, and information from the alpha channel is also used for estimating the overall alpha values. To avoid this, use ConvertToRGB24 first when working with RGB32 clips.

The following table shows which combinations of color spaces are allowed and which color spaces will be analyzed. Note that the clip returned will be twice as high as the source clip if a Mask is provided.

As with InpaintLogo, there is a problem with subsampled chroma channels. The alpha mask values of two (luma) pixels with highly different alpha values which share a common chroma pixel cannot be reliably estimated from their chroma channel values. This effect usually produces some dirt in the estimated alpha mask and subsequently in the color mask. Such effects, when observed, can be avoided (besides by converting to RGB24) by increasing the SubsamplingWeight.

There is no explicit mechanism that deals with parts of the picture which are destroyed or otherwise not part of the clip (like black bars on the top and the bottom of the movie). If you cannot Crop away these parts, mask them for analyzing. Otherwise the function would consider them as valid data and use them for reference when estimating the logo's color and alpha mask.

[edit] Distance Function

DistanceFunction computes the distance function around or inside a given set of pixels. E.g. it can create a frame, where each pixels value is the distance of the respective pixel to a given set of pixels. This is typically used to inflate or deflate masks, or to create masks with smooth borders out of binary masks.

DistanceFunction (clip Clip, float "Scale", float "PixelAspect")

clip   =

Clip is the source clip. This clip must either be a YUV clip, or RGB32. For YUV clips, only the Y channel will be altered, for RGB32 clips the alpha channel will be changed.

float  Scale = 1.0

Scale scales the gray values around the pixel set under consideration. For positive Scales the significant area for distance calculation are all pixels with values greater than 127. This area will be turned white by this function. Each pixel outside this area with distance d to the area will be assigned the value 255-d×Scale (or 0, if it would be negative). If Scale is negative, the color channel will be inverted before and after computation. If Scale is zero, the image is simply made binary (roughly the same as Levels(127,1,128,0,255)). (default: 1.0)

float  PixelAspect = 1.0

PixelAspect is the ratio between width and height of a pixel. E.g. for 16:9 DVD movies with a resolution of 720x576, one would use PixelAspect=(16.0/9.0)*(576.0/720.0)≈1.422. The result of this function is distorted according to this value. A negative value will show a copyright message (along with an error message). (default: 1.0)

[edit] Example

TODO

[edit] External Links

• GitHub - Source code repository.

Back to External Filters ←