Colorimetry

When playing back video content, several issues might arise. The luminance levels could be wrong, resulting in washed out colors (black displayed as dark gray, and white displayed as light gray). This is described in more detail on the luminance levels page. Another issue is a slight distortion in color (which often looks like a small change in brightness) and this will be described here.

This distortion arise because of an improper color space conversion at some point. Usually content is created (whether by a camera or a computer) in RGB, but for efficiency, stored as YCbCr (typically the terms YCbCr and YUV are used interchangeably, leading to some confusion; when referring to signals in video or digital form, the term "YUV" mostly means "Y′CbCr"⁽¹⁾). Besides RGB and YCbCr, there are many other colorspaces, which we won't discuss here. For an in-depth discussion, see Charles Poynton's Color FAQ.

The content must eventually be converted back to RGB for display. This means that two color conversions take place: RGB→YCbCr and YCbCr→RGB. The problem is that there are several ways of doing this conversion, involving different mathematical constants. If the two color conversions are not the same, the displayed colors won't be the quite the same as the original. So when converting RGB→YCbCr or YCbCr→RGB, the correct color conversion standard must be used, namely the one with which the content was encoded – Rec.601 or Rec.709 being the most common.

What are color conversion standards?

There are multiple ways of doing the YCbCr↔RGB conversion. The general conversion is given by

Y  = Kr*R + Kg*G + Kb*B
Cb = (B-Y)/(1-Kb)
Cr = (R-Y)/(1-Kr) = R - G * Kg/(1-Kr) - B * Kb/(1-Kr)

and the other way around:

R = Y + Cr*(1-Kr)
G = Y - Cb*(1-Kb)*Kb/Kg - Cr*(1-Kr)*Kr/Kg
B = Y + Cb*(1-Kb)

where

( 0.0 <= [Y,R,G,B] <= 1.0),
(-1.0 <= [Cb,Cr]   <= 1.0) and
Kr + Kg + Kb = 1

The red, green and blue coefficients (Kr,Kg,Kb) are standardized in specifications. The most common conversion coefficients are given below:

Standard	Kr	Kg	Kb
Rec.601/BT.601(2), SMPTE 170M(3)	0.299	0.587	0.114
FCC(2)	0.300	0.590	0.110
Rec.709/BT.709(2)	0.2126	0.7152	0.0722
SMPTE 240M(4)	0.212	0.701	0.087

How can I see if the correct standard is used upon playback?

That's a tough question, because it's hardly possible to notice if the wrong standard is used upon playback. In fact, it's hard to see any difference at all between them. Take a look at the following screenshot:

The upper picture is shown correctly (using Rec.601), and the lower one is decoded with the incorrect standard (Rec.709). What you notice is that the lower picture is more saturated (i.e., more colorful), but also that the red is shifted a bit to yellow. The other way around results in a less saturated clip, red shifting towards magenta and blue shifting to cyan.

So, how do you know then, which standard to choose upon playback? Well that's not always possible, but you have to know the origin of your file. Also, in some cases, that information can be stored in the header of the file. You should stick to the following guideline:

If your content is Standard Definition (SD) content use Rec.601; if your content is High Definition (HD) content use Rec.709 – unless the header of the content specifies otherwise. We will elaborate on this in the following sections.

Using color bars

As noted in the previous section, it's hard to see the effects of improper decoding or conversion. If the source includes color bars, your task is much easier:

		When decoding Rec709 color bars as Rec601, the Green bar is bright and over-saturated and the Red bar is dim.
		Properly decoded color bars.
		When decoding Rec601 color bars as Rec709, the Red bar is bright and over-saturated and the Green bar is dim.

Once you have trained your eyes to see this shift, you can usually spot a problem on any video frame with saturated reds and greens.

Should I correct anything when processing my content?

Yes, you should correct for it in some cases. It depends on your end format and how it is played back. In general, it is safe to assume that Rec.601 is used for Standard Definition content and Rec.709 for High Definition content upon playback. That means that often, media players will ignore any specific header information regarding colorimetry. It will be assumed here that you are processing with AviSynth.

Suppose you want to encode your content as Standard Definition content (that is, height of the encoded video < 720p (720 vertical lines)). If your source is MPEG-2 (therefore the colorimetry information can be stored in its header), get the ColorMatrix plugin, and create the following script:

# ColorMatrix does a Rec.709 -> Rec.601 conversion, if your source is Rec.709:
DGDecode_mpeg2source("D:\source.d2v", info=3)
ColorMatrix(hints=true, dest=2, interlace=false) # use interlace=true for interlaced content

Suppose you want to encode your content as High Definition content (that is, height of the encoded video >= 720p). Create the following script:

# ColorMatrix does a Rec.601 -> Rec.709 conversion, if your source is Rec.601:
DGDecode_mpeg2source("D:\source.d2v", info=3)
ColorMatrix(hints=true, dest=0, interlace=false) # use interlace=true for interlaced content

If the colorimetry information is not stored in the header (as with DivX/XviD for example), it is not really possible to tell which colorimetry was used to create it. All that you can do is assume Rec.601 for SD content and Rec.709 for HD content, and hope that the assumption is correct. In this case, you can't use the hints parameter of ColorMatrix, but you need to give the conversion explicitly using the mode parameter.

To correct a source that was badly processed at some point, use the following:

# overbright greens, faded reds: perform a 709->601 conversion 
ColorMatrix(source=0, dest=2)

# for RGB sources:
ConvertToYV24(matrix="Rec601").ConvertToRGB32(matrix="Rec709") 

# overbright reds, faded greens: perform a 601->709 conversion  
ColorMatrix(source=2, dest=0)

# for RGB sources:
ConvertToYV24(matrix="Rec709").ConvertToRGB32(matrix="Rec601") 

How can I use the correct standard upon playback?

Whether this is possible (provided that the wrong standard is used upon play back) depends on how your content is being played back. If you use a software player you need to check whether it has an option to correct it. If you use a DirectShow-based player (such as WMP or MPC) there are different possibilities which are explained below. If the renderer does the YCbCr→RGB conversion, you should keep in mind that (as found are out in a doom9 thread⁽⁵⁾):

• Windowed/renderless VMR7 and VMR9 use BT.601 for video < 720p (720 vertical lines)
• Windowed/renderless VMR7 and VMR9 use BT.709 for video >= 720p (720 vertical lines)

If you are using Haali's Video Splitter and its Renderer, you can choose the color standard.

Have a look at this doom9 thread⁽⁶⁾ for information about the different renderers.

Method 1: Adjusting graphics driver settings

TODO

Method 2: Convert to RGB32 with ffdshow

Forcing ffdshow to output RGB32 can help prevent colorimetry issues. Downside of this method is that doing this conversion in software increases CPU usage.

To force RGB32 output in ffdshow, you should uncheck all colorspaces except RGB32 on the Output page in ffdshow configuration. It is also recommended to enable "High quality YV12 to RGB conversion". On the RGB conversion page, you can choose which standard should be assumed, BT.601 or BT709. Choose the first for SD material, and the latter for HD video.

Method 3: Pixel shader in Media Player Classic

A pixel shader is a small program that runs on your graphics card and processes some graphic data. In this case each frame of your video.

Media Player Classic has a shader called "BT.601 -> BT.709". A shader which does the conversion the other way around is still not available. Use this when needed.

Some requirements for the pixel shaders in MPC:

• You need to use a compatible video renderer: VMR-7 (renderless), VMR-9 (renderless), or EVR Custom Presenter.
• Surface setting must be set to "3D surfaces".
• It requires some DirectX components that are not included with a default Windows installation. Run the DirectX Web Installer to get the required DirectX updates.

Method 4: Use AviSynth script in ffdshow video decoder

There are two ways to do this. The first one is to use the ColorMatrix plugin, as explained in one of the previous sections, and the other one is to convert to RGB using the appropriate matrix:

With the above setting, ffdshow will use AviSynth to convert your content to RGB using Rec.601.

What do the specifications say about which colorimetry is supported in a particular format?

VCD/MPEG-1

Part 2 of the MPEG-1 standard covers video and is defined in ISO/IEC-11172-2. It is heavily based on H.261. The colorimetry information is not written into the header.

DVD/MPEG-2

The DVD specs are not publicly available for free. The DVD specs should be a subset of the MPEG-2 specs (yes, that's a big assumption), and the latter is available for free; it says the following:

The older 1995 MPEG-2 spec (ISO/IEC 13818-2: 1995 (E)) says

“In the case that sequence_display_extension() is not present in the bitstream or colour_description is zero the matrix coefficients are assumed to be those corresponding to matrix_coefficients having the value 1.”

The value 1 stands for "1 Recommendation ITU-R BT.709". Note that in 1995 the first DVD titles came out, so when making those specs, the DVD didn't exist yet. In 2000, the MPEG-2 specs have changed with respect to this:

The newer 2000 MPEG-2 spec (ITU-T Rec.H262 (2000 E))⁽⁷⁾ says

“In the case that sequence_display_extension() is not present in the bitstream or colour_description is zero the matrix coefficients are assumed to be implicitly defined by the application.”

The matrix coefficients are described in "Table 6-9 – Matrix Coefficients". In other words if sequence_display_extension is not present, the colorimetry can be anything.

The ATSC standard

As noted by hkazemi on doom9⁽⁸⁾, the ATSC a_81 standard on page 18⁽⁹⁾ (section 7.3.4, 'Sequence Display Extension Constraints') talks about colorimetry and the assumptions to make when 'sequence_display_extension' does not tell you what to do:

“The values for color_primaries, transfer_characteristics, and matrix_coefficients shall be explicitly indicated in the sequence_display_extension. While all values for color_primaries, transfer_characteristics, and matrix_coefficients defined in Tables 6-7, 6-8, and 6-9 of ISO/IEC 13818-2 [16, 18] are allowed in the transmitted bit stream, it is noted that ITU-R BT.709 [27] and SMPTE 170M are the most likely to be in common use.

Note: Some previously-encoded legacy material may not have the colorimetry (i.e., color_primaries, transfer_characteristics, and matrix_coefficients) explicitly indicated in the sequence_display_extension, in which case the colorimetry is most likely ITU-R BT.709 for all formats except those formats with vertical_size_value=480, which are most likely to have colorimetry according to SMPTE 170M.”

MPEG-4/ASP

The colorimetry information is not written into the header.

MPEG-4/AVC

ITU-T Rec. H.264 (2005)/Amd.1 (06/2006) says

“When the matrix_coefficients syntax element is not present, the value of matrix_coefficients shall be inferred to be equal to 2. That is: Image characteristics are unknown or are determined by the application.”

The matrix coefficients are described in "Table E-5 – Matrix coefficients".

Application defaults

In Video Demystified⁽¹⁰⁾, the author Keith Jack, gives the following table
(as noted by ariga on doom9.org; see: Table 13.16, MPEG-2 matrix_coefficients Codewords)

Sequence Display Extension: 
Color_primaries
This optional 8-bit codeword describes the chromaticity coordinates of the source primaries, 
as shown in table. If sequence_display_extension is not present, or color_description = 0, 
the indicated default value must be used. This information may be used to adjust the color 
processing after MPEG-2 decoding to compensate for the color primaries of the display.
-------------------------------------------------------------------------------------
|    Color Primaries     |    Code     |          Application Default               |
|------------------------|-------------|--------------------------------------------|
| forbidden              |  0000 0000  |                                            |
| BT.709, SMPTE 274M     |  0000 0001  | MPEG-2, ATSC, DVB 25Hz HDTV, DVB 30Hz HDTV |
| unspecified            |  0000 0010  |                                            |
| reserved               |  0000 0011  |                                            |
| BT.470 system M        |  0000 0100  | DVD-Video 30 Hz                            |
| BT.470 system B, G, I  |  0000 0101  | DVD-Video 25 Hz, DVB 25Hz SDTV             |
| SMPTE 170M             |  0000 0110  | DVD-Video 30 Hz, DVB 30Hz SDTV             |
| SMPTE 240M             |  0000 0111  |                                            |
| reserved               |  0000 1000  |                                            |
| :                      |      :      |                                            |
| reserved               |  1111 1111  |                                            |
-------------------------------------------------------------------------------------

dragongodz claims on doom9⁽¹¹⁾ that this table should be interpreted as follows:

“Having finally heard back from a person who is in the professional encoder industry (no I will not say who), I have more reliable information.: "The table is actually the limitation of what you can use. In other words if you set the sequence_display_extension to Rec.709 for dvd it is illegal. That doesn't mean it will not play etc it just means that it is not supported and any player that strictly follows the specs is more likely just to ignore it as if no value was set at all.”

References

1. YCbCr (wikipedia.org)
2. Color conversions (avisynth.nl)
3. ColorMatrix documentation: Colorimetry (avisynth.org.ru)
4. Luma (video) (wikipedia.org)
5. ColorMatrix v2.3, page 24 (doom9.org)
6. Windowless, Renderless, VMR, Overlay - what do they mean? (doom9.org)
7. ITU-T Rec.H262 (2000 E) (itu.int)
8. HCenc 22 and Colorimetry (doom9.org; post #18: hkazemi)
9. A/81: Direct-To-Home Satellite Broadcast Standard (atsc.org)
10. Video Demystified by Keith Jack (amazon.com)
11. HCenc 22 and Colorimetry (doom9.org; post #19: dragongodz)
12. Color FAQ by Charles Poynton (poynton.com)