The Ultimate Codec Guide

How data is stored in a multimedia file

Files on a computer (or on an optical disk like DVD or BluRay) need to contain more than 1 type of data.  A typical movie will include at least 1 video “stream” and one audio “stream”.  Most movies include multiple languages and subtitles, each of which requires an additional stream.  Each of these streams is effectively a file in its own right, but they are all stored together within a “container” which also starts each of them at the right time (subtitles don’t start immediately at the start of a movie for example) and keeps them in sync as well as storing meta-data about each of them – i.e. what language they are, what framerate and resolution the video is, and what compression standards have been used.

Video streams

Video streams contain only the video part of the movie.  They are compressed using a particular system and there should be meta-data about their resolution, framerate, if they are interlaced or progressive and details of the encoding system used.

Audio streams

Audio streams contain only the audio part of the movie.  There are usually multiple audio streams included with the movie and each has its own stream.  They can be compressed in a range of ways or in some cases they are uncompressed raw audio.  There should be meta-data about their bitrate, resolution, language, number of channels and details of the encoding system used.

Other streams (e.g. Closed Captions)

Many movies contain other streams, most commonly closed captions (also known as subtitles).  These come in several formats, but are typically just text with time stamps.  They are so small relatively that no compression is used.  There should be meta-data about their language.

Containers which bundle all the streams together

All of the streams that comprise the movie are bundled together and kept in sync by a container.  The container should contain (and provide) all the meta-data about each stream.  The container also enables the collection of streams to be stored as a single file.  The most common container formats are VOB (used by DVD) and M2TS (used by BluRay).  Other common container formats on PC are TS, MPEG, AVI, MKV, WMV, WTV, DVR-MS and MP4.  The obsolete HD-DVD format used EVO containers.

Encoding

Why AV streams are encoded

Each pixel of an image has at least an 8bit value for each of the 3 primary colours.  So each pixel requires 3 bytes of information (8bits in a byte).  A full-HD video frame is 1920×1080 pixels.  So a standard single frame of uncompressed HD video will require 6,220,800 bytes (5.93Mbytes) of data.  If a typical movie is 2 hours long, and has 30 frames per second there will be a total of 216,000 frames in the movie.  The total movie size, if uncompressed, would be 1.2 Terabytes.  Modern Double-layered BluRay disks can hold only 50 Gigabytes so a movie would span 25 disks.

Both audio and video streams can be compressed and there are many different systems of compression.  Both can be “losslessly” compressed – this means that no information is lost.  For example, if a 100×100 pixel image is pure red then instead of describing every pixel in the image as red (which would take 100x100x3 = 30,000 bytes), the compression system might allow us to just say “the square is 100 by 100 and every pixel is red”.  This system is clearly much more efficient, and we still get an identical red square.  Fortunately no movies involve a constant, single colour image, and audio is rarely a monotone.  This means there are limits to how much we can reduce the size of an audio or video stream with lossless compression.  Many lossless audio compression systems can achieve approximately  a halving of the raw audio data which in some instances is sufficient.  A movie with 2 hours of audio for 6 channels (commonly called 5.1, meaning 2 front, 2 rear and 1 centre channel plus 1 subwoofer) requires approximately 3GBytes of raw audio, or 1.5Gig of lossless audio.  For a 50Gig BluRay disk this represents 3% of the total capacity and lossless or even raw audio formats are commonly used.  For older 9Gig DVDs this represented nearly 20% of total capacity which was too large and consequently the audio on DVDs is always compressed in a way that does lose some data.

The common MP3 format is a good example of what can be achieved by compression that does lose data (otherwise known as lossy compression).  A raw audio file can be reduced to around 15% of its original size using typical MP3 compression.  The details of this process are complex, but in essence lossy compression systems attempt to identify data which the human senses will not notice is missing, and discard it.

All standard video compression schemes are lossy – lossless compression of video is simply not sufficient to make the data size manageable.  Video compression schemes have a complicated 3 way trade-off.  In an ideal world the scheme will produce a video file with the smallest data size for the lowest quality loss.  However, these types of files require a lot of calculation to decode – to restore to an uncompressed format when they are output.  The most powerful schemes require so much processing that they could not be played using the hardware available at consumer prices, and so a third compromise is necessary.  As consumer hardware gets more powerful, new compression schemes are introduced that increase quality or reduce data size at the cost of requiring greater processing.  As greater processing is now available this is an appropriate evolution.

Common video formats

The most common video compression system is MPEG2.  This system is the standard used on all DVDs.  MPEG2 is also used on some BluRay disks.  Other BluRay disks use a system known as AVC or H.264.  The 3^rd and final third format approved for the BluRay standard is known as VC1.  VC1 is a variant of Microsoft’s WMV3.

Other common formats are DivX, XviD, Flash, VP6, MPEG1, WMV1 and WMV2

Common audio formats

The most common type of audio is in fact raw audio as this is used on Audio CDs.  This is stored as .wav files on PCs.  The MP3 format is another very widely used format.  The vast majority of movies use multichannel sound, rather than stereo and have traditionally been encoded using one of the Dolby schemes (usually called AC3) or one of the DTS schemes, the most common of which are Dolby Digital and the original DTS.  There are several newer formats, some of which are quite common on BluRay disk including Dolby Digital Plus (E-AC3), Dolby True HD and DTS-HD Master Audio (DTS-MA).  Some BluRays also include raw multichannel audio, commonly called LPCM.

Other common formats include MP1, MP2, AAC, Ape, FLAC and Ogg.

File types (file extensions)

File extensions and video streams

File extensions are related to containers, but imply nothing about the streams within the container.  A common misconception is that the file extension belongs to “the video” – it doesn’t!  For example, the file extension .m2ts can contain video streams encoded with mpeg2, h.264 or VC1, three completely different encoding systems.  Equally, a h.264 stream can be found within all of the most common container formats except VOB.  In some rare circumstances, some of the streams can be inferred from the container (for example VOB files contain mpeg2 video streams) but the filetype doesn’t always identify the container.  Filetypes are intended to identify the container, however, whilst in some cases this works reliably, there are others where the correlation is not perfect.  For example, M2TS and TS are largely interchangeable, although BluRay disks only use M2TS, and live streams are normally stored as TS.

File icons

File icons tell you very little about a file.  Icons are matched to extensions by Windows but each file type doesn’t have its own particular icon.  The icons are frequently changed when software is installed or the file is played back in a different player.  It would be easy to make Windows show the same icon for every type of file on the system!  The icons normally only tell you what Windows thinks the default player software for the file is.

How Windows manages playing multimedia

Windows automatic behaviour

When a file is played on Windows, most player software simply lets Windows handle the playback (Media Player Classic Home Cinema is an exception).  Windows automatically selects an appropriate splitter to “split” the container, and then selects the appropriate decoders to decode each of the streams.  Each decoded stream is then sent to a final stage known as a renderer.  The splitter, decoders and renderers are known as filters and are sometimes referred to as codecs.  The term codec has several common uses, but is generally used to mean a program which can carry out either encoding or decoding.  In this guide we will use the term codec to refer to complete software packages, which may contain multiple filters and support multiple stream types.  Each filter is a self-contained unit, usually stored inside an .ax file in the system32 folder.  The process of selecting the splitter, decoders and renderers is known as building a graph.  There are utilities available which allow you to see what graphs Windows will build for a particular file, as well as building custom graphs to test playback

Renderers

Renderers are the final stage of the graph; and are the interface between the decoded streams and the hardware that outputs the stream – for example the video card drivers, or the sound card drivers.  They manage the way the output fits into the Windows system.  So for example you may play back a video in a window (as opposed to full screen) and also have another program open.  Windows needs to how big to display the video, and where on the screen.  You may also move another program or window partially in front of the video and Windows needs to know to discard this part of the video.  This is all managed by the renderers.

Hardware decoding

Historically all decoding was done by the CPU.  The CPU is designed to be very flexible, which means it can undertake any task, but is not especially efficient at anything.  The increase in HD video content which requires substantial processing to decode has led to a demand for more efficient video decoding hardware.  All modern video cards now include some level of “hardware decoding” of the 3 BluRay encoding standards (mpeg2, h.264, VC1).  The level of support varies but with each new generation of hardware more standards are supported.  The latest generation of CPUs from Intel now also include dedicated decoding hardware which frees the main parts of the CPU for other tasks, and reduces power consumption.  Dedicated GPU and CPU hardware decoding can only be utilised if the codec used for decoding supports hardware decoding.  The level of support for this feature is variable, but it is growing, and several of the free codecs provide good support.  The codecs Microsoft provides with Windows also offer some support.  The system of hardware acceleration is commonly known as DXVA (DirectX Video Acceleration).  Many recent dedicated hardware decoders also support various image enhancing features.

In many cases, hardware decoding is incompatible with the use of subtitles, however, DXVA with subtitles is possible via the free open-source FFDshow tryouts.

Raw stream output

In some cases no decoder is required.  Some audio on BluRay disks is uncompressed and can be sent directly from the splitter to the renderer without need of a decoder.  In addition to this, many home theatre amplifiers include digital connectors (SPDIF or HDMI) and have their own hardware decoders for the 2 leading systems: Dolby and DTS.  If you have an amplifier which supports hardware decoding you may wish to leave the audio stream encoded and send the encoded stream to the amplifier for decoding.  To do this the stream from the splitter must be sent to the appropriate hardware (either a sound card in the case of SPDIF, or the video card for HDMI).  This stream must be sent in the correct way so the hardware knows to output it over the digital connection.  To do this still requires a filter, but the filter will not decode the stream.  The most common codec used for this purpose is called ac3filter.

Video is never normally raw and so always requires a decoder filter.  It must also pass through the renderer.  Subtitle information is also passed to the renderer for rendering on top of the video.

New codec system in Vista

The system Windows uses to manage all these filters is called DirectShow.  The vast majority of filters are designed to conform to the DirectShow standards.  However, with Windows Vista, Microsoft introduced a new system called Media Foundation.  This new system runs in parallel with DirectShow, but takes precedence.  Microsoft has include Media Foundation codecs in all Windows releases since Vista, and when a file is played back, Windows will try to use the Media Foundation codecs to playback the file.  This has made it more difficult to use alternative codecs.

Managing the codec system in Windows

64bit systems require twice the work

Traditionally Windows was a 32bit operating system.  However, since Vista many computers have been supplied with a 64bit edition of Windows.  When operating on a 64bit system, 64bit codecs are required.  However, this is not the end of the story.  64bit Windows still includes full support for 32bit via emulation, and many parts of Windows are still 32bit.  In Windows 7 for example, Windows Media Center is 64bit (and uses the Media Player 64bit engine behind the scenes).  If you load Windows Media Player itself however, it operates as 32bit.  This frequently causes confusion as something that works in Media Player may not work in Media Center or vice versa.  On 64bit systems it is strongly recommended to install and configure both 64bit and 32bit codecs where possible.  On 64bit systems, 64bit applications install to the Program Files folder, and 32bit applications install to the Program Files (x86) Folder.  Some codecs also install application extensions into the Windows system folders.  64bit ones will be found in the system32 folder, and 32bit ones will be found in the SysWOW64 folder.  In addition to file locations there are also extra sections of the registry.  Entries related to 64bit software will be found in the normal locations (e.g. [HKEY_CURRENT_USER\Software], [HKEY_LOCAL_MACHINE\Software] and [HKEY_CLASSES_ROOT]).  Their 32bit equivalents will be found in [HKEY_CURRENT_USER\Software\Wow6432Node], [HKEY_LOCAL_MACHINE\Software\Wow6432Node] and [HKEY_CLASSES_ROOT\Wow6432Node].

Associating files and streams with codecs

Windows uses a GUID to identify filters, rather than their names or filenames.  Each filter has its own GUID and these are stored in the registry.   Streams are also identified by GUIDs (but file extensions are not!) Various registry settings can (and usually do) specify which splitter to use for each file extension, and in some cases (but not commonly) which filter to use for each type of stream.  When installed, each filter provides a list of the GUIDs of stream types it can decode.  A filter is selected from the available (and compatible) ones available.  This selection is carried out based on the filter’s “merit” (a value they are given by Windows).  These “merit” values can be changed to make Windows pick the filters you want.  Similarly, the association between file extension and splitter can be changed to make Windows pick the splitter you want to use for each file extension.  Finally, the merits of “renderers” can also be changed, to ensure Windows uses the renderer you want.  This merit system applies to DirectShow codecs.  If there are compatible Media Foundation codecs available Windows will always prefer them.  If this is the case you may need to disable some or all of the Media Foundation codecs.

Changing Merits

The merits of codecs can be changed, either via the registry (which is unnecessarily difficult) or with various free utilities.  A company called monogram have helpfully produced an improved version of a Microsoft utility called graphedit.  Monogram GraphStudio is available as both 32bit and 64bit.  When run as administrator, these utilities can be used to change the merits of codecs as shown below.  However, some Microsoft codecs cannot, by default, have their merit changed, as the administrator user does not have permissions to the appropriate values in the registry.  This can be overcome by changing the permissions to these registry paths.  If you run into problems changing the merits of any codecs, refer to the section below on taking ownership and changing permissions.

Over-riding merits for specific stream types

The merit system can be bypassed with specific associations between stream types and decoders.  If for example you want the ffdshow codec to decode most streams, and so set it with the highest merit, but want to use the mpc-hc mpeg2 decoder, you can specify in the registry an association between the GUID for mpeg2 video streams and the GUID for the mpc-hc decoder.  This is done by creating a String (REG_SZ) in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DirectShow\Preferred] (for 32bit codecs on 64bit Windows the registry path is [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\DirectShow\Preferred]).  The name of each string should be the GUID of the stream type you want to specify a codec for.  The value of the string should be the GUID of the codec you wish to use.  Make sure you include curly brackets { } at the start and end of each GUID.

Splitter preferences

Splitter preferences are best edited in the registry.   This is a fairly simple operation to complete.  Each file extension has a key under [HKEY_CLASSES_ROOT\Media Type\Extensions] (32bit splitters on 64bit systems will be under [HKEY_CLASSES_ROOT\Wow6432Node\Media Type\Extensions]).  In each key there should be a String Value (REG_SZ) called Source Filter.  This will be set to the GUID of the relevant filter.  You can change the GUID values to change the splitters.  You can also add new keys for other file types.  The GUID values for splitters can be found in GraphStudio.  The GUIDs for the most common splitters are located in the table at the bottom of this article.

Media Center preferred decoders & Live TV

There is another registry setting that tweakers must be aware of.  Windows Media Center has 2 keys which apply only to it, which over-ride merits and all other preferences.  These keys are found in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Media Center\Decoder].  In my experience if these are set to any other codecs, Media Center cannot decode live TV.  If they are left as they are, Media Center will use these codecs to decode a lot of video files that you may not want to decode with these filters.  To bypass this problem, the 2 keys should be removed and the merits of Microsoft DTV-DVD Video Decoder and Microsoft DTV-DVD Audio Decoder should be raised to at least 00800000 (by default they are set too low for Windows to ever use them).  If you want to use a different codec for mpeg2, simply set its merit higher than 00800000 and Windows will use your preferred codec, except for live TV, where it will recognise that your chosen codec is incompatible and will instead use the DTV-DVD decoders that work with liveTV.  Before the merits for the DTV-DVD decoders can be changed the permissions of the registry path that stores their settings must be changed.  The paths in question are [HKEY_CLASSES_ROOT\CLSID\{083863F1-70DE-11D0-BD40-00A0C911CE86}\Instance\{212690FB-83E5-4526-8FD7-74478B7939CD}] and [HKEY_CLASSES_ROOT\CLSID\{083863F1-70DE-11D0-BD40-00A0C911CE86}\Instance\{E1F1A0B8-BEEE-490D-BA7C-066C40B5E2B9}].  To change the permissions of these paths see the section below on taking ownership and changing permissions.

Taking ownership & changing permissions

To complete some changes it is necessary to change the permissions of some files and some registry entries.  Before changing permissions it is often also necessary to “take ownership”.  To take ownership of files right click on a file and select properties.  Select the security tab and click on advanced.  Next, click on the owner tab and click on edit.  Select administrators in the list which appears and click ok.  Click ok twice to close the file properties.  Then right click and re-select properties.  And click on the security tab again.  Click the edit button and select Administrators and tick full-control.  Click ok and ok again.

The procedure for changing permissions in the registry is similar.  Open regedit and browse to the key you wish to change.  Right click and select permissions and then click on Advanced.  Select the owner tab, and click on Administrators to highlight it.  If it is present, tick the box which says “Replace owner on subcontainers and objects” and click ok.  In the security window that you have been returned to, highlight administrators, and tick “Allow” to full control, and finally click ok.

Disabling Media Foundation partially or fully

To use DirectShow codecs to decode formats supported by Media Foundation it is necessary to disable part or all of Media Foundation.  The easiest way to do this is to use the Win7DSFilterTweaker utility.  You can also disable Media Foundation manually – to do this requires several Windows files to be renamed or moved.  Before they can be renamed or removed you will have to take ownership and change their permissions.  These files are found in the System32 folder (32bit files on 64bit systems are found in the SysWOW64 folder).  To disable Media Foundation generally rename or move the file mfds.dll.  It is also possible to disable specific Media Foundation codecs.  mp3dmod.dll is the Media Foundation MP3 decoder, mp4sdecd.dll and mp43decd.dll are Microsoft Mpeg4 decoders, msmpeg2adec.dll is the DTV-DVD Audio Decoder and msmpeg2vdec.dll is the DTV-DVD Video Decoder.  Do not disable these last 2 if you wish to view Live TV in Media Center.

Known compatibility problems

Interlaced VC1

Currently there are very few codecs which can correctly decode interlaced VC1 content, such as that found on some BBC documentaries (for example Life and Planet Earth).  The main open source codecs, ffdshow and gabest, are both built on the open source project ffmpeg.  Ffmpeg currently doesn’t support decoding interlaced VC1.  Hopefully this issue will eventually be resolved by further development of ffmpeg.

Reference Tables & Links

Decoders

Splitters

Renderers

Stream GUIDs

Utilities

Useful Links

• Alf’s Video Codec Collection
• The Gizmodo guide to every video format you could ever need to know