Windows Media Quality Check

About a week ago Microsoft finally released their long awaited Windows Media Player - WMP - version 7.0 and a beta of the Windows Media Encoder 7.0 which can encode to the newly introduced formats that WMP7 can play. Microsoft claims better quality at even smaller sizes but so far I have yet to find any good review on the encoder and since I'm interested in video compression technology I decided to go ahead and do my own testing. What was initially intended to be a test of the new formats only turned out to a full-fledged quality control article about almost all Microsoft video compression codecs: Windows Media Video - WMV - version 1, 3 and 7 as well as DivX which is based on WMV3.

Test setup

I think I've found the almost perfect testing scene. I took a 7 minute (10'000 frames at 23.976fps - since it's a progressive source) clip from "The Matrix" starting with Agent Smith interrogating Neo and ending with Trinity blasting the Agent's head on the roof of the Police station. This clip is a real test for any encoder: Lots of very fast moving scenes, fast pans, shiny marble surfaces, a huge explosion, lots of fight scenes and lots of debris flying around. I encoded this scene in FlaskMpeg 0.594 at the following settings: 640x277 - respecting the aspect ratio but cutting the black bars away, Huffyuv 2.1 at it's default setting - this is a loss less video codec for those who don't know it - and RAW PCM audio at 48Khz sampling rate. The whole clip was about 2.1GB large. Then I loaded the clip in the new Windows Media Encoder and in VirtualDub (to encode to DivX). I always used Windows Media Audio V7 at 128kbit/s at 48Khz for the Windows Media Encoder tests and the Radium MP3 codec at 128kbit/s, 48KHz for the DivX tests. I run 3 series of tests in WME: one at 1MBit/s overall bitrate, one at 1.5MBit/s and one at 2.5MBit/s. I also tested 5MBit/s for WMV7 just to see the difference. These overall settings resulted in the following real bitrate settings: 872kbit/s (1MBit/s), 1398kbit/s (1.5MBit/s), 1848kbit/s (2.5MBit/s) and 4872kbit/s (5MBit/s). You can read the whole stats as well as the size of the finished files in this file...  The DivX 2500Kbit/s file has been encoded at the same bitrate as it's equivalent in WM7 format. Furthermore I made a DivX High motion clip at the max bitrate of 6000Kbit/s and a DivX Low Motion clip at 910Kbit/s.

I then reduced my screen's resolution to 640x480 and watched the movie at full screen - since the horizontal resolution matches the one of the movie there's no stretching involved resulting in the best possibly quality but large enough to easily spot any encoding errors if there are any. Of course I used the latest drivers for my graphics card (an Elsa Erazor 3 Vivo) and at 32bit colors to avoid dithering.

A remarkable discovery

Even without watching the generated clips there was one interesting fact I noticed: The encoded files at the same bitrate had almost exactly the same size, no matter what codec I used. This only applies to the files generated in Windows Media Encoder, curiously the file generated in VirtualDub using the DivX codec was smaller. I'm a bit unsure whether this is due to the different audio codec, the bitrate settings or these new codecs actually encoding worse than DivX. Since audio codecs are supposed to be constant bitrate and since the WMV3 codec should be more or less DivX's equivalent I think the bitrate settings are at fault here (remember I didn't set 1/1.5/2.5MBit/s in DivX but rather the expected bitrate that the Windows Media Encoder showed). Since all these Microsoft codecs are clearly VBR - you can see that they never even got close to the bitrate I set except for the bitrates below 1MBit/s - I assume that the codec will try not to cross the bitrate settings if not for a really bitrate-demanding scene. I've actually been watching the encoding process a few times and the bitrate almost never crossed the limit. The 5MBit/s clip was most interesting... in the elevator scene I noticed that at the very first the bitrate was about 1.2MBit/s then gradually increased to 2.5MBit/s when the elevator was speeding down and then when the explosion occurred it literally jumped up to 4MBit/s and at the end of the scene it was up to 5.5MBit/s.

Another interesting fact is that the 5MBit/s clip actually only has a slightly higher average bitrate than it's 2.5MBit/s counterpart - the difference is less than 300KBit/s although in several scenes the difference was really the 2.5MBit/s one could expect - so clearly once again this is a very clear proof that these codecs are VBR. This makes these codecs really interesting for quality freaks as me. We can actually increase the target bitrate significantly - say from 2MBit/s to 3MBit/s - without a significant increase in size. The clip I used in my tests is a really bitrate-demanding one and does not represent the overall situation in the movie - although even here you can see that increasing the bitrate does not really make the file much larger - this, of course, only applies to high bitrates of about 1.8+MBit/s settings. In lower settings the codec does not have enough "headroom" and therefore increasing the bitrate will increase the filesize as much as it theoretically should.

So to make everything crystal clear: WMVX/DivX is VBR at it's purest. The codec will try not to go over the bitrate you set in any case and the codec only uses so much bitrate for a certain scene. Dialogue scenes and scenes that contain almost no action will never use even 2MBit/s whereas an action scene takes as much bitrate as you initially set the it to.

The results

Despite Microsoft's propaganda I wasn't able to find any significant difference between the new and the old codecs. I've watched all the clips at the same bitrate after each other and compared them to the original clip. The files encoded in WMV7 looked exactly as their counterparts encoded in WMV3, DivX low motion and WMV1 - the MPEG 4 ISO codec. What was remarkable, tough, was that WMV3 showed some really nasty compression artifacts. I'm not sure whether Microsoft actually changed anything at that codec - I assumed that it was the same one included in Windows Media Encoder 4.1 and the same that DivX is based on. However only the WMV3 codec showed these compression errors and they even increased in number when I increased the bitrate. Here's an example - taken at 2.5MBit/s :

Also.. the encoding speed was not significantly different. It always took about 16-18 minutes for a clip to be finished, regardless of what bitrate I set and regardless of what codec I used.

Now get ready for some huge pictures: I've taken 4 screenshots of scenes that clearly show the weakness of a codec. The pictures are in the following order: Huffyuv, 5MBit/s WMV7, 2.5MBit/s WMV7, 1.5MBit/s WMV7 and 1MBit/s WMV7. I'm sorry for the lack of really frame-accurate comparisons but the problem with these WMV codecs is that there's no video editing program that can load these files. Therefore I had to play the movies in Media Player and press Print Screen at the appropriate time to take a screenshot. I hope that in the future Microsoft will allow us to use their codecs in any video editing program...

Ok here we go..

Huffyuv uncompressed. Note that the debris flying around is really clearly visible and the walls are still detailed.

WMV7 5MBit/s .. almost no difference.

WMV 2.5MBit/s

WMV7 1.5MBit/s

WMV7 1MBit/s

Can you see how the picture gradually gets blurry and loses detail? Let's get on to scene two, shall we? This one is a real codec killer, you can see that even at a high bitrate not all details are preserved and the codec uses more than the max bitrate you set when encoding this particular scene. 

Now it gets pretty clear, doesn't it? You can see that a bitrate below 2MBit/s is not suited to encode this scene. In the next scene watch the lower right corner.. it's where you can see lots of blockyness if the codec doesn't' have enough bitrate available. 

You'll probably have to save these pictures and look at them at full-screen 640x480 or blow them up to double size... the difference becomes quite striking if you do so.

This is another scene that literally kills an encoder. Only the 5MBit/s could save the codec from total annihilation.

Since this page is such a large load I decided to make all pictures available in a zip archive. It also contains the pictures of the DivX quality comparison. 

So how do the codecs stand up against DivX?

I haven't run so many tests as with the WMV codecs but what I saw was pretty much the same as what WMV7 generated. Remember that there should be no difference in size, encoding speed and as I saw neither in quality. But let's be fair: The encoder is still in beta stage and the codecs most likely as well. So maybe Microsoft will still be able to live up to their promises but I wouldn't count on it. And in any case the fact that only Windows Media Encoder is capable to use these codecs does not really make them appealing to the majority of people out there who are in the digital video business. Let's face it: Adobe Premiere is probably the most used tool for non-linear editing. The way the codecs can be used now requires you to do all your video manipulations in an MJPEG AVI format and then later load the AVI in the Windows Media Encoder to convert it but that's just one extra step that's not really necessary. I understand that for really good quality one should work with a lossless codec as Huffyuv - as I did for these tests - but semi-professional users shouldn't worry about quality that much - if you use a high bitrate in your WMV files that is. 

To sum it all up: I don't see any reason to use the new generation of codecs at the moment. First of all most people still use Windows Media Player 6.4 which does not include the necessary codecs to play these files. Of course these codecs can be downloaded if you have an internet connection but still... many people still don't have an internet connection or do not always want to connect to the internet to download a codec before they can watch a movie. Also... the seeking time could not stand up to DivX... somehow the WMV format is (while still being better than asf) still not up to AVI standards. 

DivX Quality check

Although these results can be applied to DivX I decided to provide a few pictures taken right of a DivX file.. and this time they are frame accurate since I can edit a DivX AVI. I have also uploaded very small clips without audio which allow you to even better visualize the difference there is between the two DivX flavors and different bitrates. I've 3 files encodec in DivX high motion at 6000kbit/s, 3 in DivX low motion at 2500kbit/s and 3 in DivX low motion at 910kbit/s. DivX low motion at 2500kbit/s: File1, File2, File3. DivX low motion at 910kbit/s: File1, File2, File3. DivX high motion at 6000kbit/s: File1, File2, File3.  These series of clips should give you an idea about the whole issue. Needless to say that everybody has a different sensory perception so some might not see the differences as I see them or they will not appear to them as striking as they appear to me.

So.. enough talk let the pictures talk for themselves. The first two sets of images compare DivX low motion at 2300Kbit/s to the same format at 910KBit/s at the codec's default settings. The better quality images come first..

Can you see all the blockyness and the loss of details on the walls? This is exactly why I favor high bitrates regardless of the number of CDs it may take.

Once again look for loss of detail and blockyness.

The next set of pictures compares the same 2300KBit/s picture with a DivX high motion at max bitrate of 6MBit/s. You can clearly see that the high motion has a bitrate limit that's set quite low... maybe 600-700KBit/s.

Well... the picture speaks for itself, doesn't it? So take my advice and don't use DivX high motion EVER! The results are truly crappy.

Final conclusion

A long time I've hesitated to do write any quality comparison article because I believed that people should figure it out themselves since everybody notices different things when watching a video. But when the new WMV codecs came along and nobody felt like testing them I decided that it was about time and now you have your full video codec comparison. I hope that this article answers all your questions about codecs and bitrates once and for all.

As always I welcome comments on the issue but comments only.. Although I must have stressed this a thousands of times: The ideal bitrate depends on each and everybody of you. Everybody has a different goal when encoding a video and a different perception of encoder flaws.

The Windows Media Player 7.0

Why many people have called this document a test of the Windows Media Player is unknown to me. But since I'm about to reinstall Windows I thought I'd give the player a shot, too. Once again I must stress that in no way do you need the new player to play the new formats you can create in Windows Media Encoder 7.0. You can perfectly well play them in WMP 6.4, the player will automatically download the necessary codecs from the web. 

The first noticeable thing about the new player is that it uses skins. There are 8 different skins, the old one and 7 new. I think the new designs are seriously bloated and went straight back to the usual look. As the encoder the player installs it's own version of the Fraunhofer MP3 codec, eliminating the one that was installed with DivX and the new version can't be used to encode, it's just a playback codec. But reinstalling DivX helps (just run the Register_DivX.exe file). My DivX rips still played fine, and the DivX codecs (both audio and video) were still available and worked just fine. Unfortunately Microsoft removed the detailed properties window, you can no longer get to the CPU slider when playing DivX files, which in some cases could be a serious disadvantage. The player incorporates many Microsoft Windows Media Sites like the radio site which allows you to directly access to internet radio stations. It also incorporates some support for portable MP3 player, tough since I have no such hardware, I wasn't able to test it. 

What I think is quite nice is the fact that you can now directly rip CDs out of Media Player to the WMA format. And in that application the use of the codec is not so limited as it is when doing DivX files: You can chose many different bitrates and the quality is quite good. Microsoft also licenses some technology from Adaptec allowing to burn CDs from MP3, WMA or WAV files. This is quite a nice feature and may convince some of you to upgrade.

If you like the new features I mentioned you may consider upgrading since I found no problem whatsoever if ripping is concerned. Maybe you should install the DivX codec before Media Player, just in case. However I prefer to stay with the old release, although I usually upgrade if MS releases new software, but in that case I don't really see an advantage for me. And in any case WMP is not the best DivX player ... I suggest you use one of the smaller and faster DivX player you can download from this site.