New "Scientific" Auto Focus Test

A new auto focus test that's touted as being "scientific" has been posted online.

On the surface it looks like the 1D Mark IV wins this competition handily, easily besting the D3 and the new D3S.  The new Mark IV also tops the 1Ds Mark III and 1D Mark III in this test.

Actually, the news that the 1D Mark IV tops the 1Ds Mark III and 1D Mark III is about the only useful information I can pull from this test as the testing methodology is seriously flawed.  The D3 and D3S are not given a fare shake in this test, and honestly, its this factor that completely invalidates this test IMHO.

The test parameters are listed on the site and this sentence in particular is what concerns me most about the validity of this test:

These tests are all done using the default camera settings and default custom function settings.

It certainly sounds fair that all cameras are tested with default settings, but in the case of the D3 and D3S by default they give priority to shutter release over auto focus.  That means, by default, the camera will fire at its max continues shutter release speed without giving any consideration to whether the camera actually has focus or not.

Conversely, the 1D4 by default gives priority to AF lock over shooting speed.  This means it will slow the frame rate down if necessary as it won't fire until it thinks it has AF lock.  So in essence you have one camera (D3/D3S) firing wildly as fast as it can with no consideration to AF lock and you have another camera firing only when it has AF lock and likely at a slower frame rate.

If they wanted to make a truly representative test, they would set "a1 AF-C priority selection" on the D3's to "release + focus" allowing the camera to behave in the same manner as the 1D Mark IV and redo the tests. Withing correcting this oversight in their testing methodology, I don't see any validity to the test results.

EDIT (3/4/10):

I posted my concerns about the testing methodology to the blog and the author amended his blog post with additional information.  Check out the total number of frames fired by both cameras.

Canon 1D MKIV Results

Trial 1

45 in focus out of 50= 90% in focus

Trial 2

56 in focus out of 60= 93% in focus

Trial 3

48 in focus out of 55= 87% in focus

Trial 4

49 in focus out of 55= 89% in focus

Trial 5

52 in focus out of 58= 90% in focus

Canon 1D MKIV Overall= 89.4% in focus

Nikon D3s Results

Trial 1

29 in focus out of 46= 63% in focus

Trial 2

37 in focus out of 47= 79% in focus

Trial 3

46 in focus out of 53= 87% in focus

Trial 4

40 in focus out of 52= 77% in focus

Trial 5

38 in focus out of 47= 81% in focus

Nikon D3s Overall= 77.4% in focus

I find these numbers to be quite interesting.  It would appear that the D3S doesn't shoot 9fps as claimed by Nikon even under ideal circumstances and with the camera set to ignore focus priority and to fire as quickly as it can.  It continuously posted fewer frames than the 1D4.  It stands to reason that if the camera had been set to focus priority the number of frames fired would have gone down even further.

That means the 1D4 can shoot much more quickly than the D3S which might be a very important feature to sports or action shooters.