Updated December 31, 2013
This site will no longer be maintained from January 1, 2014.
New submissions will no longer be processed
CS6 anomalies and AME problems.It has been some time since this page was updated. But so many things have happened and are happening, that an update was absolutely necessary.
CS6 has serious caching problems.
For the attentive reader, it is clear that CS6 has serious problems with caching of material in a timeline. Take a look at the Top-20 Chart and closely look at the 4-th chart, named MPEG2-DVD Time in seconds. Notice there are 14 observations in the range of 18 to 29 seconds. The rest of the observations are in the 45+ second range. It looks like there is a clear dichotomy in this chart. That is correct, because when you scroll down to the 10-th chart, named MPEG2-DVD Time in seconds, CS6 only you will see that all 14 observations from the other chart have mysteriously disappeard. In fact only 3 are left from the previous chart.
Investigating this further revealed that identical systems running the same test took a performance penalty of more than 250% in CS6 over CS5/5.5.
But that is not all. The performance delta between Direct Export and AME Queue has never been that big.
Testing on the same machine with CS6 showed huge differences in performance, with a delta of nearly 500% in extended tests. In other words, using Direct Export is almost five times faster than using the AME Queue. If I take my own results as a starting point and then correct the results for both of these problems, the score would have been widely different. This is called the 'Hypothetical Monster'.
|Computer ID||Total||RPI||Disk I/O||MPEG DVD||H.264 BR||MPE Off||MPE On|
The conclusion is obvious, CS6 carries severe limitations in the Disk I/O and MPEG2-DVD department.
New dynamic charts and downloads available
We have made available two download versions, one for CS5 and one for CS5.5. This was necessary because of the different instructions that were required for each version of PR. See the Benchmark Instructions page for further details. Secondly, the Test Results page now has active dynamic Top-20 and MPE Gains Charts, that will automatically show the test results, based on the latest database information. You can be assured that these charts will always reflect the current database contents, as they are updated automatically without any action from your side. Also the readability of the charts has been improved significantly.
Background on the new test results
With over 500 entries in our Excel spreadsheet, we managed to really bring Excel to its knees. Adding new entries was getting more cumbersome by the day and outputting its results in huge .html files and uploading to the website was taking several hours for each addition. In short an unworkable situation.
Therefore we decided to abandon Excel altogether and move to a dynamic SQL database. The beginning is easy, design the database tables and make the .php pages to query the database, but the problem we next encountered was how to get the existing results from Excel into the SQL database. Additional problem was that for feedback we needed date and time of submission and email addresses in the database, plus some other added fields that did not exist in the Excel sheet. The only source we could use effectively was the data from the submission forms we have received, because those data contain the date and time of submission, the email address and the added fields. However since we only started using this submission form on 01-29-2011 the consequence was that all data submitted earlier would be lost.
Pro's and con's
Let's start with the con's. All entries that were submitted without the use of the submission form are lost in the new results page. The only way to get them back in is to resubmit your data using the submission form. To have the old results still available for comparison we left the old pages intact and modified the Benchmark Results menu accordingly. However the discrepancies between the old Excel based sheet and the new results will increase over time. The old sheets will no longer be updated. If you resubmit your exsisting results using the submission form they will be automatically included in the new database and show up in the new results. We urge you to follow that route.
The pro's are significant however and the potential even greater. The submission form reduces typo's and mistakes and ensures that all requested information is submitted, making data entry much easier and faster, both for yourself and for us to add them to the database.
The menu structure on the results page is much clearer and cleaner. It starts on the interactive option "Narrow the Search" and shows all the results from the database dynamically as default. Now you can easily limit your selection to compare with by using the 8 tabs, each with a number of radio buttons to do an AND Query. Example:
You want to see only Asus systems, so you click on that radio button and the data are automatically updated. Next you want to further narrow the search to i7 systems only, so you click on the tab "Model CPU" and click on the button i7 series. The data is updated automatically again. Next limit the search to systems with 24 GB RAM, using CS5 and a GTX card in the same way.
If you just want to see all results, move to the Summary Results or All details where results are displayed with page navigation to easily move through all the entries. On all these pages clicking on a column header will toggle sorting on that column, but only for the table shown.
We are investigating methods to make data submission even easier, so you no longer need to fill out your hardware, but instead use another way to collect more information that can be shown in a popup window a la CPU-Z for individual records. Another idea we have is then to link those hardware data to external price databases to generate up-to-date BFTB charts with more components included.
Video cards performance
With the number of observations available now (500+) we see a slight tendency that more cores do help performance. The usual performance increase of hardware acceleration over software is around 10 x, with the cards with fewer cores lagging behind and the ones with more cores pulling ahead. Average times still do not show a clear tendency to measure performance. The increase in performance over software shows that indeed the 9800 GT and GTX 260 are clearly lagging when compared to the GTX 480/580. The newer architecture shows its benefits.
With sufficient financial means it is pretty easy to build a top performing system. But with limited means it is a completely different story. Now that the benchmark has over 500 entries, it can be useful to investigate which systems offer the best value for money. In order to make this a practical endeavour, we have taken some short-cuts, but believe that the results will be beneficial to you.
Keep in mind that with fluctuating prices for CPU, memory and video cards this is just an instant picture and next week's results may differ, but will show the same trends and overall conclusions. The recent price drop in CPU, memory and video prices are the cause that the results have changed in detail, but not in overall tendency.
What is the approach taken?
We simply do not have complete listings of all components in each system, so it would be impossible to accurately calculate the cost of a system. What we do know is the CPU, the amount of memory and the video card in use. Often we do not know whether a raid controller is in use or what brand and model, we know the number of disks, but not the brand and model. Not knowing a lot about each system led us to take a shortcut.
We know the going prices for all the CPU's and the video cards that are used and we can make a rough estimate of the cost of the memory installed.
For each system the investment cost of CPU, memory and video card are calculated, based on going prices per April 2011. This means that the individual amount paid for these components may be different than the calculated price, but by taking this approach we safeguard consistency.
Now that we know the investment cost of CPU or two CPU's in the case of a dual processor system, memory and video card, the question arises what performance is achieved by each system per $ or € invested? Which system gives the best performance per $ or € spent?
There is no relationship in the score to the actual cost of a system. There is no way to tell whether a score of 7 relates in anyway to that amount of $$$. It is all about the value for money.
If there are two systems identical in CPU, memory and video card, they have the same investment cost. If one system is faster than the other, because of better disk setup (not included here) or better tuning (not quantifiable in $) it will give a better BFTB score because per $ it delivers better performance.
The chart below shows the results of the Top 25 systems:
The best area to be is the blue area. The further to the right, the better the performance in the test, the higher the Bang-for-the-Buck (BFTB) score, the better the value for the money. To help interpreting these results, the three scores in the blue area and others close to it, are:
- 14.1 : Harm's Beast (i7-920 @ 3.7, 12 GB, GTX 480)
- 13.5 : Cristobal OC/PC (i7-950 @ 4.2, 12 GB, GTX 460)
- 11.3 : JES3 (i7-950 @ 4.3, 24 GB, GTX 480)
- 6.9 : Studio North (i7-980X @ 4.3, 14 GB, GTX 480)
- 6.2 : BillG (i7-980X @ 4.3, 24 GB, GTX 580)
Keep in mind that all these systems are seriously overclocked to improve performance, are equipped with (sizable) disk arrays and mostly good raid controllers, that make the total system-cost much higher and that is not reflected in this chart.
A noteworthy but not unexpected conclusion is that nearly all scores below 5.0 are dual Xeon systems and in the range from 5.0 to 9.0 are the i7-980X systems. Seeing these results, one can clearly see that the BFTB score of 6.9 is much more attractive than the 2.8 score of the top performer, or the 1.8 score of the Juggernaut. It makes it obvious that dual Xeon systems are not the most economical choice. It also shows that a simple i7-920/930/950 is more economical than the i7-980X, and the new Sandy Bridge is a nice third in the 6.0 to 10.1 range.
Tuning the system will lead to improved performance, moving the benchmark result to the right while at the same time increasing the BFTB score, moving up, so the general direction of the improvements is towards the upper right corner, the area to be.
Since hardware prices are relatively volatile, these results will only be updated occasionally.
Currently the i7-960 with 12 GB memory and a GTX 470 or over-clocked 460 seems to be the sweet point for economic high performance systems if properly tuned and overclocked.