Is Vista SP1 really the shot in the arm your Vista system needs? We've spent
many hours strapped to our benchmarking system in a caffeine and pizza fuelled
haze to uncover these very interesting results.
Although Vista SP1 has many documented improvements, we aimed to test a
particular scenario which has proved to be a major problem for pre-SP1 users:
file copy speed, particularly over a network.
Our scenario was a home user running Vista Home Premium on a fast,
low-latency network with decent PC hardware. All file copy tests were initiated
from the main machine.
Each system used the latest available vendor (non-Microsoft) drivers and the
November release of DirectX. No modifications were made to the operating system,
so as to represent as closely as possible the configuration of an OEM
machine.
We uses two test file batches – the first was a single 2.5 GB ISO, and the
second was 2.5GB of small files (over 300 MP3s). Each file batch was copied to a
remote destination (write), and then written back across the wire to the test
system (read/write).
File copies were timed from the time “Copy” was clicked to the time the copy
shell disappeared.
File copies were timed from the time “Copy” was clicked to the time the copy
shell disappeared.
We also ran PCMark Vantage across the test systems to
get an overall impression of system performance.
The aim of the tests
was to see how changes in the test machine’s patch level affected performance.
We therefore ran the tests three times – once with Vista Home Premium
RTM, once with all the available patches applied from Windows Update, and once
with SP1 RTM applied.
The secondary Vista system was also patched to
maintain consistency between it and the test machine.
The XP system was
unchanged throughout the tests, running fully patched XP SP2.
Hard
drives on all the three systems were defragmented before each file copy
test.
The Hard Numbers
There were some very surprising results and some interesting anomalies in our
test results. We reported these back to Microsoft to try to get an
explanation.
PCMark Vantage (click the image to view larger)
PCMark
Vantage results
The RTM results of PCMark Vantage provided some baseline performance metrics
for our test system. Vantage essentially tests system hardware performance, so
any changes across the various benchmarks would be the result of changes in
patch level only, as neither hardware nor drivers were changed.
Overall PCMark results jumped 12.8% from RTM to RTM Patched – quite an
increase. Remember, this is even before Vista SP1 was applied, which shows
Microsoft has been making steady progress in improving Vista's performance since
its release.
This overall increase was actually expected, as the two main performance,
compatibility and stability updates for Windows Vista – KB938979 and KB938194 –
are installed by default via Windows Update and were therefore deployed to the
RTM Patched system.
Most variation between individual benchmarks tests was very slight, with
three notable exceptions – Music and Communications actually dropped by 1.6% and
6.5% respectively, while Productivity increased by 5.25%.
The results after installing SP1
Moving to SP1 demonstrated general improvements across the board. The overall
score was 1.5% higher than RTM patched and 14.5% higher than RTM.
The anomalous results for Music and Communications in RTM Patched were
reversed, the end results being 4% and 12.8% higher than RTM respectively.
Productivity was another stand-out result, being 9.7% higher than RTM and
4.2% higher than RTM Patched.
Graphics and multimedia performance demonstrated consistent improvements
across the tests, but the Gaming benchmarks demonstrated a very slight decline
of 0.2% compared with RTM.
Microsoft couldn’t specifically explain why there was a downward trend with
this particular benchmark, but isn't too concerned -- after all, it's a tiny
drop.
PCMark’s gaming benchmarks are very sensitive to driver performance
(especially graphics drivers) and the version of ATI Catalyst installed isn’t
configured to take advantage of Vista SP1. Microsoft says it expects to see this
trend reversed in future months with the release of updated vendor drivers along
with the general release of SP1.
We also reported the anomalies encountered between the RTM and RTM Patched
systems to Microsoft.
The explanation given wasn’t specific to the problems encountered, but did
give us an idea as to why you might expect to see a drop in some areas of
performance, particularly when they are fixed later on.
Essentially a patch or hotfix is designed in relative isolation to the rest
of the operating system. It’s tested for compatibility, but in general it’s only
designed to fix a very specific issue. As such, there is a risk that fixing one
component actually causes issues elsewhere, and Microsoft believes that this is
the main reason for the data anomalies we encountered in the RTM Patched
tests.
A service pack on the other hand, is with a much more holistic approach,
examining all components and testing them for inter-operability. SP1
incorporates a number of architectural changes as well as various fixes and
updates, which is why we encountered benchmark data which was not only much
better than RTM, but reversed any negative trends in RTM Patched.
Vista SP1 is being marketed as a significant update to improve the
performance and responsiveness of Vista systems, and these benchmark results
certainly bear this out. The biggest improvement are in communications and
overall performance, which have been the main areas of criticisms of Vista to
date.
Has Microsoft fixed Vista's biggest problem?
Undoubtedly, the biggest frustration (perhaps apart from UAC) that Vista
users have encountered is absolutely attrocious file copy performance compared
to XP. We put all our test systems through gruelling APC Labs testing here to
see whether Microsoft had fixed this.
Disk-to-Disk File Copy
Performance in this benchmark was already quite good and few changes were
anticipated. The two most notable metrics was that in RTM there was an initial
difference of up to 12 seconds between the single file and multi-file tests, but
that this variation was largely eradicated in the SP1 performance data, with the
tests producing almost identical results.
Disk-to-USB File Copy
These results produced an interesting trend between the two tests. The test
results were exactly the same in RTM, but the write test data started to deviate
in RTM Patched, and were almost a minute apart in SP1 in favour of the single
file test.
Multi-file write times under SP1 were actually worse than RTM/RTM Patched,
whereas single file write times improved significantly. This was reported to
Microsoft who reported that this was not an expected result. The data has been
submitted for further examination. In both cases the writeback test improved
marginally with SP1.
Vista-to-XP File Copy
Given the problems which have dogged Vista in the realm of XP/2003
networking, this was the area we expected to see the greatest improvements...and
we weren’t disappointed.
There was almost no variation in the test results between RTM and RTM
Patched, but it’s worth mentioning just how bad performance was. The single file
write test was almost 3.5 minutes slower than the same test copying to USB, and
the multi-file test was only a minute quicker.
The writeback test results were appalling, at between 6 to 8 minutes slower
than the USB copy tests, and interestingly there was a difference of between 2
to 3 minutes between the single file and multi-file tests in favour of the
multi-file.
With SP1 however, the improvements were astounding. The single file write
test dropped from just under 9 minutes to less than 2 – a 77% improvement –
while the writeback test dropped from 9:33 to just over a minute – an 86%
improvement.
The results in the multi-file tests were equally impressive, improving by 64%
and 83% respectively. Interestingly, the massive variation between the tests on
RTM was almost completely ironed out in SP1.
Vista-to-Vista File Copy
This test produced the greatest number of anomalous results, on which we
consulted with Microsoft for explanation and clarification.
While the overall performance times were good, they worsened across the board
from RTM to RTM Patched by a factor of 2 to 5 seconds. However, this was all
resolved in SP1 where test results were 6 to 14 seconds better than RTM, with
the notable exception of the single file writeback test which was a substantial
21 seconds slower than RTM.
Microsoft's explanation for this has a lot to do with how file copying in
Vista SP1 has been redesigned. It also explains why the Vista-to-XP tests in RTM
and RTM Patched were so poor.
Microsoft's explanation for dodgy Vista-to-XP copy performance
The changes to the file copy algorithm introduced in Windows Vista were
designed to remedy a number of problems with the algorithm used in Windows XP.
One of the big problems was XP's caching (the "cache manager write-behind"
thread) not being able to keep up with the necessary rate of writes, especially
when copying large amounts of data.
What would happen was eventually, the target system’s memory would become a
bottleneck in the copying process. Also, file copies from a remote system cached
the content twice – once as the source file was read and once as the target file
was written. This puts extra memory pressure on the client system as well as
extra CPU load because of the Cache Manager working overtime.
PCMark
test results: click the image to view in a larger size
Vista now uses an algorithm which uses cached disk I/Os for files of 256KB or
less, and non-cached I/Os for larger files. It works out like this:
- Two I/Os for files 2MB in size or smaller
- Up to a maximum of 8 I/Os for files larger than 8MB.
The size of the I/O also varies, depending on the file being copied – the
actual file size for files less than 1MB, 1MB for files between 1MB and 2MB in
size, and 2MB for anything bigger than 2MB.
Therefore, to copy a 16MB file, Vista would use 8 x 2MB non-cached I/Os, wait
for the writes to complete, and then start another cycle.
Although this algorithm does improve file copies over the previous algorithm,
it does suffer from a number of drawbacks, the most significant ones being that
on network file copies, the process sporadically write the I/Os back out of
order.
This causes excessive disk seek activity on the destination system and, over
time, bottlenecking as the Cache Manager effectively backs up at both ends of
the wire.
Another issue – and probably the most significant – is poor copy performance
on file copies involving large files and/or large groups of files.
The exact reason for this is explained in detail by Mark Russinovich,
Technical Fellow in the Platform and Services Division at Microsoft, and
previously of Winternals and Sysinternals.com.
In his blog, he explains that “...the previous algorithm’s use of cached
file I/O lets Explorer finish writing destination files to memory and dismiss
the copy dialog long before the Cache Manager’s write-behind thread has actually
committed the data to disk.
"With Vista’s non-cached implementation, Explorer is forced to wait for
each write operation to complete before issuing more, and ultimately for all
copied data to be on disk before indicating a copy’s completion.
"In Vista, Explorer also waits 12 seconds before making an estimate of the
copy’s duration and the estimation algorithm is sensitive to fluctuations in the
copy speed, both of which exacerbate user frustration with slower
copies.”
Vista’s reliance on non-cached I/O copies effectively increases the
perception of poor performance, regardless of whether the data is being moved
more efficiently or not. This certainly explains our benchmark data, which was
measured from the “perceived start” to the “perceived finish” of each file copy,
as well as the serious drop in system responsiveness experienced during each
file copy.
SP1 seeks to remedy the situation by going back to using cached I/Os in all
file size instances, except in the case of remote file copies. In this case, it
issues a command not to cache remote file locally during the copy process, which
prevents the double-buffering problem previously encountered. It also spools the
I/Os passed from source to destination into memory where they assembled in the
correct order before being written to the disk. This prevents the hard drive
thrashing, high CPU utilisation and sluggish system performance previously
encountered.
Additionally, for SMB 1.0 (the file transfer protocol used by Windows
XP/2003), the I/Os passed on the wire have been reduced from 60KB in size (the
previous default) down to 32KB. This essentially means that file transfers
between Vista SP1 and non-SP1 systems have been made vastly more efficient and
require fewer resources from both the source and destination systems, but at a
cost of throughput in some scenarios, hence the unexpected slightly higher
scores encountered in some of our tests.
muBlinder 3.54 (81369)
wpa_registry (19013)
