The CPAN Testers group do some pretty cool work. According to the group itself, they are
... a group of over 100 volunteers who test as much of CPAN as possible across a plethora of Perl versions and operating systems, which is usually many more environments than authors have available to themselves.
Don't write for Windows, and don't have access to a Windows VM courtesy of Microsoft? At least the CPAN Testers will send you the error report from a failed attempt to build on Windows.
The downside of CPAN Testers is that you may not care about certain configurations. If one of the CPAN Testers mistakenly tries to build Win32::OLE on a Mac system, the author of Win32::OLE isn't going to be very interested. Along those same lines, if your module for Perl 5.8 doesn't degrade nicely for 5.6 or earlier, then you're going to get error reports as well.
Now, the CPAN Testers site gives authors flexibility in how they'd like to tests on their modules reported. I can specify which types of reports (pass, fail, NA, etc) I get, and on which versions of Perl I'm interested. I can set up a default profile and then set up per-distribution profiles as well.
It's fantastic. Anyone who is doing CPAN modules should take a look at this excellent service and its new customizations. Thanks to the CPAN Testers for making it available.