Month: November 2007

Schwern has killed off Perl 5.5, and I thank him

November 18, 2007 Opinion, Perl 5 3 comments

I applaud Michael Schwern’s announcement today that he will no longer be supporting Perl 5.5 in any of his modules. Toolchain modules like Test::More and ExtUtils::MakeMaker will be compatible with Perl 5.6.0, and others with 5.8.0. As Schwern puts it, “5.5 is effectively end-of-lifed.” And not a moment too soon, I believe. Perl 5.6.0 came out seven years ago, and 5.8.0 five.

Schwern’s breaking point was seeing the Perl Survey results that only 6% of respondents use Perl 5.5. Most of all, he points out:

Finally, I’m coming around to chromatic’s philosophy: why are we worried about
the effect of upgrades on users who don’t upgrade? Alan Burlson’s comments
about Solaris vs Linux are telling: if you’re worried more about supporting
your existing users then finding new ones, you’re dead.

I applaud Schwern’s radical break from the past. No longer will he be “hamstrung from
using ‘new’ features of Perl,” as he puts it. This will allow him the freedom to do more great things as I fully expect he will.

Most of all, I’m glad that he just did it. No committee, no call for consensus, no poll of people to see what everyone thought. JFDI, baby, JFDI.

Who among us will be the first to write a module that takes advantage of Perl 5.10’s new features, urging us all forward, instead of mired in the mud of the past? I can’t wait to see it happen.

Installing a module? Do you feel lucky?

November 17, 2007 CPAN No comments

David Cantrell has come up with his CPAN dependencies and test results checker. It checks out all the dependencies for a distribution, and displays the installation results for each of the dependencies, as as reported by the CPAN testers. It’s interesting to see the visual representation of all the modules’ test histories, but the idea of multiplying all the “probabilities” of installation is specious.

Gerard Goossen talks about Kurila, a Perl 5 fork

November 15, 2007 Interviews, Perl 5 No comments

A few days ago Gerard Goossen released version 1.5 of his kurila project to the CPAN, a fork of Perl 5, both the language and the implementation. I talked with about the history of this new direction.


Andy: Why Kurila? Who would want to use it? What are your goals?

Gerard: Kurila is a fork of Perl 5. Perl Kurila is a
dialect of Perl. Kurila is currently unstable, the language is
continuously changing, and has just started.

There are a few goals, not all of them going in
the same direction. One of the goals is to simplify the Perl internals
to make hacking on it easier. Another is to make the Perl syntax
more consistent, remove some of the oddities, most of them historical
legacy.

What is currently being done is removing some of the more
object/error-prone syntax like indirect-object-syntax and removing
symbol references. Both of these are not yet very radical yet,
most modern Perl doesn’t use indirect-object-syntax or symbol
references.

But I am now at the stage of doing more radical changes, like
not doing the sigil-change, so that my %foo; $foo{bar}
would become my %foo; %foo{bar} .

Andy: Where do you see Kurila getting used? Who’s the
target audience for it?

Gerard: Kurila would be used for anything where currently
Perl is being used. I am using Perl for large websites so changes
will be favored in that direction.

I am working for TTY Internet Solutions,
a web development company. We develop and maintain websites in
Perl, Ruby and Java. Websites we develop include www.2dehands.be,
www.sellaband.com, www.ingcard.nl and www.nationalevacaturebank.nl.
Of these www.2dehands.be and www.nationalevacaturebank.nl are
entirely written in Perl.

We are not yet using kurila in production, but I have a testing
environment of www.2dehands.nl which is running on Kurila. Developing
Kurila is part of my work at TTY.

Many of the changes in Kurila are inspired by bugs/mistakes we
made developing these sites. It started with the UTF8 flag. We
encountered many problems making our websites UTF-8 compatible. In
many cases the UTF8-flag got “lost” somewhere, and after combining
it with another string, the string got internally upgraded and our
good UTF-8 destroyed. Because everything we have is default UTF-8.
The idea was simply to make UTF-8 the default encoding, instead of
the current default of latin1.

Andy: Did you raise the possibility of changing the default
encoding in Perl?

Gerard: The problem is that changing the default encoding
the UTF-8 is that is destroys the identity between bytes and
codepoints. So it’s not a possibility for Perl 5. Like what does
chr(255) do? Does it create a byte with value 255 or
character with codepoint 255?

I made a patch removing the UTF-8 flag and changing the default
encoding to UTF-8 and sent it to p5p.

Andy: What was the response?

Gerard: There was as good as no response to it, I guess
because it was obvious that it seriously broke backwards compatibility
and the patch was quite big, making it difficult to understand.

About two weeks after the utf8 patch, I announced that I wanted
to change the current Perl 5 development to make it a language which
evolves to experiment with new ideas, try new syntax and not be
held back by old failed experiments. One of the interesting things
about Perl is that it has a lot of different ideas and these are
coupled to the syntax.

There was of course the question of why not Perl 6.  That it
should/could be done in backwards-compatible way. That there is no
way of making the Perl internals clean, that is better to start
over.

And about half a year ago I announced that I had started Kurila,
my proof of-concent for the development of Perl 7. Rewriting some
software from scratch is much more difficult then it seems, and I
think starting with a well proven good working base is much easier.
Perl 5 is there, it is working very good, has few bugs, etc., but
it can be much better if you don’t have to worry about possibly
breaking someone code, and just fix those oddities.

Andy: Do you have a website for it?  Are you looking for
help?

Gerard: There isn’t a website yet, and also no specific
mailing list, currently all the discussion is on p5p. There is a
public git repository at git://dev.tty.nl/perl.

Andy: What can someone do if he/she is interested in
helping?

Gerard: Contact me at gerard at tty dot nl. Make
a clone of git://dev.tty.nl/perl and start making changes.

Bind output variables in DBI for speed and safety

November 14, 2007 Perl 5 5 comments

When in a tight loop of many records from a database, using the quick & dirty solution of calling $sth->fetchrow_hashref can be expensive. I was working on a project to walk through 6,000,000 records and it was slower than I wanted. Some benchmarking showed me that I was paying dearly for the convenience of being able to say my $title = $row->{title};.

When I converted my code to bind variables to the columns in the statement handle, I cut my run time about 80%. It was as simple as adding this line:

$sth->bind_columns( my $interestlevel,
my $av_flag, my $isbn, my $title );

before calling the main loop through the database. Now DBI knows to put the data directly in there, without creating an expensive temporary hash. I also can’t make a typo such as $row->{ISBN} inside the loop, so there’s a measure of safety as well.

The benchmarks below show the relative speeds of each of four techniques:

hashref     took 31.5048 wallclock secs
array       took 8.83724 wallclock secs
arrayref    took 5.5308 wallclock secs
direct_bind took 4.46956 wallclock secs

Here’s the key parts of the benchmark program I used:

use Benchmark ':hireswallclock';
sub hashref {
while ( my $row = $sth->fetchrow_hashref ) {
my $interestlevel = $row->{interestlevel};
my $av_flag = $row->{av_flag};
my $isbn = $row->{isbn};
my $title = $row->{title};
}
$sth->finish;
}
sub array {
while ( my @row = $sth->fetchrow_array ) {
my ($interestlevel, $av_flag, $isbn, $title) = @row;
}
$sth->finish;
}
sub arrayref {
while ( my $row = $sth->fetchrow_arrayref ) {
my $interestlevel = $row->[0];
my $av_flag = $row->[1];
my $isbn = $row->[2];
my $title = $row->[3];
}
$sth->finish;
}
sub direct_bind {
$sth->bind_columns( my $interestlevel,
my $av_flag, my $isbn, my $title );
while ( my $row = $sth->fetch ) {
# no need to copy
}
$sth->finish;
}
for my $func ( qw( hashref array arrayref direct_bind ) ) {
my $sql = <<"EOF";
select interestlevel, av_flag, isbn, title
from testbook
limit 1000000
EOF
# This sub calls the SQL and returns a statement handle
$sth = sqldo_handle( $sql );
my $t = timeit( 1, "$func()" );
print "$func took ", timestr($t), "n";
}

Did you find this article useful? Or does it not belong on Perlbuzz? Let us know what you think.

Perl 5.10’s first release candidate coming soon

November 13, 2007 Perl 5 3 comments

The first release candidate of Perl 5.10, with the first new syntax and major features since 2002, will be released soon, probably in the next week or two. The code has been in feature freeze for weeks, and only minor patches are being accepted. Lately the VMS porters have been working on compatibility problems with File::Spec and File::Patch, and Jos Boumans and Ken Williams are syncing core CPAN modules with the Perl source trunk.

For a gentle and well-presented introduction to the features in Perl 5.10, see Ricardo Signes’ slides for his talk Perl 5.10 For People Who Aren’t Totally Insane.

Please note that Perl 5.10 is not Ponie. Ponie was the project that was to put Perl 5.10 on top of the Parrot virtual machine, but Ponie has been put out to pasture.

Perl Survey results released

November 12, 2007 Community 3 comments

primary_language_spoken.png

It’s been a while, because I had all kinds of overblown expectations of being able to offer the results in a range of formats, but I have to admit that a month in a developing country with only intermittent satellite internet has bested me.

Therefore, I’ve decided to release the results of the Perl Survey in PDF format for now, with the HTML version to follow as time permits.

Some points of interest:

  • 4% of the Perl community are women
  • Average salary for Perl programmers worldwide is $68,687 (USD equivalent)
  • 87% of Perl people are using 5.8.X at least some of the time; that means that 13% are still on 5.6.X or lower
  • 33% of respondents are members of their a Perl Mongers group
  • However, 27% of respondents don’t participate in the Perl community at all (or at least not in any way that we asked about)

Find out all about this, and more, at the link above.

Also, the entire data set for the survey has been released under a Creative Commons license, and I’d like to encourage everyone to download it and play with it. A number of people have already contributed their own analyses of certain parts of the data.

I’d like to thank everyone involved for their help, and I hope that this information will be of use to the Perl community going forward.

Perl has supported threading for years

November 7, 2007 Uncategorized 6 comments

A recent column in Software Development Times lamented the lack of support for threading in software today. In it, Andrew Binstock says:

Dynamic languages are even further behind. To wit, Python, allows only one thread to run at a time (except for I/O); this means you can have threads but not running in parallel. Ruby can run threads only within the one VM, which is arguably better but nowhere near good enough. And OpenMP, which might be a solution for some, is limited to C++ or Fortran.

Unfortunately, Binstock ignores the dynamic language that should be first on his mind, Perl. Perl’s threading got stable back with Perl 5.8.7 in 2005, after a few years of experimental support. Today, Mac OS X and most Linux distributions ship with a threaded Perl enabled.

How can you tell if your Perl supports threads? Look at the output of perl -V for the string “useithreads”:

$ perl -V | grep ithreads
config_args='-ds -e -Dprefix=/usr -Dccflags=-g  -pipe  -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib'
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define

While I agree with the basic premise that demand for threaded apps on the desktop is low, the tools for supporting threads are not quite as sparse as Binstock would have us think.

P.S. O’Reilly & Associates is now O’Reilly Media, and has been since April 2004.

Andy Armstrong talks about the road to Test::Harness 3.00

November 7, 2007 Interviews No comments

Test::Harness 3.00 has finally been released, and it’s a huge opportunity for anyone who writes tests with Perl, if only for the ability to run prove -j and run tests in parallel. I took a few minutes as
the maintainer of the old 2.xx series to interview Andy Armstrong, the new maintainer of 3.x, about the history of the new Test::Harness, and what it took to get here.

Andy Lester: So, Andy Armstrong, a joyous day has come: Test::Harness 3.00 has been released.

Andy Armstrong: Yes. I was relieved.

Andy Lester: What has it taken to get to this point?

Andy Armstrong: I think we had a fair bit of paranoia about breaking the toolchain for everyone, and thus becoming extraordinarily unpopular.
That made us cautious. We spent a lot of time building our own smoke testing setup. And running lots of people’s tests against our code.

Andy Lester: Who is the “we” in this? Back in June of 2006, Schwern and I started the kick off to Test::Harness 3.00 at YAPC::NA in Chicago. What’s happened since then?

Andy Armstrong: Ovid got the code started and had just about everything in place by April 2007.
Around then I volunteered to look at a Windows problem.
And I sort of got dragged in. I really liked the code Ovid had written and enjoyed working on it – so that was an attraction.
The Windows problem took a few minutes – but I’m still here.

Andy Lester: You’ve done a lot more than being dragged in. You hosted the Subversion repository, and the mailing list. What else?

Andy Armstrong: My monopolist plan laid bare for all to see…
I have a server which is nominally so I can do things like that – so then I have to do them to justify its existence.
So I’m hosting the perl-qa wiki, the TAP wiki. Just sites that needed a home. Like an orphanage 🙂

Andy Lester: You’ve uploaded T::H 3. Are you now the maintainer? I thought Ovid was going to be the maintainer of T::H3. (I ask both for the benefit of the Perlbuzz readers, and for my own knowledge:-))

Andy Armstrong: I think I made a move on Ovid somewhere back there and he didn’t struggle. So now I’m it. I honestly can’t remember how that happened.

Andy Lester: Glad to have two Andys maintaining different versions of the same module. 🙂
So why does someone want to upgrade to Test::Harness 3? What’s in it for the average Perl user?

Andy Armstrong: If you do nothing else – just install it – you’ll get better looking test reports. Color even 🙂
And when people start writing test suites that use TAP version 13 features you’ll get even more informative reports as an indirect result of T::H 3.00.

Andy Lester: And it’s completely compatible?

Andy Armstrong: It’s very slightly more fussy about completely crazy syntax errors. But generally yes, compatible – foibles and all.
That’s syntax errors in TAP (Test Anything Protocol) – just for folk who don’t know what’s going on behind the scenes.

Andy Lester: So what’s in the future for Test::Harness and prove, its command-line interface?

Andy Armstrong: Well we’re just talking about TSP (Test Steering Protocol) on
the perl-qa list. And we need to do something interesting with the
YAML diagnostic syntax we have now. I’ve written a module for
TextMate that uses that so that the cursor jumps to the right line
in the test program when you click on the diagnostic.

Andy Lester: What’s the benefit of TSP? How would a tester use that?

It would give a test suite more active control over its own execution.
Particularly in the case of things like user interface toolkits or
modules that are highly platform or configuration dependent you may
have large number of tests you’d like to skip conditionally. So
TSP would be a convenient way to have a single controller program
that would decide which other parts of the test suite to execute.
And then you’d probably grow on that to expose more control over
which tests to run via prove or whatever UI you’d be using.

Andy Lester: So a more advanced version of SKIP blocks, and you wouldn’t have to figure out what tests to run when you ran Makefile.PL or Build.PL.

Andy Armstrong: Yes. And an area where people are likely to find new applications too.

Andy Lester: Anything else people should know?

Andy Armstrong: That there’s still plenty more to do with
T::H and testing in general. And that I’m surprisingly cheap 🙂 I
also want to thank people who have worked on Test::Harness 3, in
alphabetical order: Sebastien Aperghis-Tramoni, Shlomi Fish, David
Golden, Jim Keenan, Andy Lester, Michael Peters, Curtis “Ovid” Poe,
Michael Schwern, Gabor Szabo and Eric Wilhelm. All helped immensely
– even you 🙂

Andy Lester: Well, I do want to thank all of you for doing
this massive overhaul of Test::Harness. Abandoning the existing
code and starting from scratch has given T::H a new lease on life,
and a new platform to move forward.