Perl must decentralize, diversify and colonize

| 23 Comments

Perl's big problem is one of perception. The other day, a job candidate asked me "You're moving your web apps from PHP to Perl? Shouldn't it be the other way around?" Why did he think that? The candidate knew no Perl, and only a bit of PHP, so had no technical reason to believe that PHP was better than Perl. So why did he think Perl was subservient to PHP in the web arena?

What I suspect is that Perl is just less visible, and not just in the sense of crap like the TIOBE index where it equates hits for "Foo programming" popularity of the Foo language.  I'm talking about in the real way people see the world of programming.

I'm certain that PHP has become a de facto choice for basic web apps because it's just How You Do It these days. You see enough PHP in the context of the web, it starts to sink in.

Why is Ruby on Rails so popular? Is it better than Perl on Catalyst? Or is it just that people hear about Rails more? I suspect the latter, because perception is reality. When people perceive Perl as being dead, or not as powerful as other tools, it might as well be.

There are three goals I'd like to address:

  • Perl needs diversity, needs new blood, both in users and community as well as tools.
  • Perl needs more active mindshare in the programming world.
  • Perl 5 must continue to be seen as a viable language.

The three actions to take: Decentralize, diversify and colonize.

Perl must decentralize

Perl has a great infrastructure. We have the CPAN. We have the various sites of perl.org. We have Perl Mongers and perlmonks, and so on. Unfortunately, we've grown complacent. It's time to start looking elsewhere for infrastructure and community.

Standardization on tools makes sense if the tools are the best possible tools available. In many cases, the infrastructure already in place in the Perl community may not be the best there is.

Is the Mailman installation running on lists.perl.org the best mailing list solution for you when you need to start a mailing list? If not, then consider other options. Starting a list at groups.google.com is trivial, for example.

Is rt.cpan.org the best bug tracking solution for your modules or projects? If not, then look at other options. Many of my modules, such as ack and WWW::Mechanize are hosted at Google Code, where I use the bug tracking system there.

Is use.perl.org the best place to host your blog? Maybe you should post somewhere else. Certainly that was the case with me. I found that use.perl.org wasn't the news site I wanted it to be, so I started Perlbuzz as an alternative, literally doubling the number of news sites devoted to Perl.

Note that "alternative" does not me "replacement". It's possible to have two similar but diferent projects, websites or whatever that both fill a need. If, say, Perlbuzz replaced use.perl.org, we'd still only have one news site, which is no better a situation than we started with.

Some of the alternatives might not be community-based. Google Code, is not run by altruistic volunteers. Commercial interests will usually have more time and money to spend on providing useful infrastructure. That's no knock on the volunteers, but merely taking advantage of what the commercial interest has to offer.

(I know at some point someone will say "At some point decentralization is harmful." I agree, but we're nowhere near that point. Such a problem to have!)

Perl must diversify

The mantra "There's more than one way to do it" is at the core of Perl the language, but this is not always the case in the Perl community. We must always remember the guiding principle of TMTOWTDI.

The first place to diversify is with people. I suspect, but cannot prove, that the Perl community's size is a net loss as time goes by. We must constantly be trying to bring new people into the fold, to take advantage of their innovations, and to make what we do more fun.

Second, we must be willing to launch new projects, even if they're similar to existing projects. It could be a module, or a website, or an entire application. The knee-jerk resistance to diversity often sounds like this:

  • "Why do we need Getopt::Whatever, we already have Getopt::Long"
  • "Why do we need another templating system?"
  • "Why do we need Jifty/Catalyst/whatever? We already have Maypole."
  • "Why do we need Perlbuzz, we already have use.perl.org?"

I've never understood this fear. It sounds like the resistance is based on the premise of "I wouldn't want to do that project, so you shouldn't either." It's the refrain of someone with a closed mind, unable or unwilling to imagine what could be. In the case of similar modules, the refrain usually goes "It's too hard to find the module you want anyway," but that's not a problem of the module, but of the way things are found in CPAN itself. (And if you'd like to address that little bit of decentralization, please take a look at my Rethinking CPAN mailing list.)

Ultimately, fear of the new is counter to the principle of TMTOWTDI. Indeed, plurality is at the very heart of open source, and it's additive, not subtractive. Template Toolkit need not take away from Mason. Perlbuzz need not take away from use.perl.org. Perl didn't take away from awk and shell. In any of these cases, if the new solution does take away from another, then the other solution was an inferior one to begin with.

Decentralizing and diversifying do two things. First, it opens our minds to alternatives that we may not have considered. It makes us more likely to find solutions that are better than what we started with. Second, it helps with colonization, the ultimate goal.

Perl must colonize

The other day, Tim O'Reilly posted on his Twitter feed a link to a cool article about life in the universe which mentioned Von Neumann probes, theoretical space craft that can replicate themselves using materials found in their travels. A single probe could travel some distance, replicate a thousand cloned probes, which would then launch in a thousand different directions, repeating the cycle.

That's what we need to do: Colonize the mindshare of the world to let them know that Perl is still viable, and a hell of a lot of fun. These new Perl lovers will spread the love as well, and the cycle will continue. In terms of action, it means simply "helping make more people aware of Perl as a cool & useful language." Fortunately, it's a case where we can think globally and act locally, even passively.

The biggest effect I see you, the reader, as being able to have is by spreading out Perl's web footprint. I want more places for people to find out about Perl, rather than a few big ones.

This colonization approach goes hand-in-hand with decentralization. Take my ack project, for example. ack has a number of footprints out there:

That's three times places for people to stumble across Perl, to see it mentioned. A separate home page also makes it more likely to get linked from somewhere else, as when Daring Fireball linked to ack and thousands of people came to see about ack. That's thousands of people who went and saw "Huh, here's a tool in Perl."

Of course, a few thousand visitors isn't going to change any mindshare. That's why it's not just me that needs to work on this. Fortunately, decentralizing and diversifying makes colonization easy.

Our Google footprint is pretty bad, anyway. When I search for "perl" in Google, the top five hits are:

  • perl.org, which is a good base.
  • perl.com, O'Reilly's Perl page, also a good base.
  • The wikipedia entry for Perl, which certainly isn't much help for a beginner
  • www.cpan.org, with its front page that expects that you know what you want.
  • An old perl 4 man page from Carnegie-Mellon University

Is this the best impression that we have to give the world?

Adam Kennedy gets it

Adam Kennedy's Strawberry Perl is a marvel of the principles I've talked about. Strawberry decentralizes, as it uses none of the core perl.org infrastructure, and it diversifies and colonizes by giving Windows users an alternative to ActivePerl, which for many users is not robust enough. When people talk about Strawberry Perl, it helps with mindshare as well.

Adam is a great example of someone who has set out to make improvements to a part of Perl, and implemented it without worrying about permission or duplication, helping Perl as a whole along the way. I thank him for it. I suspect his Perl On a Stick project will have similar results.

How you can help?

The strategy of decentralize/diversify/colonize takes actions at all different levels. You don't have to go create your own Perl distribution, or even write any code. Here are some other ideas to get you started.

  • Post a Perl-related web page to Digg, reddit or any of the other social bookmarking websites.
  • Bring someone new to a Perl Mongers meeting, even if Perl isn't his/her primary language. Especially if it isn't his/her primary language.
  • Go to a user group meeting for another language. See what they're talking about. Share the Perl perspective.
  • Start a blog outside of the use.perl.org hierarchy. If you already have a u.p.o journal, keep it, and post to both places.
  • Start a mailing list related to a project of yours. Host it somewhere like Google Groups. Don't worry that "people won't be able to find it," because the list of groups at lists.perl.org is no model of organization, either. Besides, the only people who know to look at lists.perl.org are the people that already know there are lists at lists.perl.org. They're not who we're after.
  • Start an alternative to perlmonks.org. Perlmonks is a fine site, but it's long in the tooth, daunting to newbies, and frustrating to search. Surely there's a different way to have an online Perl forum that is better in many cases.
  • Work on a project that helps cross-pollinate Perl to the rest of the world. The Test Anything Protocol is a good example of this.
  • Review a book about Perl. Get it posted to Slashdot or another big tech site.
  • Write about Perl in your blog. Even if you think it's not interesting to Perl people, that's fine. Someone will want to read it, and you'll spread that mindshare.
  • When you talk about Perl, don't be afraid to use the F-word and the L-word: "fun" and "love".

What do you think?

I'd like to hear your ideas. How can we expand Perl's reach? What have you done to help decentralize, diversify and colonize? How can we keep Perl fun and exciting?

Two comments I don't need to hear: "This will never work" and "This is a waste of time." If that's what you have to contribute, trust that I've taken your insight into account already, and save yourself some typing. Alternative courses of action, however, are more than welcome. Also unnecessary: "None of this would be a problem if not for Perl 6."

Acknowledgements

Thanks to a number of people who have helped discuss these ideas with me as I put them together. These include Pete Krawczyk, Jeffrey Thalhammer, Ricard Signes, Liz Cortell, David Hand, Jesse Vincent and Elliot Shank. I'm sure I've forgotten some, so I apologize if I left you off the list.

23 Comments

I totally agree with this article and support the actions needed for helping. Some plans are on the way already.

I'm def. not saying that this is a replacement for lists.perl.org, but there's a lot of instances where a MailMan list can be replaced by a program written by Perl called Dada Mail (which is a project of mine)

http://mojo.skazat.com

It's a lot more user friendly that MailMan, from a user perspective.

A lot of the questions on the boards ask if the upgrade to php on their hosting accounts will break it, which I have to reply, "it ain't a php app, it's Perl".

The space for web-apps that are as easy to install as PHP web apps seems small for Perl-dom. I know a lot of my questions on Perlmonks dealing with the difficulties is answered with, "You just don't do that". I usually reply with, "Well, that's how it's gotta be done"

One of the problems is relying on CPAN modules, but having users that don't know how to install CPAN modules themselves, so you have to bundle them yourself. That's usually when the holy war on PM starts...


Although I agree that the visibility of Perl could and should be improved, I don't think decentralization is the answer. A thing that bugs me is that I see modules pop up on freshmeat and other sites.



I don't want to check sites besides CPAN to find the best modules. Sure, you can build a community around it at google or wherever, but do keep CPAN fresh and up to date. Freshmeat is not the right way to go in my humble opinion.



Also I disagree to move "everything" over to google (as your examples show). But that's mainly because of the fact that I'd rather not touch anything that will give me -and requires me to accept- the infamous g-cookie.



Diversify. Mmm, not sure on that one either. Newcomers to Perl often have a lot of problem with simple tasks such as parsing XML or creating RSS (sending mail, anyone?). What module to use for this? Of course, choice is good, TIMTOWTDI, but seriously, what is "better" for the newcomer? XML::LibXML? XML::Simple? XML::Twig? XML::Easy? XML::Tree? There is an overload of modules that can handle this for you and it all boils down to your preference, I think. Hard for newcomers.



A thing that works for PHP and less for Perl is that Perl authors mainly publish modules to CPAN, where PHP authors mainly post "complete" projects. Also the installation is usually a lot easier with PHP than bulky Perl projects (although I now and then see some good catching up). Maybe we need some perlMyAdmin (ok, that does exist ;-) ) and other big, heavily used applications with easy installation.

One of the ways of attracting new blood to a language or project is to make sure it's a lot of fun.

I started using Perl more than a decade ago because of work, but I kept using it because it was fun. Larry's humour in the first edition Camel was fantastic, and that made learning the language a delight.

However I feel that Perl has recently been falling behind in the fun department. I certainly don't see any game development environments in Perl, and let I'm constantly hearing about pygame and pyglet from my Python associates, and talks on these technologies attract big crowds at OSDC. Indeed, the last awesome game I saw in Perl was Frozen Bubble, and you wouldn't even know it was a Perl program looking at the website.

Writing a Perl game development framework hits all the right buttons. It's new, it's cool, it's fun, and it's modern. It's easy to encourage new people to use, it's easy to have lots of milestones that can be reached, and so it's easy to generate lots of news.

I think this is all pretty interesting and goes along with the talks everyone has been having about "rethinking CPAN". Just to add my two cents:

1. I think frameworks like Ruby on Rails (and by association the Ruby language itself) caught on fire because a lot of attention was focused on making things 'easy' for the beginner. This usually means being very opinionated and rigid in what steps you make beginners go through to find success with their projects...which of course goes against a core idea of Perl. We all seem to be afraid to guide newbies with an iron fist, and as such we provide so many paths that a beginner becomes frozen or overloaded with choices.

We need to lay down some clear cut, simple, newbie paths to success with popular projects.

Quick ideas:

- A starters guide to writing a Blog system with Perl

- A starters guide to writing Twitter bots with Perl (I actually covered this in a few of my recent blog posts but smarter people could cover it better!)

- Any popular topic being covered in Ruby, Python, or Java..."how to do X with Perl".

2. Personally, I think Perl is also suffering from being somewhat of a mature language...there are not a lot of books being published on Perl because most of the 'important' things about Perl have already been published...as a result, newbies don't have a lot of 'new' books to make them consider Perl...so there are less newbies interested in Perl...and so there is less demand for Perl books to be written...and the cycle feeds itself until "nobody uses Perl anymore" becomes common...

We need to get more things published about using Perl with hip ideas.

Quick ideas:

- SMS with Perl.

- Perl bots (agents for Twitter, Jabber, and other interactive agents).

- Ajax with Perl (probably catalyst related).

- Web Services with Perl.

- Anything and everything catalyst related (Perl and the MVC concept).

- Any popular topic being written about for Ruby, Python, or Java..."how to do X with Perl".

@b10m: I'm absolutely not suggesting "moving everything to Google." I am suggesting that you simply ask if the "default" tools in CPAN are the best for you, and if not, use something else. My examples use Google, because that's what I use. There are plenty of other choices as well.

"There is an overload of modules that can handle this for you and it all boils down to your preference, I think. Hard for newcomers." That's why we need to have additional resources to help users navigate these choices. Here's a baby step from the Perl wiki: http://www.perlfoundation.org/perl5/index.cgi?recommended_xml_modules

For some things centralisation is good - for others not. Perl main strenght is the centralised repository of CPAN. The same with diversification - Perl is already very diversified in terms of available modules (just take the XML libraries example that someone mentioned above) - you cannot reverse the negative tendencies of Perl adaptation by doing more of the same. One exception from this is the DBI module - there are no other database access layers used - and in effect this is a big win for Perl - because standardisation on this one abstraction let the developers concentrate on the diversity of the DBD drivers.


In general I would be careful with such unqualified statements. What we need is a case by case analysis - and in many cases that you mention in the article above I do agree with you.

And for example see http://www.madstop.com/ruby/ruby_has_a_distribution_problem.html for arguments why for some things centralisation and standardisation on some process is good.

Just wanted to say that I thought that Andy here makes a lot of sense.

I was inspired to throw together http://perlchan.org/ by this post. It's an anonymous message board in the style of 2ch and 4chan. I'm interested in any thoughts anyone has on such a site.

Perceptions. Yep, they do define what is "hip/pop/in". I'm not a Perl-guy and only happened here because I thought Perl was indeed dead, or dieing. Now, really I know there is a strong Perl contingent and am aware of many who would use it before (foo). I know enough about these languages and frameworks to get myself through a decision matrix...and doing so _most likely_ wouldn't ever reveal Perl as the winner - unless it was weighted accordingly.

There are two factors that are missing from this discussion. One, the language itself and the ability to learn it. Two, is that pesky academic twist. Combined, these two factors reveal that people just aren't _choosing_ to learn Perl. This could be based on misconceptions (though I doubt it).

As was mentioned above - Ruby is really attractive because it is really easy to get started...and even easier to pick the pieces that are necessary to get the job done. I think Python has this appeal too.

That said, the gap from PHP to Perl - for someone who doesn't know either will definitely seem awkward. But, really it does seem awkward for someone who knows enough to make a decision.

Just my perception...

I guess another problem, is just how crazy the Catalyst project appears. It wasn't that long ago, when one of the core members decided to leave. And in a case of extreme overreaction, the mailing list was locked down, and all postings restricted.


The Catalyst web site has gotten bad too. The Roadmap on their site is an embarrassment:

http://dev.catalystframework.org/roadmap

Catalyst may be a good product, but it exhibits so many signs of being just another dysfunctional open source product.

I agree with Andy, but one of the most troubling areas I see for Perl's general perception is application packaging/installation. Perl is far and away my favorite language when I am developing anything, but I am embarrassed to say that I often shy away from other peoples applications developed in Perl because of the installation headaches. I love CPAN and the associated tools when developing, but if I am just installing an app, I don't want to worry about module dependencies, test failures, etc. More often than not, when installing dependencies from CPAN one of them is going to require manual intervention. The few times I have used gems (Ruby) or easy_install (Python), they have been quick and painless even if there are many fewer modules. That includes installations under Cygwin and big packages like Rails... I've never had such luck with Catalyst.

A good way to get new users comfortable with Perl is to lure them in with great apps. Once they see that apps written in Perl and deployed painlessly, they will think about developing in Perl. If users become used to cpan install App::Ack, and it works without a hitch, they will be more likely to spend more time with Perl.

So I have a few suggestions. One, get rid of the Scripts portion of CPAN. Move all those projects to the App:: namespace, and actively promote (and police) the usage of that space for full applications. Two, recruit popular projects written in Perl to distribute their applications through CPAN and document the standard install process on their websites. Apps like GoldenPod, Frozen Bubble, SlimSever could draw people to Perl. Three, make cpan install App::myapp as painless as gems/easy_install. Maybe the tests suites could be segmented into development and absolutely necessary, and only the basic tests would be run for application installation.

I would be happy to help implement/promote this more, but I'm not exactly sure how/where to do that.

It's fine to post your modules in places other than CPAN, but I think it would be mistake to post them other places *instead* of CPAN.

There's a nearly 0% chance I'm going to use a module, no matter how cool, that I can't install from the CPAN shell.

Some thoughts from a non-programer:

I think PHP has become a default web language at Perl's expense mainly for one reason that's unrelated to its strengths or weaknesses as a language compared to Perl: venue.

PHP's venue / IDE is the web page editor/browser. By default, Perl's is not. PHP therefore attracts the great majority of new web coders - not because PHP is better but because the tools they need to begin with PHP are already in their hands.

I think new coders see PHP as an extension of HTML! And I think they see Perl and others as too hard/fussy about implementation.

Now I'm wildly speculating, but I'd even venture to guess that most new PHP programmers do not even pick up another language in 5 years. Taking the path of least resistance doesn't exactly encourage curiosity about the things that lay outside of it..the superiority of those things notwithstanding.

@Tom Samplonius said that Catalyst looks dysfunctional. Thanks for the comments, although I think you're wrong. Not very good at creating buzz/mindshare or whatever, probably, but I really think that's as far as it goes.

The whole "originator of the project leaving" about 18 months ago was threatening to blow out in a way that put the future of the project, and people's livelihoods at risk. It was dealt with in a way that prevented forking and means we have a stable future. As for the roadmap link, we don't actually use that feature in trac at all these days. I've asked for that link to be nuked.

So the website needs some love - we're slowly migrating away from trac to RT/Mojomojo (catalyst wiki) and svnweb. We're nearly there, but none of this is on any of the developer's core time, so the process is slow. I did some estimates of the catalyst user-base, and I think that it grew by about 50% over the past year.

@andy/others

That's why we need to have additional resources to help users navigate these choices.

I indeed would really like a fancy/hip site where you can find solutions to common problems, as mentioned above as well. Maybe a mix between the perlfoundation XML suggestions, and the Q&A site on Perlmonks (which is hard to navigate). If anyone gets a nice site online, I'm willing to contribute :)

@Tom Samplonius said that Catalyst looks dysfunctional. I agree!!!!

Great article! One thing that could make it much more easy to have your own blog outside of use perl would be a feature that would allow you to add your blog's rss feed to your use perl account, so that entries with a perl tag from your blog automatically appear as journal posts on use perl. That way one would retain the visibility in the Perl community that comes with posts on use perl while having the wide spread visibility in the overall blogging community.

This is a great idea--but I'm surprised to see it's missing advice on how to combat the massive amounts of Perl / Parrot FUD out there. Just last night I was discussing Parrot and Perl with someone. First words out of their mouth were "Perl's past it's [sic] hayday". I asked them what they thought about Parrot and they called it "vapor". I responded saying that you can download and use Parrot right now; the response was "parrot 1.0 is vapor". I happen to use a very popular and respected IRC client for which 1.0 is also vapor--does this mean that the many irssi users across the planet should abandon it for something with a 1.0 release?

I may just suck at talking to people, and I may just have bad luck, but non-Perl people consistently throw FUD in my face whenever I mention Perl or Parrot or anything related to the two. I don't know how to deal with it.

There is no Mailman running on lists.perl.org. lists.perl.org isn't even lists.perl.org, it's really lists.cpan.org, and isn't run by the perl.org people.

just my very personal 2c:

I was working with perl for more than 14 years and will work with perl because I love this language.
I dont think the problem with general IT market perception of perl can be solved by making perl cooler or decentralizing CPAN or by blogging on every possible blogpost and podcasting every step of perl6 misadventure. It seems to me that current focus of the perl community (ok, leftovers of the great perl community) are way off the target. There is nothing wrong with perl5. The problem is not in perl but rather in software built with perl, more importantly with absolutely inappropriate by IT industry standards ways how perl software packages are maintained, supported and developed. Try to search those infamous XML:: modules.
90% of the will have word Simple,Tiny,Bare ...etc and all of those modules are not good for serious job.
I think this is a time for perl community to grow up and come out of infinite hackaton into the real world.
The start could be taken with cleaning up CPAN.
If module is obsolete, support is nonexistent and
it wasn't downloaded once in the past 2 month then move this abandoneware into the "cemetery". And start moderate new upcoming modules. The reviews are great, add "downloaded" rating on each module as well and make release process of the module more formal.
The Perl::Critic started very promising trend in bringing enterprise software qualities into the perl. Although, its hard to follow in the real production environment some of the Perl::Critic rules, it would benefit acceptance of perl as the best choice of software language if tomorrow every perl developer start using it.
Java jumped out of the "nerds only" ring because of the gang of 4 dubious patterns book. In the past time the only comments on the patterns for perl I've seen so far were that perl already is so superior that it does not need any patterns.
Try to search for java jobs in Chicagoland and repeat the same search for perl. The ratio would be 4:1 in any given day. Why, because standard way to do things makes project managers on large scale software projects very happy and "many ways to do the almost the same thing and then forget what exactly it was meant to be" doesn't.
And Java delivers just that and perl does not.
So, the major point here is to shift attention of the perl community into the delivering highly supportable by any average perl developer packages with standard, I repeat standard interfaces. Yes perl software has a lot of great examples of the nice code and yes, the DBI module is the greatest example of all. Actually, the flock of ORM modules are trying to employ very similar patterns and interfaces as well. The only question remains who is going to undertake this kind of initiative?

Firstly a disclaimer: I am a complete and total newbie both with perl and posting on perlbuzz so please be gentle if I do or say anything completely asinine.

Secondly, and more importantly, regarding the need to spreadperl to the rest of the world, I have noticed one project that could help greatly here. There is an offshoot of the wikipedia project called wikiversity: http://en.wikiversity.org/
Wikiversity is supposed to be an attempt to create a free repository for learning projects and materials.

Like wikipedia, it is just another wiki that anyone can post to. There is a full category dedicated to perl (http://en.wikiversity.org/wiki/Perl) that has some useful information on it. However, the category itself could be much better developed and could use quite a few more veteran perl coders inputs. Furthermore, having a central learning repository with the prefix 'wiki' attached to it would help drawn in readers that just generally like to fiddle about on wikipedia and other such information sites.

I guess what I am getting at is that contributing to something like the wikiversity perl category would help make perl more accessible to folks who are interested in working more in depth with their computers, but don't know enough about where to find good learning materials about something like perl. I've already posted a couple of things on wikiversity, but, frankly, the perl category is lacking terribly. I think this might be an artifact of wikiversity being relatively young, but still, developing some content here could help, especially if wikiversity gets as big as wikipedia.

Just a thought.

Also, I apologize if by posting on this article I am reviving what might be considered a dead topic. This article just happened to hook my interest as someone who is working hard to integrate into the perl culture and finding it terribly overwhelming and yet equally enticing.

Cheers,
Brady

Leave a comment

Job hunting for programmers


Land the Tech Job You Love, Andy Lester's guide to job hunting for programmers and other technical professionals, is available in PDF, ePub and .mobi formats, all DRM-free, as well as good old-fashioned paper.