Tools

All about the new Test2 framework and how it will help your tests

May 30, 2016 CPAN, Interviews, Perl 5, Tools No comments

The new Test2 framework has been released after a couple years of development. I wanted to find out about what this means for users of Test::Simple and Test::More, so I chatted with the project leader, Chad Granum (exodist).

Andy Lester: So Test2 has just been released after a couple of years of work, and a lot of discussion. For those of us who haven’t followed its development, what is Test2 and why is it a good thing?

Chad Granum: The big changes will be for people who write test modules. The old Test::Builder was tied to specific generation of TAP output. That’s been replaced with a flexible event system.

It all started when David Golden submitted a patch to change the indentation of a comment intended for humans who read the test. The change would help people, but meant nothing to the machine. I had to reject the patch because it broke a lot of downstream modules. Things broke because they tested that Test::Builder produced the message in its original form. I thought that was crazy, and wanted to make things easier to maintain, test, and improve.

Andy: Test::Builder’s internals were pretty fragile?

Chad: That is true, but that’s not the whole picture. The real problem was the tools people used to validate testing tools. Test::Builder::Tester was the standard, and it boiled down to giant string comparisons of TAP output, which mixes messages for the computer’s use, and messages for human use.

While most of the changes are under the hood, there are improvements for people who just want to write tests. Test2 has a built-in synchronization system for forking/threading. If you modify a test to load Test2::IPC before loading Test::More, then you can fork in your tests and it will work in sane/reasonable ways. Up until now doing this required external tools such as Test::SharedFork which had severe limitations.

Another thing I want to note is an improvement in how Test2 tracks file+line number for error reporting purposes. As you know diagnostics are reported when a test fails, and it gives you the filename and line number of the failure. Test::Builder used a global variable $Test::Builder::Level which people were required to localize and bump whenever they added a stack frame to their tool. This was confusing and easy to get wrong.

Test2 now uses a Context object. This object solves the problem by locking in the “context” (file + line) when the tool is first called. All nested tools will then find that context. The context object also doubles as the primary interface to Test2 for tool writers, which means it will not be obscure like the $Level variable was.

Andy: I just counted 1045 instances of $Test::Builder::Level in my codebase at work. Are you saying that I can throw them all away when I start using Test2?

Chad: Yes, if you switch to using Test2 in those tools you can stop counting your stack frames. That said, the $Level variable will continue to work forever for backwards compatibility.

Andy: Will the TAP output be the same? We’re still using an ancient install of Smolder as our CI tool and I believe it expects TAP to look a certain way.

Chad: Extreme care was taken to ensure that the TAP output did not change in any significant ways. The one exception is David Golden’s change that started all this:

Ok 1 - random test
    # a subtest
    Ok 1 - subtest result
    1..1
Ok 2 - a subtest

This has changed to:

Ok 1 - random test
# a subtest
    Ok 1 - subtest result
    1..1
Ok 2 - a subtest

That is the change that started all this, and had the potential to break CPAN.

Andy: So Test2 is all about possibilities for the future. It’s going to make it easier for people to create new Test:: modules. As the author of a couple of Test:: modules myself, I know that the testing of the tests is always a big pain. There’s lots of cut & paste from past modules that work and tweaking things until they finally pass the tests. What’s different between the old way of doing the module testing and now?

Chad: Test::Builder assumed TAP would be the final product, and did not give you any control or hooks into everything between your tool and the TAP, as such you had to test your final TAP output, which often included text you did not yourself produce. In Test2 we drop those assumptions, TAP is no longer assumed, and you also have hooks in almost every step of the process between your tool and the final output.

Many of the actions Test::Builder would accomplish have been turned into Event objects. Test tools do their thing, and then fire events off to Test2 for handling. Eventually these events hit a formatter (TAP by default) and are rendered for a harness. Along with the hooks there is a tool in Test2::API called intercept, it takes a codeblock, all events generated inside that codeblock are captured and returned, they are not rendered and do not affect the global test state. Once you capture your events you can test them as data structures, and ignore ones that are not relevant to your tools.

The Test::Builder::Tester way may seem more simple at first, but that is deceptive. There is a huge loss of information. Also if there are changes to how Test::Builder renders TAP, such as dropping the ‘-‘ then everything breaks.

Using Test::Builder::Tester

test_out("ok 1 - a passing test");
ok(1, 'a passing test');
test_test("Got expected line of TAP output");

Using intercept and basic Test::More tools

my $events = intercept {
    ok(1, 'a passing test');
};

my $e = shift @$events;

ok($e->pass, "passing tests event");
is($e->name, "a passing test", "got event name");
is_deeply(
    $e->trace->frame,
    [__PACKAGE__, __FILE__, 42, 'Test2::Tools::Basic::ok'],
    "Got package, file, line and sub name"
);

Using Test2::Tools::Compare

like(
    intercept {
        ok(1, 'a passing test');
    },
    array {
        event Ok => sub {
            call pass => 1;
            call name => 'a passing test';

            prop file    => __FILE__;
            prop package => __PACKAGE__;
            prop line    => 42; 
            prop subname => 'Test2::Tools::Basic::ok';
        };
    },
    'A passing test'
);

Andy: What other features does Test2 include for users who aren’t creating Test:: modules?

Chad: Test2’s core, which is included in the Test-Simple distribution does not have new features at the user level. However Test2-Suite was released at the same time as Test2/Test-Simple, and it contains new versions of all the Test::More tools, and adds some things people have been requesting for years, but were not possible with the old Test::Builder

The biggest example would be “die/bail on fail”, which lets you tell the test suite to stop after the first failure. The old stuff could not do this because there was no good hook point, and important diagnostics would be lost.

It’s as simple as using one of these two modules:

use Test2::Plugin::DieOnFail;
use Test2::Plugin::BailOnFail;

The difference is that DieOnFail calls die under the hood. The BailOnFail will send a bail-out event which will abort the current file, and depending on the harness might stop the entire test run.

Andy: So how do I start using Test2? At my day job, our code base has 1,200 *.t files totalling 282,000 lines of code. Can I expect to install the new version of Test::Simple (version 1.302019) that includes Test2 and everything will “just work”?

Chad: For the vast majority of cases the answer is “yes”. Back-compatibility was one of the most significant concerns for the project. That said, some things did unfortunately break. A good guide to what breaks, and why can be found in this document. Usually things that break do so because they muck about with the Test::Builder internals in nasty ways. Usually these modules had no choice due to Test::Builder’s limitations. When I found such occurrences I tried to add hooks or APIs to do those things in sane/reasonable ways.

Andy: Do I have to upgrade? Can I refuse to go up to Test-Simple 1.302019? What are the implications of that?

Chad: Well, nobody is going to come to you and force you to install the latest version. If you want to keep using your old version you can. You might run into trouble down the line if other Test:: tools you use decide to make use of Test2-specific features, at which point you would need to lock in old versions of those as well. You also would not be able to start using any new tools people build using Test2.

Andy: And the tools you’re talking about are Test:: modules, right? The command line tool prove and make test haven’t changed, because they’re part of Test::Harness?

Chad: Correct. Test::Harness has not been touched, it will work on any test files that produce TAP, and Test2 still produces TAP by default. That said I do have a project in the works to create an alternative harness specifically for Test2 stuff, but it will never be a requirement to use it, things will always work on Test::Harness.

Andy: So if I’m understanding the Changes file correctly, Test-Simple 1.302012 was the last old-style version and 1.302014 is the new version with Test2?

Chad: No, Test-Simple-1.001014 is the last STABLE release of Test-Simple that did not have Test2, then Test-Simple-1.302015 was the first stable release to include Test2. There were a lot of development releases between the 2, but no stable ones. The version numbers had to be carefully crafted to follow the old scheme, but we also had to keep it below 1.5xxxxx because of the previous maintainers’ projects which used that number as well as 2.0. Some downstream users had code switched based on version number and expected an API that never came to be. Most of these downstream distributions have been fixed now, but we are using a “safe” version number just in case.

Andy: What has development for this been like? This has been in the works for, what, two years now? I remember talking to you briefly about it at OSCON 2014.

Chad: At the point we talked I had just been given Test-Simple, and did not have any plans to make significant changes. What we actually talked about was my project Fennec which was a separate Test::Builder based test framework. Some features from Fennec made their way into Test2, enough so that Fennec will be deprecated once I have a stable Test2::Workflow release.

Initially development started as a refactor of Test::Builder that was intended to be fairly small. The main idea was to introduce the events, and a way to capture them. From there it ballooned out as I fixed bugs, or made other changes necessary to support events.

At one point the changes were significant enough, and broke enough downstream modules that I made it a complete fork under the name Test-Stream. I figured it would be easier to make Test::Builder a compatibility wrapper.

In 2015, I attended the QA hackathon in Berlin, and my Test-Stream fork was a huge topic of conversation. The conversation resulted in a general agreement (not unanimous) that it would be nice to have these changes. There was also a list of requests (demands?) for the project before it could go stable. We called it the punch-list.

After the Berlin hackathon there was more interest in the project. Other toolchain people such as Graham Knop (Haarg), Daniel Dragan (bulk88), Ricardo Signes (rjbs), Matt Trout (mst), Karen Etheridge (ether), Leon Timmermans (leont), Joel Berger (jberger), Kent Fredric (kentnl), Peter Rabbitson (ribasushi), etc. started reviewing my code, making suggestions and reporting bugs. This was one of the most valuable experiences. The project as it is now, is much different than it was in Berlin, it is much better from the extra eyes and hands.

A month ago was another QA hackathon, in Rugby UK, and once again Test2 was a major topic. This time the general agreement was that it was ready now. The only new requirements on the table were related to making the broken downstream modules very well known, and also getting a week of extra cpan-testers results prior to release.

I must note that at both QA hackathons the decisions were not unanimous, but in both cases there was a very clear majority.

Andy: So what’s next? I see that you have a grant for more documentation. Tell me about that, and what can people do to help?

Chad: The Test2 core API is not small, and has more moving pieces than Test::Builder did. Right now there is plenty of technical/module documentation, but there is a lack of overview documentation. There is a need for a manual that helps people find solutions to their problems, and tied the various parts together. This is the first part of the manual docs for tool authors.

Test2::Suite is also not small, but provides a large set of tools for people to use, some are improvements on old tools, some are completely new. The manual will have a second section on using these new tools. This second part of the manual will be geared towards people writing tests.

The best way for people to help would be to start using Test2::Suite in their tests, and Test2 in their test tools. People will undoubtedly find places where more documentation is needed, or where things are not clear. Reporting such documentation gaps would help me to write better documentation. (Test::More repo,
Test2::Suite repo)

Apart from the documentation, I have 2 other Test2 related projects nearing completion: Test2-Workflow, which is an implementation of the tools from Fennec that are not a core part of Test2, and Test2-Harness which is an optional alternative to Test::Harness. Both are pretty much code-complete on GitHub, but neither has the test coverage I feel is necessary before putting them on CPAN.

Andy: Thanks for all the work that’s gone into this, both to you and the rest of those who’ve contributed. It sounds like we’ll soon see more tools to make testing easier and more robust.

ack 2.0 has been released

April 19, 2013 Tools No comments

ack 2.0 has been released. ack is a grep-like search tool that has been optimized for searching large heterogeneous trees of source code.

ack has been around since 2005. Since then it has become very popular and is packaged by all the major Linux distributions. It is cross-platform and pure Perl, so will run on Windows easily. See the “Why ack?” page for the top ten reasons, and dozens of testimonials.

ack 2.0 has many changes from 1.x, but here are four big differences and features that long-time ack 1.x users should be aware of.

  • By default all text files are searched, not just files with types that ack recognizes. If you prefer the old ack 1.x behavior of only searching files that ack recognizes, you can use the -k/--known-types option.
  • There is a much more flexible type identification system available. You can specify a file type based on extension (.rb for Ruby), filename (Rakefile is a Ruby file), first line matching a regex (Matching /#!.+ruby/ is a Ruby file) or regex match on the filename itself.
  • Greater support for ackrc files. You can have a system-wide ackrc at /etc/ackrc, a user-specific ackrc in ~/.ackrc, and ackrc files local to your projects.
  • The -x argument tells ack to read the list of files to search from stdin, much like xargs. This lets you do things like git ls | ack -x foo and ack will search every file in the git repository, and only those files that appear in the repository.

On the horizon, we see creating a framework that will let authors create ack plugins in Perl to allow flexibility. You might create a plugin that allows searching through zip files, or reading text from an Excel spreadsheet, or a web page.

ack has always thrived on numerous contributions from the ack community, but I especially want to single out Rob Hoelz for his work over the past year or two. If it were not for Rob, ack 2.0 might never have seen the light of day, and for that I am grateful.

A final note: In the past, ack’s home page was betterthangrep.com. With the release of ack 2.0, I’ve changed to beyondgrep.com. “Beyond” feels less adversarial than “better than”, and implies moving forward as well as upward. beyondgrep.com also includes a page of other tools that go beyond the capabilities of grep when searching source code.

For long time ack users, I hope you enjoy ack 2.0 and that it makes your programming life easier and more enjoyable. If you’ve never used ack, give it a try.

Low-tech high-speed OS X program launching

July 19, 2010 Tools 4 comments

I work in my OS X Terminal window all day long. When I want to run iCal or Address Book, I don’t want to be bothered with clicking around to find the app, even though they live in my Dock. I could also use a program launcher like [Alfred](http://alfredapp.com/), which I like, but want it even faster.
For me, the fastest way to open iCal while I’m in the shell is to run “ical” from the prompt, which launches the app.
My ~/bin/ical program is simply
#!/bin/sh
open /Applications/iCal.app/
and my ~/bin/addr is
#!/bin/sh
open “/Applications/Address Book.app/”
You might think that it’s overkill to write a shell program for such a silly task, but it’s all about optimizing my time at the keyboard for my common cases.
Someone will note that I could have used a shell alias, and that’s true, too. Either way, I want a super simple way to get the apps I use most often.

Fixing my #1 bash annoyance

July 19, 2010 Tools 2 comments

Ever open up a Terminal window and hit Tab and Bash sits for a few seconds until it finally comes back and asks
Display all 2224 possibilities? (y or n)
Because it went and compiled a list of EVERY executable you could possibly want to execute?
Yeah, me, too, and I hate it. The fix is simple. Add this to your ~/.bashrc
shopt -s no_empty_cmd_completion
No more completion on nothing!

Cool vim plugin of the day: surround.vim

December 8, 2008 Tools No comments

The always amazing Peteris Krumins is starting a series of blog postings on cool plugins for vim, starting with [this post about surround.vim](http://www.catonmat.net/blog/vim-plugins-surround-vim/).
Just reading the article about the “old way” to do what surround.vim does was informative, showing me vim tricks I didn’t know.
Seriously, anyone who reads Mechanix should have [Peteris’ blog](http://www.catonmat.net/) in her syndication reader.

Git is my hero

September 24, 2008 Tools No comments

By Selena Deckelmann

Last night, an incredible number of people showed up to Code-N-Splode to hear about Git.

I was not at all prepared for that number of people, but grateful that a couple experts were on hand to help – Sarah Sharp, Audrey Eschright and Michael Schwern (and others) all contributed to the discussion. Thanks for all the great questions! I’m looking forward to Sarah’s advanced tutorial next month.

Selena Deckelmann leads the Portland PostgreSQL Users Group and regularly contributes to Code-N-Splode, a group dedicated to getting more women involved in open source. She likes Perl. She’ll be running a code sprint and giving a lightning talk on having more fun with open source at PostgreSQL Conference West, on October 17-19, 2008.

Use Getopt::Long even if you don’t think you need to

May 25, 2008 Code craft, Tools 3 comments

Thread over on perlmonks talks about Tom Christiansen’s assertion that you should use it, by default, even when you only have one command-line argument to parse:

What seems to happen is that at first we just want to add–oh say for example JUST ONE, SINGLE LITTLE -v flag. Well, that’s so easy enough to hand-hack, that of course we do so… But just like any other piece of software, these things all seem to have a way of overgrowing their original expectations… Getopt::Long is just *wonderful*, up–I believe–to any job you can come up with for it. Too often its absence means that I’ve in the long run made more work for myself–or others–by not having used it originally. [Emphasis mine — Andy]

I can’t agree more. I don’t care if you use
Getopt::Long or
Getopt::Declare or
Getopt::Lucid or any of the other
variants out there. You know know know that you’re going to add more arguments down the road, so why not start out right?

Yes, it can be tricky to get through all of its magic if you’re unfamiliar with it, but it’s pretty obvious when you see enough examples. Take a look at prove or ack for examples. mech-dump is pretty decent as an example as well:

GetOptions(
'user=s'        => $user,
'password=s'    => $pass,
forms           => sub { push( @actions, &dump_forms ); },
links           => sub { push( @actions, &dump_links ); },
images          => sub { push( @actions, &dump_images ); },
all             => sub { push( @actions, &dump_forms, &dump_links, &dump_images ); },
absolute        => $absolute,
'agent=s'       => $agent,
'agent-alias=s' => $agent_alias,
help            => sub { pod2usage(1); },
) or pod2usage(2);

Where the value in the hashref is a variable reference, the value gets stored in there. Where it’s a sub, that sub gets executed with the arguments passed in. That’s the basics, and you don’t have to worry about anything else. Your user can pass –abs instead of –absolute if it’s unambiguous. You can have mandatory flags, as in agent=s, where –agent must take a string. On and on, it’s probably got the functionality you need.

One crucial reminder: You must check the return code of GetOptions. Otherwise, your program will carry on. If someone gives your program an invalid argument on the command-line, then you know that the program cannot possibly be running in the way the user intended. Your program must stop immediately.

Not checking the return of GetOptions is as bad as not checking the return of open. In fact, I think I smell a new Perl Critic policy….

What commands do you run?

April 16, 2008 Tools 2 comments

People have been posting in their blogs about what command they run, based on their shell histories. The command that I’ve seen looks like this:

history|awk '{a[$2]++} END{for(i in a){ 
printf "%5dt%s n",a[i],i}}'|sort -rn|head

That works, of course, but who wants to use awk and the shell? I pulled out the old Data::Hash::Totals module I wrote a while back, along with Perl’s built-in awk simulation:

$ history | perl -MData::Hash::Totals -ane'$x{$F[1]}++;' 
-e'END{print as_table(%x, comma => 1)}' | head
207 vim
143 svn
125 make
90 ack
77 cd
45 sdvx
34 ssq
31 ls
25 ./login-fixup
19 tail
alester:~ : cat `which sdvx`
#!/bin/sh
svn diff -x -w $* | view -

and ssq is just an alias for svn status -q.

Don’t use Plucene for real work

March 25, 2008 Tools 2 comments

Just in case anyone’s using Plucene as a search engine, please don’t. It’s terribly slow. Take a look at KinoSearch instead. Plucene is written in pure Perl, mostly as a proof of concept. It is literally orders of magnitude slower than KinoSearch, which uses C for the hard bits. Here’s a page of benchmarks to make the point.

ack 1.78 is out

March 24, 2008 Tools 1 comment

After three months of lots of development work and intermediate releases, I’ve released ack 1.78. There are tons of new features and lots of compatibility fixes for Windows. ack is a replacement for grep that is geared to working with trees of code.

Highlights in this release include:

  • Files specified on the command line are always searched, even if they don’t match a known filetype
  • Ability to ignore directories
  • Pager support
  • More flexible grouping options
  • Many more languages recognized and existing ones improved, including CFMX, Actionscript, assembly, Tcl, Lisp, Smalltalk
  • Ability to define your own languages based on filetype

ack may well change the way you work on the command-line with source code. Try it out and let me know what you think. You can install it by installing App::Ack from CPAN, or downloading the standalone version to your ~/bin directory.