stats about cpan modules shipped by mandriva

adam and gabor asked me about some information on what cpan modules are shipped as rpm packages by mandriva... i finally found some time to work on it and produced ordb::cpan::mandriva.

this module is basically a sqlite database replicated automatically by orlite::mirror. using it is very simple - for example, couting the cpan dists shipped by mandriva is achieved by the following snippet:

(fyi, the result is 1899 as of writing, +1 since i just imported the module itself)

at module use time, it will automatically download (and cache for a week) the sqlite database. and one can then use the module as an ordb for this db... the module does not however produces the stats for you - so alias & gabor, now is your turn to work! :-) i'm waiting for your top-100 most wanted mandriva or whatever crazy stats you'll want to produce.

the database itself is generated by module::packaged::generator. this module uses different drivers depending on the current linux distribution (but nothing prevents *bsd to also have a driver) it runs on. only mandriva is supported currently, but ryan will work on a debian driver. all other contributions are most welcome - the code is on github: fork it and send me pull requests!

next padre version will (finally) recognize dist-zilla projects

padre 0.55, due tomorrow, will finally recognize dist-zilla projects correctly. among other things, it means that it will set @INC accordingly for syntax checking, keep the directory browser at the project root, etc.

it took a bit of time since the "installer" detection is spread out in different places in padre... this needs refactoring, for those interested.

some dzil goodness coming

i scratched some itches in dist-zilla, and hopefully the result will be useful to you too... some are rather trivial, while some are more feature-ful.

on the it-s-the-details-that-count front, dzil clean will now remove *~ files lingering in your local copy.

dzil command now accepts a -I argument that adds a directory to perl @INC (same as perl's -I option). which means dzil plugin author can now do:

$ dzil -Ilib release

and release their distribution using the latest version of their plugin...

a new run subcommand has been added, which is doing more or less the following:

$ dzil build
$ rsync -avp My-Project-version/ .build/
$ cd .build
$ perl Makefile.PL     # or perl Build.PL
$ make                 # or ./Build     
$ export PERL5LIB=$PWD/blib/lib:$PWD/blib/arch
$ launch command defined by rest of your args

which means you can now do:

$ dzil run ./bin/myscript
$ dzil run prove -l t/unit.t
$ dzil run bash

the first one is specially useful if you're using file::sharedir and don't want to add extra hooks detecting whether it runs in a dev checkout or not.

speaking of sharedir, i crafted a quick'n'dirty implementation that made modulebuild plugin automatically install an existing share directory (which is possible since module::build 0.36). rjbs then re-factored it to use the installdirs plugin. nperez then finalized the work for the makemaker plugin. so if you have a share directory, it will now be automatically recognized as such by dzil and installed accordingly.

rjbs just released a new version of dist-zilla with those enhancements (and other stuff).

cpan task for dist-zilla

dist-zilla has a lot of prereqs. and even when you have installed it, you may need to install other plugins manually. therefore, i just uploaded task::dist::zilla that pulls in all dzil plugins and plugin bundles.

it still takes time to install all those modules, but at least you can do it in one go with cpan or cpanplus: no need to install them one by one manually.

of course, if you're on mandriva, running:

urpmi perl-Dist-Zilla

will install the package perl-Dist-Zilla and all its deps while also suggesting to install all the dzil plugins...

retrospective 2009

2009's over, but the year was quite a busy one for me on the perl front:

• i finally took the time to investigate moose, and decided to port some of my code to use it.
• i discovered dist::zilla, and definitely plan to use it in all my dists.
• i discovered some other cool new perl modules (thanks guys).
  
• i uploaded some new dists on cpan, including games::pandemic (which i'm quite proud of, even if i need to continue hacking on it).
• i updated & released some of my existing dists.
  
• i contributed to other projects such as padre, dist-zilla, etc. sending patches is a good way to thank the author for their work.
• it's even easier to contribute with github, which i'm using more and more for my projects (and it's no more slow by now, woohoo!). of course, git is the master piece allowing this easy sharing.
  
• on mandriva's front, i resurrected parrot rpm, and finally shipped rakudo.
  
• 2009 was also the year of the great migration to %perl_convert_version
• ... with lots of new perl modules available as rpm (i am managing 450 of them).
  
• ... and finally i took ownership of perl rpm. (thank you perl5 porters btw for 5.10.1 and the push for next stable version!)

so, busy indeed 2009 was... i just hope that 2010 will be as fruitful on the perl front for me, and for the perl ecosystem by large.

rt.cpan.org updated - thank you bestpractical

so it seems that we got a nice present for christmas from bestpractical: rt.cpan.org has been updated to last version 3.8

thank you, and keep up the good work!

helping perl packagers package perl modules (for real this time)

chromatic posted a long rant (who would have guessed? :-) ) about perl modules shipped by linux distributions. however, he doesn't have all the answers... nor the experience needed for this rant. since i'm a perl author and mandriva packager for perl and lots of perl modules, i think i have more enlightened information about this topic.

first, let me state that using system perl is fine, but i really discourage it for your enterprise application. the perl version will change from time to time, ditto for the perl modules you are relying upon. so if you want to be in control your software foundation for your app (and you should) - compile your perl and your modules yourself.

second thing: i also discourage mixing using perl modules installed by your package system and by running cpan as root. you'll end up with a mix of files in /usr/perl5 that either belong or not to a system package, which sucks. installing packages in a local lib of yours is fine, which is made quite easy with local::lib by now.

if you're comfortable with those rules, then you're welcome to using the system perl and the modules shipped by your distribution. after all, we packagers are going through this work in the hope of being useful to others - that is, you!

but back to chromatic post. if you want to install cpan modules as system packages, there's already a tool that does it for you: it's called cpan2dist, and is part of cpanplus. it works as long as a backend for your distribution exists. there's currently one for debian, mandriva (that i wrote), fedora and gentoo. it's not that difficult to write, and allows you to write:

# cpan2dist --format CPANPLUS::Dist::Mdv --install Foo::Bar

this will automatically download foo::bar, check its dependencies (and build & install them if needed recursively), build the module as a mandriva package and install it. what else exactly do you want/need? (as a cpanplus backend writer, i do have some things that i'd like cpan2dist to have, but none as a regular user). and if you're a packager and want to integrate cpan2dist with your linux distribution build system, cpan2pkg can help you (even if it's currently stalled due to sthg missing in cpanplus).

however, if you want to help perl packagers package your modules for a distribution, here's a list of thing that you should not do. this is a list of real, practical things to do as a module author - not some generic hand-waving towards the perl community out there. this comes from my experience as packager for mandriva of more than 400 modules, and makes me curse the module author everytime i'm encountering one of those problems...

• test your dist before shipping. really, i'm not kidding. lots of dists just fail their tests. and not just on linux, on all the platforms. so if you make an update that "just can't fail" (yeah, right) to your dist just before shipping, please run your test suite nevertheless. just in case, you know, it might fail.

• if you're shipping pod tests that are skipped depending on the presence of test::pod and test::pod::coverage, make sure you have those modules installed, so you are running those tests, too. even better: skip those tests unless RELEASE_TESTING or AUTHOR_TESTING is set. after all, it's nice for you to know you still have some documentation work to do, but i don't care as a packager... and, you know, it's now the standard & recommended way of shipping those tests.

• those 2 items lead me to another easy thing for module authors to do to help us: check the cpantester status of your dist. investigate all the fails that you have. if you see a fail that is your fault, fix it and upload a new version. it helps us because this prevents us from having to report a bug against your dist. i generally wait 3 or 4 days before reporting a bug on a dist that has some failure reports, hoping (what a fool) that the author will notice by herself that sthg is going wrong.
  

• speaking of bug reports, if we take the time to open a report for your dist (very often with a patch attached)... please read it. and act. or at least answer us. either apply the patch, or explain why you don't want to apply it like that... and ship a new version of your dist, with the fix included.

• but of course, before reporting a bug, we should find the bug tracker. so, by using rt.cpan.org, you really help us to have a single unified point of contact. i know that rt is kind of slow, not very intuitive, has some problems and could be cleaned out a bit... but it is here, bestpractical is providing & administering it for us for free, and has this nice feature of having a queue for every perl dist on cpan. if you don't want to use it, there are some more polite ways of saying it... and giving the url of your tracker helps, too. oh, and if i took the pain to play by your rules and report a bug to your non-standard bug tracker, i would greatly appreciate that you act on my ticket. or at least, you know, just acknowledge the fact that you received the report.

• if you want to really piss off a packager, a simple but effective way is to change your versioning scheme every now and then, by (ab-)using your knowledge of perl way of understanding versions. in the same major version, of course. going from version 1.470 to 1.50 is not funny. if you want to change your versioning scheme, you can change the major number to. after all, i'm pretty sure that you're not paying any extra money per major number used in your dist. this is what caused us to mangle the version of perl modules shipped in mandriva.
  

• speaking of regular changes, it's irritating to have to follow you through your use of makefile.pl to build.pl to makefile.pl to build.pl to... well, you understand what i mean. even to use this shiny replacement that is module::build, or this oh-so-marvelous module::install, oh no finally module::build way of working fits me better in retrospective... it's ok for you to change from time to time, but changing at every version of your dist - just make up your mind dammit!

• speaking of it, i hate module::install. and especially its feature that prompts and tries to handle the deps itself coz-it's-so-cool-it-can-do-it-for-real. sorry, but that's not your job in the tool chain. just report that you miss some deps. i know that there's a flag to make this feature go away while launching makefile.pl. but i don't want to bother and would rather expect that the whole stuff has sane defaults...

• oh, and in case you're wondering - every prompting in the configure phase (makefile.pl or build.pl) sucks and should be banned.

• having clear and up-to-date dependencies would be fine, too. i know it's not always easy to have them correctly, but you can change your tools and adopt one that extract your prereqs for you.

• try to avoid dependency on modules that are known to fail. even if it works in your setup, trust cpantesters if they tell you that it fails 95% of its reports: it might not be a good idea to depend on it.

• trying to support old perl versions and old releases of modules is fine, but update your modules and see if your code work. some functions may become deprecated, or you were relying on a buggy behaviour, or whatever. we update the perl packages as we see new versions, not only your modules. so a linux dist will usually have latest & greatest version of all the modules - you'd better be sure that your code work with them, hmm?
  

• finally, if you're developping under macosx, make sure that you don't ship resource files, or textmate temp files. having ._Foo.pm in the dist is not fun: automatic compile tests will fail, unless that's your manicheck or signature check. and even if everything in perl dist is fine, things may bork in the repackaging of the system package due to a file not listed. so, be extra careful when shipping your dist - or change platform and burn your shiny toy that calls itself a computer (careful & clever readers may have guessed from previous sentence that i don't like macosx - but that's not a reason to ditch this post and not to follow the advices i'm reporting).
  

there, i think that's a pretty good start. i've encountered each and every item of this list at least once (i stopped counting exactly how much a long time ago). don't take it personally if you made some of those mistakes - i have done almost all of them by myself as module author (except of course using module::install and using a mac, but you could have guessed for at least the last part). what's important is to realize that those behaviours are annoying for packagers, and that changing those habits is quite easy to do (except for the burning your mac part, because i agree that it's not very environmental-friendly - see, i'm not that stubborn! ;-) ).

if you're following all those advices, packagers of your modules will love you. (or at least, not hate you - which is still a win :-) ). i know i will...