cpan2pkg: prettier gui, ready for mageia!

remember cpan2pkg? this is the tool built around cpan2dist to create a native linux package, taking dependencies into account, and integrated with linux distribution repository + buildsystem.

it used to be mandriva only, and curses-based. now that mageia is in full swing mode, i had to port cpan2pkg to support it. and it was a perfect opportunity to clean up the module, and prettify the interface.

here's the result:

on the left, you can see the modules currently being processed, with their status both locally and on the build-system. green means available, and here you can see that moosex::alwayscoerece is being built locally (orange) and is not yet available on mageia (yellow).

note that mageia is fully supported: cpan2pkg will poll the buildsystem status page to follow a module build, and make sure a module is ready before submitting those depending on it.

it is still using poe underneath, but this time with tk. adding support for a platform can be done quite easily - patches welcome!

more tightened spec file produced by mageia cpanplus backend

cpanplus::dist::mageia 1.111720 has just been released, with 2 changes:

• no more %defattr definition
• no more default %clean section

indeed, rpm will handle those automatically, as dexter mentioned (cf here and here). this way, we'll have spec files a bit more tightened, which is always better from a maintainer pov.

magpie: how to list non up-to-date perl modules

the final stone is now built for magpie: it's possible to list on your mageia system the perl modules that have up to date versions upstream. it allows to see at a glance which perl packages need to be updated. it's based on cpanplus, and only tackles perl modules installed on your system.

the new command is:

$ magpie old
** core packages: 11

Devel::DProf                             20080331.00 20110228.00
Devel::SelfStubber                              1.03        1.05
[...]
if                                              0.05      0.0601

** normal packages: 57

DateTime::TimeZone                              1.28        1.29     perl-DateTime-TimeZone                                 1.280.0
Digest::SHA                                     5.50        5.60     perl-Digest-SHA                                        5.500.0
[...]
parent                                         0.224       0.225     perl-parent                                            0.224.0

** orphan packages: 7

KENTNL                                      0.010173Dist::Zilla…
[...]
inc::latest                                   0.3624        0.38

** strange packages: 3

App::cpanminus                                1.2001      1.4002     cpanminus(1.200.100),perl-App-cpanminus(1.200.100)
HTTP::Response                                  6.00        6.01     perl-libwww-perl(5.837.0),perl-Test-Mock-LWP(0.50.0),perl-HTTP-Message(6.0.0)
Perl::MinimumVersion                            1.27        1.28     perl-Perl-MinimumVersion(1.270.0),perl-Perl-Critic-Pulp(47.0.0)

** ignored modules: 12

Boulder::Unigene(28051999.00), File::MimeInfo::Rox(0.20), Getopt::Lucid(0.19), HTML::Table(2.08), Imager::Font::Type1(1.011), Inline::Python(0.38), Lingua::Features(0.30), Parse::RecDescent::FAQ::Original(6.00), Switch(2.16), Tie::Watch(1.301), WebFetch(0.13), XML::Grove(0.46)

the columns are: module, old version, new version, mageia package name, mageia package version.

as you can see, the modules are sorted in various categories:

• core - modules shipped with perl (or perl-base)
• dual-lifed (not shown above) - modules that are both shipped with perl (or perl-base) and as a stand-alone package
• normal - regular modules with their own package
• orphan - modules that do not belong to a mageia package (either inherited from mandriva, or not yet pushed on buildsystem).
• strange - modules that belong to more than one magiea package
• ignore - those are modules that either cannot be built (with an upstream bug) or that confuse cpanplus when comparing versions

with this information, it's therefore very straightforward to update the packages, launching the following command as needed:

$ eval $( magpie co -s $pkg ) && magpie update

of course, since all the building blocks are now available, that's what the last command to be implemented (magpie dwim) will do!

how to obsolete a cpan dist?

my autoprereq plugin for dist-zilla is now going core. however, it's currently existing as a dist of its own within cpan... since the module does not change name, what is needed to make sure rjbs can upload dzil v2 when it's ready? is giving co-maint rights on pause enough? will i have to remove previous tarballs?

the same kind of question can be asked for obsoleting a given module. there's (afaik) no way to tell cpan / cpanplus that foo::bar is replacing bar::foo... those problems are tackled for linux distribution packages, could we reuse some of their logic here?

helping perl packagers package perl modules (for real this time)

chromatic posted a long rant (who would have guessed? :-) ) about perl modules shipped by linux distributions. however, he doesn't have all the answers... nor the experience needed for this rant. since i'm a perl author and mandriva packager for perl and lots of perl modules, i think i have more enlightened information about this topic.

first, let me state that using system perl is fine, but i really discourage it for your enterprise application. the perl version will change from time to time, ditto for the perl modules you are relying upon. so if you want to be in control your software foundation for your app (and you should) - compile your perl and your modules yourself.

second thing: i also discourage mixing using perl modules installed by your package system and by running cpan as root. you'll end up with a mix of files in /usr/perl5 that either belong or not to a system package, which sucks. installing packages in a local lib of yours is fine, which is made quite easy with local::lib by now.

if you're comfortable with those rules, then you're welcome to using the system perl and the modules shipped by your distribution. after all, we packagers are going through this work in the hope of being useful to others - that is, you!

but back to chromatic post. if you want to install cpan modules as system packages, there's already a tool that does it for you: it's called cpan2dist, and is part of cpanplus. it works as long as a backend for your distribution exists. there's currently one for debian, mandriva (that i wrote), fedora and gentoo. it's not that difficult to write, and allows you to write:

# cpan2dist --format CPANPLUS::Dist::Mdv --install Foo::Bar

this will automatically download foo::bar, check its dependencies (and build & install them if needed recursively), build the module as a mandriva package and install it. what else exactly do you want/need? (as a cpanplus backend writer, i do have some things that i'd like cpan2dist to have, but none as a regular user). and if you're a packager and want to integrate cpan2dist with your linux distribution build system, cpan2pkg can help you (even if it's currently stalled due to sthg missing in cpanplus).

however, if you want to help perl packagers package your modules for a distribution, here's a list of thing that you should not do. this is a list of real, practical things to do as a module author - not some generic hand-waving towards the perl community out there. this comes from my experience as packager for mandriva of more than 400 modules, and makes me curse the module author everytime i'm encountering one of those problems...

• test your dist before shipping. really, i'm not kidding. lots of dists just fail their tests. and not just on linux, on all the platforms. so if you make an update that "just can't fail" (yeah, right) to your dist just before shipping, please run your test suite nevertheless. just in case, you know, it might fail.

• if you're shipping pod tests that are skipped depending on the presence of test::pod and test::pod::coverage, make sure you have those modules installed, so you are running those tests, too. even better: skip those tests unless RELEASE_TESTING or AUTHOR_TESTING is set. after all, it's nice for you to know you still have some documentation work to do, but i don't care as a packager... and, you know, it's now the standard & recommended way of shipping those tests.

• those 2 items lead me to another easy thing for module authors to do to help us: check the cpantester status of your dist. investigate all the fails that you have. if you see a fail that is your fault, fix it and upload a new version. it helps us because this prevents us from having to report a bug against your dist. i generally wait 3 or 4 days before reporting a bug on a dist that has some failure reports, hoping (what a fool) that the author will notice by herself that sthg is going wrong.
  

• speaking of bug reports, if we take the time to open a report for your dist (very often with a patch attached)... please read it. and act. or at least answer us. either apply the patch, or explain why you don't want to apply it like that... and ship a new version of your dist, with the fix included.

• but of course, before reporting a bug, we should find the bug tracker. so, by using rt.cpan.org, you really help us to have a single unified point of contact. i know that rt is kind of slow, not very intuitive, has some problems and could be cleaned out a bit... but it is here, bestpractical is providing & administering it for us for free, and has this nice feature of having a queue for every perl dist on cpan. if you don't want to use it, there are some more polite ways of saying it... and giving the url of your tracker helps, too. oh, and if i took the pain to play by your rules and report a bug to your non-standard bug tracker, i would greatly appreciate that you act on my ticket. or at least, you know, just acknowledge the fact that you received the report.

• if you want to really piss off a packager, a simple but effective way is to change your versioning scheme every now and then, by (ab-)using your knowledge of perl way of understanding versions. in the same major version, of course. going from version 1.470 to 1.50 is not funny. if you want to change your versioning scheme, you can change the major number to. after all, i'm pretty sure that you're not paying any extra money per major number used in your dist. this is what caused us to mangle the version of perl modules shipped in mandriva.
  

• speaking of regular changes, it's irritating to have to follow you through your use of makefile.pl to build.pl to makefile.pl to build.pl to... well, you understand what i mean. even to use this shiny replacement that is module::build, or this oh-so-marvelous module::install, oh no finally module::build way of working fits me better in retrospective... it's ok for you to change from time to time, but changing at every version of your dist - just make up your mind dammit!

• speaking of it, i hate module::install. and especially its feature that prompts and tries to handle the deps itself coz-it's-so-cool-it-can-do-it-for-real. sorry, but that's not your job in the tool chain. just report that you miss some deps. i know that there's a flag to make this feature go away while launching makefile.pl. but i don't want to bother and would rather expect that the whole stuff has sane defaults...

• oh, and in case you're wondering - every prompting in the configure phase (makefile.pl or build.pl) sucks and should be banned.

• having clear and up-to-date dependencies would be fine, too. i know it's not always easy to have them correctly, but you can change your tools and adopt one that extract your prereqs for you.

• try to avoid dependency on modules that are known to fail. even if it works in your setup, trust cpantesters if they tell you that it fails 95% of its reports: it might not be a good idea to depend on it.

• trying to support old perl versions and old releases of modules is fine, but update your modules and see if your code work. some functions may become deprecated, or you were relying on a buggy behaviour, or whatever. we update the perl packages as we see new versions, not only your modules. so a linux dist will usually have latest & greatest version of all the modules - you'd better be sure that your code work with them, hmm?
  

• finally, if you're developping under macosx, make sure that you don't ship resource files, or textmate temp files. having ._Foo.pm in the dist is not fun: automatic compile tests will fail, unless that's your manicheck or signature check. and even if everything in perl dist is fine, things may bork in the repackaging of the system package due to a file not listed. so, be extra careful when shipping your dist - or change platform and burn your shiny toy that calls itself a computer (careful & clever readers may have guessed from previous sentence that i don't like macosx - but that's not a reason to ditch this post and not to follow the advices i'm reporting).
  

there, i think that's a pretty good start. i've encountered each and every item of this list at least once (i stopped counting exactly how much a long time ago). don't take it personally if you made some of those mistakes - i have done almost all of them by myself as module author (except of course using module::install and using a mac, but you could have guessed for at least the last part). what's important is to realize that those behaviours are annoying for packagers, and that changing those habits is quite easy to do (except for the burning your mac part, because i agree that it's not very environmental-friendly - see, i'm not that stubborn! ;-) ).

if you're following all those advices, packagers of your modules will love you. (or at least, not hate you - which is still a win :-) ). i know i will...

cpan2dist oddity

cpanplus is great - really, i mean it. i don't like its internals, and having to adapt to it for cpanplus::dist::mdv is not my fondest memory. however, it gets the job done, and that's what counts in the end.

but cpanplus has some oddities in the user-space also. cpan2dist, which is a tool allowing you to create a native package for your platform, sometimes makes me cry.

it insists on building prereqs for the module you want. that's fine, except for one thing: it will also build packages for modules that are not up to date... and this is where the fun starts.

you can think that --nobuildprereqs would prevent this, but nope, it's the default and really means: "don't rebuild prereqs if we're at latest version, but rebuild anyway if we're not at latest version". the --buildprereqs indeed means "i really really want to loose my time, please rebuild a package for all the prereqs even if i already have them installed and working". like this option is of any use... erm.

so in app::cpan2pkg, in order not to build the prereqs, i ended up using --ignore flags (flags as in plural form, since using a negative look-ahead regex such as /(?<!$name)$/ do not work). it took me quite some time to come up with this solution, since --ban was not working at that time.

so life was good, i've used cpan2pkg quite some time to create rpms for mandriva, and updated it till v1.0.0. however, since some time, i can see strange things in the packages submitted on upstream buildsystem: the build prereqs were not set. and i had to update the spec file manually, adding all those missing prereqs. i got tired of that, and investigated what happened.

after some experiments, it appears that cpan2dist seems to have somehow changed the way it handles its --ignore flags. ignored modules now are removed from the prereqs completely, and thus the cpanplus::dist backend do not even see them any more. and thus, buildrequires end up empty. of course, --nobuildprereqs still retained its old silly behaviour...

fortunately, it seems that --ban now works correctly (at least on 2 tests that i made). so i've just updated cpan2pkg, and version 1.1.0 should be available soon on cpan, this time really adding the needed buildrequires.

till the next cpan2dist change, of course... :-|

note: i don't want to sound too harsh to jos in this post... i totally understand that cpanplus is still 0.x software - moreover, this is a bug fix in my mind, since --ban was not working.

rationalizing perl module versions

versioning sucks. nobody agrees on what version numbers should look like. everybody has some good reasons not to like a given scheme. and i would say that everybody is right - at least from their point of view.

if you throw in the way that perl deals with version (used by cpan and cpanplus), you add more fuzzyness to the whole thing. eg, 1.40 is supposed to be more recent than 1.300 (yes, 1.40 is understood as 1.400). and i won't even talk of digits 3 to 6, or developer releases...

add to that developers who change their versioning scheme every now and then, or the ones who knows the way perl works and use 2 or 3 digits after the dot (yes module::starter, i'm staring at you - but you're not the only one)...

yup, versioning is a nightmare for packagers. this sucks.

so, after some time bugging mandriva admins, we finally have a shiny rpm macro %perl_convert_version that normalizes perl versions to a x.y.z scheme. here is its content for whoever is interested:

perl -Mversion -le '$v=version->new(%{1})->normal; $v=~s/^v//; print $v'

and we're going to use it on all perl modules packaged by mandriva. yes, this means that the rpm version will potentially be different than the upstream version on cpan. but that's the price to pay to have rpm and other non-perl aware tools to work correctly.

for the migration to be as smooth as possible, the following needs to be done:

1. push the rpm macro on mandriva build system. done.
2. fix cpanplus::dist::mdv to use it.
3. update youri's update check to recognize it.
4. update mdvsys to recognize it during mdvsys update.
5. fix our perl packages spec files.
   

in order to move on, i'm therefore happy to announce that cpanplus::dist::mdv 1.1.0 has been released, which (among other changes), now uses the rpm macro %perl_convert_version. that's one ticked off the list. :-)

dependencies done right

dave has a point regarding cpan module dependencies: it can be troublesome to install everything correctly. however, we can get the best of both worlds... if we create packages from cpan modules for linux distributions. indeed, we can then use all the tools that the distributions have worked on and polished since so long a time. not counting the fact that's definitely the goal of a linux distribution to simplify things, and integrate software ready to be used.

to do so, we're standing once again on the shoulders of giants: cpanplus comes with various backends. of course, the most used is the one that really installs the modules. but with cpanplus::dist, it's easy to create modules for your distribution: debian, mandriva or fedora backends exist. and thus, creating a package for your distribution is as easy as (for mandriva in this case):

$ cpan2dist --format CPANPLUS::Dist::Mdv Foo::Bar
$ sudo rpm -Uvh ~/rpm/RPMS/noarch/perl-Foo-Bar.rpm

of course, this is only half of the story. after all, we don't want the users to do this - we're indeed trying to save them the hassle of building the modules.

so, we need to create the rpms (or debs) and provide them in the distribution. which is a bit more difficult, since we need to create the packages for the wanted module, and all the prereqs. and for each of them, we need to import them in the distribution build system. all that can take quite some time.

which is why i'm currently working on app::cpan2pkg (readers of this blog have already heard about that), which does everything automatically - well, that's the goal, at least. if you're a contributor, just run:

$ cpan2pkg Foo::Bar

it will build automatically the packages for Foo::Bar and all its dependencies, import them in the distribution repository, and send them (in order) on the distribution build system, where they will be made available for users later on...

and the end result is achieved, as an end user:

$ sudo urpmi perl-Foo-Bar

that will install Foo::Bar along with all its dependencies. no build fuss, no dependency done wrong, no need to have this gcc or that lib installed. everything will be fetched & installed for you...

so, we still retain the testing culture because packagers continue the regular build with all the test suite. users have their modules ready to be installed as easy as ever. and maybe we'll stop hearing about people complaining about long dependency chains. because dave is right: who gives a shit about that?

note to self: now just go back to work on it so i can release v1.0.0!

app::cpan2pkg 0.4.0 released

app::cpan2pkg 0.4.0 is live. packages are now being built (with cpan2dist from CPANPLUS) and installed - if there's no missing prereqs, of course.

note that not all the dependency updates are treated afterwards, since it needed to have the build-package-and-install-it-locally step completed first (obviously). that would be next on the agenda.

once this is done, everything related to package generation / install will be done. and i'll then work on submitting packages upstream...

cpanplus::dist::mdv 1.0.0 released

cpanplus::dist::mdv, the cpanplus backend to create mandriva rpms, has now reached version 1: much rejoicing and all that...

it doesn't mean that there won't be any new features in the future, but it is now somehow complete - read that it can install generated rpms now.

there's also a minor bug fixed: previously, the -debug package was selected (over the regular one) for binary rpms.

note that this version is already available in cooker, the development mandriva version.

cpan modules & native linux packages

cpan is great - one finds a lot of modules for really everything. the problem comes from installing those modules.

indeed, your system comes with a perl and some cpan modules packaged for your distribution. and mixing them with modules installed directly by cpan(1) or cpanp(1) is the best way to wreck havoc on your system.

enters cpanplus backends, allowing one to easily create native linux packages for you distribution using:

cpan2dist Foo::Bar

pretty straighforward, assuming there's a backend available for your distribution...

so i created cpanplus::dist::mdv, and life was good: i created rpms when i wanted some new modules. it has even become the official way to create rpm packages from cpan modules for mandriva.

unfortunately, it's still not enough, even if jos agreed to integrate a --install flag in cpan2dist to install prereqs after building them.

indeed, you don't want to build a prereq when this one might be already available from your distribution. moreover, once a package is installed, you'd like to submit the new package to your distribution, for other users to benefit from them (assuming you're a contributor for your distribution).

lazyness helping, i always let it go. but lazyness strikes back: always repeating the same steps to check if a prereq is provided by mandriva, building them or installing them, and then submitting them is not that a good way to spend my time.

so one week ago, i've decided to create an app::cpan2pkg module (not yet on cpan), a curses application build around cpan2dist and taking care of every itch mentioned earlier.

it is really a glue application, meant to reuse existing command lines, but taking the decisions automatically and chaining the steps.

note that it's called app::cpan2pkg, and not app::cpan2mdv. i intend to write it so that one can adapt it to other distributions workflows. of course, currently i'm focusing on mandriva, the distribution i'm using. but if others want to plug in, they're more than welcome!

it is currently in version 0.1.0 (except to see it soon on cpan), and is able to extract the module prereqs. i'll let you know about the progress...