cpan2pkg now available in text mode!

cpan2pkg is the tool used to automatically package a module from cpan to mageia as a rpm, along with all its dependencies. it's built on shoulders of giants (cpan2dist, mgarepo, poe, etc)

it used to feature a tk window with one tab per module being investigated - but this was troublesome for people wanting to have a cli. so i just released a new version of cpan2pkg which brings a --text flag (or -T for the lazy ones) which works on the tty. of course, we loose the ability to interact with the application, but that's still convenient sometimes. and of course, the tk version is still available!

cpan2pkg: prettier gui, ready for mageia!

remember cpan2pkg? this is the tool built around cpan2dist to create a native linux package, taking dependencies into account, and integrated with linux distribution repository + buildsystem.

it used to be mandriva only, and curses-based. now that mageia is in full swing mode, i had to port cpan2pkg to support it. and it was a perfect opportunity to clean up the module, and prettify the interface.

here's the result:

on the left, you can see the modules currently being processed, with their status both locally and on the build-system. green means available, and here you can see that moosex::alwayscoerece is being built locally (orange) and is not yet available on mageia (yellow).

note that mageia is fully supported: cpan2pkg will poll the buildsystem status page to follow a module build, and make sure a module is ready before submitting those depending on it.

it is still using poe underneath, but this time with tk. adding support for a platform can be done quite easily - patches welcome!

helping perl packagers package perl modules (for real this time)

chromatic posted a long rant (who would have guessed? :-) ) about perl modules shipped by linux distributions. however, he doesn't have all the answers... nor the experience needed for this rant. since i'm a perl author and mandriva packager for perl and lots of perl modules, i think i have more enlightened information about this topic.

first, let me state that using system perl is fine, but i really discourage it for your enterprise application. the perl version will change from time to time, ditto for the perl modules you are relying upon. so if you want to be in control your software foundation for your app (and you should) - compile your perl and your modules yourself.

second thing: i also discourage mixing using perl modules installed by your package system and by running cpan as root. you'll end up with a mix of files in /usr/perl5 that either belong or not to a system package, which sucks. installing packages in a local lib of yours is fine, which is made quite easy with local::lib by now.

if you're comfortable with those rules, then you're welcome to using the system perl and the modules shipped by your distribution. after all, we packagers are going through this work in the hope of being useful to others - that is, you!

but back to chromatic post. if you want to install cpan modules as system packages, there's already a tool that does it for you: it's called cpan2dist, and is part of cpanplus. it works as long as a backend for your distribution exists. there's currently one for debian, mandriva (that i wrote), fedora and gentoo. it's not that difficult to write, and allows you to write:

# cpan2dist --format CPANPLUS::Dist::Mdv --install Foo::Bar

this will automatically download foo::bar, check its dependencies (and build & install them if needed recursively), build the module as a mandriva package and install it. what else exactly do you want/need? (as a cpanplus backend writer, i do have some things that i'd like cpan2dist to have, but none as a regular user). and if you're a packager and want to integrate cpan2dist with your linux distribution build system, cpan2pkg can help you (even if it's currently stalled due to sthg missing in cpanplus).

however, if you want to help perl packagers package your modules for a distribution, here's a list of thing that you should not do. this is a list of real, practical things to do as a module author - not some generic hand-waving towards the perl community out there. this comes from my experience as packager for mandriva of more than 400 modules, and makes me curse the module author everytime i'm encountering one of those problems...

• test your dist before shipping. really, i'm not kidding. lots of dists just fail their tests. and not just on linux, on all the platforms. so if you make an update that "just can't fail" (yeah, right) to your dist just before shipping, please run your test suite nevertheless. just in case, you know, it might fail.

• if you're shipping pod tests that are skipped depending on the presence of test::pod and test::pod::coverage, make sure you have those modules installed, so you are running those tests, too. even better: skip those tests unless RELEASE_TESTING or AUTHOR_TESTING is set. after all, it's nice for you to know you still have some documentation work to do, but i don't care as a packager... and, you know, it's now the standard & recommended way of shipping those tests.

• those 2 items lead me to another easy thing for module authors to do to help us: check the cpantester status of your dist. investigate all the fails that you have. if you see a fail that is your fault, fix it and upload a new version. it helps us because this prevents us from having to report a bug against your dist. i generally wait 3 or 4 days before reporting a bug on a dist that has some failure reports, hoping (what a fool) that the author will notice by herself that sthg is going wrong.
  

• speaking of bug reports, if we take the time to open a report for your dist (very often with a patch attached)... please read it. and act. or at least answer us. either apply the patch, or explain why you don't want to apply it like that... and ship a new version of your dist, with the fix included.

• but of course, before reporting a bug, we should find the bug tracker. so, by using rt.cpan.org, you really help us to have a single unified point of contact. i know that rt is kind of slow, not very intuitive, has some problems and could be cleaned out a bit... but it is here, bestpractical is providing & administering it for us for free, and has this nice feature of having a queue for every perl dist on cpan. if you don't want to use it, there are some more polite ways of saying it... and giving the url of your tracker helps, too. oh, and if i took the pain to play by your rules and report a bug to your non-standard bug tracker, i would greatly appreciate that you act on my ticket. or at least, you know, just acknowledge the fact that you received the report.

• if you want to really piss off a packager, a simple but effective way is to change your versioning scheme every now and then, by (ab-)using your knowledge of perl way of understanding versions. in the same major version, of course. going from version 1.470 to 1.50 is not funny. if you want to change your versioning scheme, you can change the major number to. after all, i'm pretty sure that you're not paying any extra money per major number used in your dist. this is what caused us to mangle the version of perl modules shipped in mandriva.
  

• speaking of regular changes, it's irritating to have to follow you through your use of makefile.pl to build.pl to makefile.pl to build.pl to... well, you understand what i mean. even to use this shiny replacement that is module::build, or this oh-so-marvelous module::install, oh no finally module::build way of working fits me better in retrospective... it's ok for you to change from time to time, but changing at every version of your dist - just make up your mind dammit!

• speaking of it, i hate module::install. and especially its feature that prompts and tries to handle the deps itself coz-it's-so-cool-it-can-do-it-for-real. sorry, but that's not your job in the tool chain. just report that you miss some deps. i know that there's a flag to make this feature go away while launching makefile.pl. but i don't want to bother and would rather expect that the whole stuff has sane defaults...

• oh, and in case you're wondering - every prompting in the configure phase (makefile.pl or build.pl) sucks and should be banned.

• having clear and up-to-date dependencies would be fine, too. i know it's not always easy to have them correctly, but you can change your tools and adopt one that extract your prereqs for you.

• try to avoid dependency on modules that are known to fail. even if it works in your setup, trust cpantesters if they tell you that it fails 95% of its reports: it might not be a good idea to depend on it.

• trying to support old perl versions and old releases of modules is fine, but update your modules and see if your code work. some functions may become deprecated, or you were relying on a buggy behaviour, or whatever. we update the perl packages as we see new versions, not only your modules. so a linux dist will usually have latest & greatest version of all the modules - you'd better be sure that your code work with them, hmm?
  

• finally, if you're developping under macosx, make sure that you don't ship resource files, or textmate temp files. having ._Foo.pm in the dist is not fun: automatic compile tests will fail, unless that's your manicheck or signature check. and even if everything in perl dist is fine, things may bork in the repackaging of the system package due to a file not listed. so, be extra careful when shipping your dist - or change platform and burn your shiny toy that calls itself a computer (careful & clever readers may have guessed from previous sentence that i don't like macosx - but that's not a reason to ditch this post and not to follow the advices i'm reporting).
  

there, i think that's a pretty good start. i've encountered each and every item of this list at least once (i stopped counting exactly how much a long time ago). don't take it personally if you made some of those mistakes - i have done almost all of them by myself as module author (except of course using module::install and using a mac, but you could have guessed for at least the last part). what's important is to realize that those behaviours are annoying for packagers, and that changing those habits is quite easy to do (except for the burning your mac part, because i agree that it's not very environmental-friendly - see, i'm not that stubborn! ;-) ).

if you're following all those advices, packagers of your modules will love you. (or at least, not hate you - which is still a win :-) ). i know i will...

cpan2dist oddity

cpanplus is great - really, i mean it. i don't like its internals, and having to adapt to it for cpanplus::dist::mdv is not my fondest memory. however, it gets the job done, and that's what counts in the end.

but cpanplus has some oddities in the user-space also. cpan2dist, which is a tool allowing you to create a native package for your platform, sometimes makes me cry.

it insists on building prereqs for the module you want. that's fine, except for one thing: it will also build packages for modules that are not up to date... and this is where the fun starts.

you can think that --nobuildprereqs would prevent this, but nope, it's the default and really means: "don't rebuild prereqs if we're at latest version, but rebuild anyway if we're not at latest version". the --buildprereqs indeed means "i really really want to loose my time, please rebuild a package for all the prereqs even if i already have them installed and working". like this option is of any use... erm.

so in app::cpan2pkg, in order not to build the prereqs, i ended up using --ignore flags (flags as in plural form, since using a negative look-ahead regex such as /(?<!$name)$/ do not work). it took me quite some time to come up with this solution, since --ban was not working at that time.

so life was good, i've used cpan2pkg quite some time to create rpms for mandriva, and updated it till v1.0.0. however, since some time, i can see strange things in the packages submitted on upstream buildsystem: the build prereqs were not set. and i had to update the spec file manually, adding all those missing prereqs. i got tired of that, and investigated what happened.

after some experiments, it appears that cpan2dist seems to have somehow changed the way it handles its --ignore flags. ignored modules now are removed from the prereqs completely, and thus the cpanplus::dist backend do not even see them any more. and thus, buildrequires end up empty. of course, --nobuildprereqs still retained its old silly behaviour...

fortunately, it seems that --ban now works correctly (at least on 2 tests that i made). so i've just updated cpan2pkg, and version 1.1.0 should be available soon on cpan, this time really adding the needed buildrequires.

till the next cpan2dist change, of course... :-|

note: i don't want to sound too harsh to jos in this post... i totally understand that cpanplus is still 0.x software - moreover, this is a bug fix in my mind, since --ban was not working.

cpan2pkg reaches maturity

i finally took some time to work on app::cpan2pkg - and i'm happy to report that it now chains upstream builds, which means that all basic features are in.

therefore, i just released app::cpan2pkg 1.0.0 on cpan. detailed changelog:

• updating prereqs + moving on when module installed from upstream
• splitted module in poe session + worker
• moved info from app:cpan2pkg to module attributes
• don't submit on upstream build-system if missing prereqs on it
• cpan2pkg documentation written

yes, cpan2pkg can build, install, import and automatically build packages from cpan for linux, while following dependencies. does it work? sure, i already used it successfully.

note however that there are some caveats...

the first one of course is that it only supports mandriva currently. this is not a fatality, i expect other distributions (and why not other platforms such as *bsd or solaris) to be handled later on if they want. after all, i'm using cpan2dist underneath to generate the packages, and there exist some backends besides cpanplus::dist::mdv. if you're interested, here's the git repository - and i'm ready to give push rights. heck, if you insist i might even agree to move to github which seems to have all the hype those days.

the second one is that the interface is rather ugly and has a lot of rough edges. i'm waiting for dams to continue his work on curses::toolkit, so i can drop curses::ui::poe. i might also provide a tk or wx version at some point. in the meantime, you have to know the application a bit to understand what's going on. i intend to polish it later on.

finally, the code can be cleaned - but you don't care as a user.

so, that's it. cpan2pkg is live... enjoy!

dependencies done right

dave has a point regarding cpan module dependencies: it can be troublesome to install everything correctly. however, we can get the best of both worlds... if we create packages from cpan modules for linux distributions. indeed, we can then use all the tools that the distributions have worked on and polished since so long a time. not counting the fact that's definitely the goal of a linux distribution to simplify things, and integrate software ready to be used.

to do so, we're standing once again on the shoulders of giants: cpanplus comes with various backends. of course, the most used is the one that really installs the modules. but with cpanplus::dist, it's easy to create modules for your distribution: debian, mandriva or fedora backends exist. and thus, creating a package for your distribution is as easy as (for mandriva in this case):

$ cpan2dist --format CPANPLUS::Dist::Mdv Foo::Bar
$ sudo rpm -Uvh ~/rpm/RPMS/noarch/perl-Foo-Bar.rpm

of course, this is only half of the story. after all, we don't want the users to do this - we're indeed trying to save them the hassle of building the modules.

so, we need to create the rpms (or debs) and provide them in the distribution. which is a bit more difficult, since we need to create the packages for the wanted module, and all the prereqs. and for each of them, we need to import them in the distribution build system. all that can take quite some time.

which is why i'm currently working on app::cpan2pkg (readers of this blog have already heard about that), which does everything automatically - well, that's the goal, at least. if you're a contributor, just run:

$ cpan2pkg Foo::Bar

it will build automatically the packages for Foo::Bar and all its dependencies, import them in the distribution repository, and send them (in order) on the distribution build system, where they will be made available for users later on...

and the end result is achieved, as an end user:

$ sudo urpmi perl-Foo-Bar

that will install Foo::Bar along with all its dependencies. no build fuss, no dependency done wrong, no need to have this gcc or that lib installed. everything will be fetched & installed for you...

so, we still retain the testing culture because packagers continue the regular build with all the test suite. users have their modules ready to be installed as easy as ever. and maybe we'll stop hearing about people complaining about long dependency chains. because dave is right: who gives a shit about that?

note to self: now just go back to work on it so i can release v1.0.0!

app::cpan2pkg 0.4.1 released

now that padre is working again on mandriva, i've been able to make some progress on cpan2pkg...

as explained in a previous post, version 0.4.0 of app::cpan2pkg was providing the basis to completely generate and install local packages.

so, the changelog for 0.4.1 is pretty straightforward:

• storing when a module is complete
• build module when all prereqs have been built & installed
• better prereqs extraction

i had the, ahem, pleasure (for a given definition of pleasure) to discover that cpan2dist is pretty dumb in its arguments parsing.

option --nobuildprereqs does not work as one might think - it really means: "don't rebuild prereqs if they are at latest version, but rebuild anyway if we're not at latest version". yeah, right, i'm stuffing --nobuildprereqs but please rebuild them anyhow if you want...

so i cheated and wanted to use the --ignore option of cpan2dist, but somehow, using a regex with look-behind assertion such as /(?<!$name)$/ does not seem to work - while it is working as intended when testing it on various strings. sigh.

anyway, i've managed to go through this. and this means that the local part of cpan2pkg is complete!

so, now i can work on upstream package importing. and maybe on some mechanism to prevent 50+ processes to be launched at the same time, for modules with a lot of prereqs... but that's, as they say, refinment! :-)