Woes of Linux packaging and installers

I commented under a very good article:


1) You wrote:
In creating nearly all packages, there is always some amount of shell scripting that needs to be done.

Yes, this is historically what is being done. However the more I think about it, the more problematic I see it. Why are there during installation things that has to be done by shell scripting??? What are these things?

Installing a package means to copy its files in the right places (binaries, docs, images, launchers, etc.), create/update config files, restart services. All these things could be done by the installer, and the pre-inst and post-inst scripts IMO hide the deficiencies of the package manager and/or package format.

For Java there is the "ant" tool, and this way for compilation the tested, proven and versatile (but old) makefiles are not needed. And the ant's build.xml file is, well, XML. Even more, ant is extendable by providing it with new "actions" so it it not limited to a single set of XML tags. This lets me have a confidence it would be doable to use XML for package management, so please don't spread misinformation and prejudices here.

Even more:
Many packages require some form of post-install and pre-uninstall scripting to be done. The most universal method for doing this on any distribution is a script.

Yes, you're right. However, if the package managers would run such a script providing it with the settings like paths to config files, description of package dependencies, and other things that differ between distributions today - this would be an even more powerful mechanism. And at this point I would agree that there is a place for shell scripting (which I like and use quite often in my everyday work).

2) You mentioned a myth:
The user should be able to get a package from just about anywhere, and it should run automatically on their Linux system.

I fail to consider this a myth. Either we want uniformization of Linux environment or we don't. Either it is a good thing to expand Linux penetration of the market or it's bad. If these goals are to be met, this is one of the means to achieve these goals.

Why do distributors and distributions exist? Because people have different views on how to do things. Different gcc and toolchain versions, different compiler settings, different locations of config files (think: apache2), different package formats. If there was some common ground in a form of installer API, no need for pre-inst and post-inst scripts and to some extent uniformity of the OS environment, all the above differences would become less important.

Maybe it's time to really think about more cross-distributions standardization?

3) However:
Trying to funnel everything down through some common layer at the bottom really reduces the chances for big innovation to happen.

And here I agree with you 100%. So in fact I'm not completely sure what to do w.r.t. software packaging in Linux. But uniform approach to this problem and uniform solution would surely bring commercial world to Linux. So they would be able to package their software once and it would run in almost every distro. One thing remains to be answered, whether this is really a goal that we would like to reach?


Microsoft works against mankind?

Analysis of why an open source Windows is inevitable contains such quote:

Well, let's just say that, while Microsoft's success was due in large part for being at the right place at the right time, it now finds itself in the way of progress and the betterment of mankind.

While I find it humorous, there is some sad truth in this...



UNA - collaborative free source code editor

Editor that presents unique approach to collaborative work. Supports many programming languages, integrates with Exuberant CTags, offers superior configurability and has a highly usable user interface.

Interview: John De Goes Introduces a Newly Free Source Code Editor


Linux package differences

I put a long comment under the article Ubuntu 7.10 to PCLinuxOS 2008. Here's a copy of it:

It's not so easy to create a unified, centralized repository for all Linux distros as you might think.

The distros exist because all these people have different opinions on how a Linux system should work. I'd say this is a real-time experiment, a Darwinian effort to see which approach works best. This is the price of innovation. At the other end is stagnation, and I don't think any of Linux users would like it to stagnate.

Reasons for why it's hard to create universal packages:
1) Improvements/differences in GCC compiler and so called "toolchain".
2) Different init systems and services management.
3) Different config files locations and syntax.
4) Different package formats and package management systems.


1) All binary programs that constitute a working system have to be compiled (translated from human-readable form into a form suitable for CPU to execute). In case of Linux the kernel, basic GNU tools and other useful programs (be it X Window System, GNOME/KDE/XFCE/etc, Firefox, K3B, and so on), this is done by a suite of GCC compilers and a set of tools like: automake, linkers, loaders, binutils and so on.

These tools are not set in stone, they are improved, changed, restructured over time, not to mention the occasional bugfixing. Sometimes these changes are INCOMPATIBLE, like changing the GCC from version 2.95 to 3.x line and then to 4.x series. The same is for accompanying tools. They all change as their requirements change (according to the user's demand).

So when a particular distro or a package developer decide to use a specific set of tools and libraries, with their specific versions and behaviour, these decisions can be a source of incompatibility across distros and package maintainers. So for example, an mplayer package from distro A may *NOT* work at all in distro B, because of so called ABI differences (Application Binary Interface). The compatibility is maintained at the source level instead. So it's the distro maintainers that worry about this, so they just recompile a problematic package and put a new version for use.

2) Even when the package is compatible at the binary level (which is true for most packages and distros), distro creators may have different views on how their systems work internally and how they handle system services. I mean the locations of service scripts (/etc/rc.d versus /etc/rc0.d versus /etc/init.d), their configuration files' locations (/etc/default versus /etc/conf.d) and structure, where they place working directories of services, and so on.

This may not apply to the end-user applications like web browser, multimedia player or CD/DVD burning application. But for things like cups (printing service) or apache (web server) it is a serious concern. Each distro may offer a unique approach here, because of people disagreeing on the details. So it may happen that a specific package from distro A would not work in distro B or may even break its installation. That's why users are advised *NOT* to install foreign packages on their own without thinking about consequences.

Compiling a package manually under a given distro typically yields in a working package that's tailored to this distro (and honestly it's not so hard as one can imagine). But since then it is the user that's responsible for maintaining such package and upgrading it when some bugs are found.

3) Sometimes even a basic structure of config files may be radically different. From my experience the network configuration varies wildly between distros (e.g. Debian, Mandriva, RedHat and Gentoo have it completely different). Apache configuration files are also well known to be differently separated and placed in different directories and under different names in distros. Just because people have different opinions and vision on how it should be done.

[Note: this section has been added here] Also, KDE installations may vary between distros: there can be different directory names, different number of config files, application launchers and so on. These are the parameters that can be set at the compilation time. So when people pick different values here, incompatibilities arise. For instance, Debian/(K)Ubuntu and PCLinuxOS have it completely different.

4) Package formats of DEB, RPM, TGZ, .recipe, .ebuild are different enough to be incompatible. In some packages there are little programs (maintainer scripts) that are run at the beginning of the installation or after a package is installed.

Each package format has a different way of describing dependencies (which other packages need to be installed first before a given package would run correctly). Even RPM or DEB packages from different distros may describe dependencies incompatibly. Even packages from different releases of the same distro may be incompatible!

Package formats also evolve, for example recently Ubuntu and Debian introduced the concept of package triggers, independently of pre-inst or post-inst scripts. Yes, this mechanism is backward compatible, but you get the idea.

Concluding, there are incentives to minimize the differences, like the noble FHS (Linux Filesystem Hierarchy Standard) or LSB (Linux Standard Base). So the problems are worked on, but for now it's really hard to create a package that would run unchanged in a number of distros. I don't say this would not change in the future, but I hope you now understand the problems here.