Improving cooperation in volunteer projects
Martin F. Krafft <mail@martin-krafft.net>
- Ph.D. student, University of Limerick, Ireland
- Developer with the Debian, Zope, and Plone projects
FrOSCon 2006, Sankt Augustin, Germany
24 June 2006
Background
- My Ph.D. topic
- Origin in the Debian project, main point of focus
- Important implications in case of success
- open source projects
- software development
- corporations
- ...
- So far, mainly ideas and "vapourware"
Overview
- The need for change
- Thoughts on possible improvements
- Tools that have made a difference
- Tools that have failed
- Characteristics of successful tools
- Discussion
Volunteers
In gardening and botanical terminology, a volunteer is a plant that grows on
its own, rather than being deliberately planted by a human farmer or
gardener (Wikipedia).
- Seems to square well with volunteers in open-source projects:
- developers arrive at working for a project because they want to.
- bringing in people to do specific things (for money) rarely works
(and can cause serious friction if it does work).
Volunteers
Less botanical:
One who enters into, or offers for, any service of his own free will
(Webster's (via dict.die.net)).
Any service?
Volunteers
I need a volunteer, please...
Motivation
Sorry; moving on...
One who enters into, or offers for, any service of his own free will
(Webster's (via dict.die.net)).
Other works on motivation (e.g. The Cathedral and the Bazaar), we'll
concentrate on one main aspect:
- Many volunteers in open-source projects help because they want to (for
whatever reason)
- People (and geeks especially) like to do things their own way
- Put differently: you cannot tell a volunteer what or how to do things
(or you're not dealing with a voluneer anymore)
Giving direction
If "volunteers do just what they want", how can projects follow a direction?
- common vision (e.g. Debian Social Contract)
- accepted leader(s), governors, decision makers (e.g. Plone, Apache)
- "benevolent dictators" (e.g. Linux, Python, Perl, Ubuntu, Fedora)
- "chaotically tolerant" (e.g. Gentoo)
- "passive" leadership (e.g. Debian)
(this list is not supposed to be flame bait!)
Size matters
In smaller projects, leaders look up to governors who are decision makers.
As the project grows more diverse,
- governors tend to fade away.
- leaders take a more passive role.
- decision makers are only used to resolve conflicts.
Diverse projects have a strong self-dynamics.
It is impossible to change existing processes by force in volunteer
projects.
Overview
- Thoughts on possible improvements
- Tools that have made a difference
- Tools that have failed
- Characteristics of successful tools
- Discussion
Goal
What am I actually trying to do?
Revolutionise project leadership?
Goal
No.
(you can stop worrying now)
Goal: improve cooperation
... where "improve" primarily means "facilitate" or "streamline".
Why? What problem am I trying to solve?
Stagnation and inflexibility due to high inertia
Stagnation and inflexibility in Debian
- The Debian project turns 14 this year
- Processes are in flux, but fundamentals have not changed much
- Much time is lost
Claim: Debian could be further and more agile
(I don't want to paint a black picture though)
I have two hours to work on a Debian package, I don't want to spend most of
that on learning how a certain package is managed!
Overview
- The breed of volunteers
- The need for change
- Thoughts on possible improvements
- Tools that have made a difference
- Tools that have failed
- Characteristics of successful tools
- Discussion
Improving Cooperation
How can cooperation be facilitated, streamlined?
Some ideas:
- Technical: version control (VCS), issue tracking, standardisation
- Social: tolerance/variety, integration, convincing of technical merits
- Organisational: workflow, best practices, policy
Example: package maintenance
- Maintaining packages has not changed fundamentally in years
- Various bottlenecks call for co-maintenance
- Complexity calls for smart(er) patch management
- Different approaches and their learning curves, confusion
- Manual bug triage
The principles should be:
Don't do manually what you can automate!
Keep it simple, stupid!
Top-down vs. bottom-up
- Top-down (in this context) means:
- we are at the current state
- we need to change the processes to get to the desired state
- Bottom-up (in this context) means:
- the foundation was geared for the desired state from the start
- the processes emerged from the foundation
Comparisons: Plone and Debian
Differentiate between projects, take Plone and Debian as examples:
| Criterium |
Plone |
Debian |
|---|
| Age |
relatively young |
relatively aged |
| Jumpstart |
sprang from Zope |
started from scratch (pioneer) |
| Size |
small number of contributors |
large number of contributors |
| VCS usage |
used one VCS from the start |
tried to introduce VCS later,
no standard |
| Bug tracking |
in flux, standardised now |
early introduction of debbugs |
| Workflow |
simple(r), one product |
Various, many constituents |
Main points of focus
Debian package maintenance:
- Version control
- Bug tracking
- Quality assurance with distributed contributors
- Inherent/emergent workflow
Improving cooperation in Debian
Some ideas:
- Standardise the use of VCS, but not the type of VCS (if possible)
- Switch users to VCS by making VCS more intuitive, not by expecting users to
learn
- Integrate with other aspects of package maintenance, for example:
- debian/changelog
- Bug tracking
- Upload coordination
The problem with VCS
- VCS are a technical solution to a social problem
- generally more geared towards technical soundness than usability
- Thus: steep learning curve of concepts
- So far no integration with other development processes (in Debian)
How to get people to adopt?
We cannot just revolutionise the process.
Remember: volunteers don't like to be told how to do things
Two strategies for adoption:
- seamless integration with existing processes
- technical merits of alternatives to attract users
Seamless integration
First phase of my research is the assessment of existing processes
One approach involves "live command-line sessions" on IRC:
<oracle> i want to work on the mutt package
<martin> downloading mutt (1.5.9-2) to ./mutt-1.5.9 ... done.
<oracle> i want to fix bug #654321
<martin> preparing isolated environment for fixing of bug #654321 ... done.
< martin records intent ...
< oracle edits a file, fixes the bug
<oracle> i have fixed the bug
<martin> thank you. please give a brief description of the bug fix.
< martin spawns editor-of-choice
< oracle describes the bug fix and closes editor
<martin> fix recorded in change log, what else can i do for you?
<oracle> please prepare a new release
< [...]
Other approaches include: mockup interfaces, investigation and exchange with
other similarly complex projects, such as the BSDs
What to integrate?
- Generally, this question should be answered only after extensive research
of existing methods.
- Some ideas already include the integration of VCS with:
- debian/changelog
- Bug tracking
- Upload coordination
Integration of VCS with debian/changelog
- VCS changesets have log messages.
- debian/changelog entries should ideally correspond to one or more
changesets each.
- Figure out a good, unobtrusive way to reuse log entries
Idea:
- Use of branches to group related changes between versions
- Merging a branch into the main tree requires a commit and thus a log entry
- Those log entries (combined with the log entry corresponding to the branch
creation) could make up
Integration of VCS with debbugs
- At the moment only through debian/changelog
- Everything else is manual, mainly associating patches with bugs and keeping
track of bugs and versions that fixed them.
Idea:
- One branch per bug, automatic branching (?)
- Automatic submission of patches to BTS (if still necessary)
The word "automatic" does cause chills among many. Things should be automated
the Debian way, as in: facilitated but requiring manual action.
Integration of VCS with the upload process
Problem: maintenance is distributed, but we still need one developer for the
upload. This can lead to a myriad of problems, ranging from bottlenecks to
quality control.
Idea:
- Upload requests instead of direct upload
- Archive server fetches source, building on all architectures
- Per-package policy requiring quality control/authenticity certificates
Overview
- The breed of volunteers
- The need for change
- Thoughts on possible improvements
- Tools that have made a difference
- Tools that have failed
- Characteristics of successful tools
Technical merits
What are technical merits that will lead users to adopt?
Successful introduction: debian-installer
- Project started around 2001
- Consistent use of VCS from the beginning, first CVS, soon SVN
- Most contributors didn't want to learn VCS, project decided to keep it very
simple
- First project to make consistent use of a wiki in Debian
Successful introduction: debhelper
- Consistent set of scripts to fill an ecological niche. Compare this
debian/rules file to this one using debhelper.
- Logical modularity: one command per packaging aspect
- Consistent yet various specification of parameters
- Early introduction (1997)
- Bottom-up (building block) vs. top-down (c.f. cdbs (which uses
debhelper); see this cdbs rules file).
Unsuccessful introduction: pkg-zope
- Around 100 packages and 30 maintainers (maybe 5 active)
- Infected by the hype around distributed VCS, I tried to introduce GNU arch
- Despite efforts and documentation, no adoption
- Instance of "useful but not usable", developer-centric approach
This is only the beginning...
Thank you for your attention!