Mark Shuttleworth, founder of Canonical and maintainer of Ubuntu Linux, said blending agile development practices with open source community development is a key to successful software projects in the new world.

If you mix the best of agile with the best of what open source development has to offer, you’re looking at a recipe for a successful software development experience.

Mark Shuttleworth, founder of Canonical, essentially said as much in a packed session at the O’Reilly Open Source Convention recently. His words were timely, as the Agile 2008 conference is running the week of Aug. 4 in Toronto.

However, Shuttleworth said from the practices of his team at Canonical, things seem to work best when they incorporate development practices from the open source community—in with the agile methods they use. That way the efforts tend to run even better than with simply agile practices alone, he said.

Shuttleworth spoke about "philosophy, principles, practices—things that work," and said Canonical has taken on "a community inspired methodology." He said he was filling in for Ian Clatworthy, a senior software engineer at Canonical, who was scheduled to be at the event.

"Free software and the process we are learning from communities are changing the professional practice of software development," Shuttleworth said. He noted that James Dixon of Pentaho has come up with a process known as Open Scrum, which resembles the methods Canonical is adopting.

"I think free software is going to have a profound impact on the practice of software development in a community setting," he said, and I agree. "Even Microsoft is taking advantage of the practices."

Yeah, Microsoft and a whole lot of other companies, including IBM, which refers to its practice of pursuing an internal open source development model as its "community source" effort.

"When worlds collide, wonderful things happen," Shuttleworth said, adding that most people who move toward agile development tend to look at Scrum, Extreme Programming and LD (Lean Development), to name a few styles. "So we try to synthesize some of these ideas" regarding free software, he said.

For instance, with LD, Shuttleworth said the lesson learned is to "eliminate waste, ruthlessly, measurably." He said "lessons learned in classic production environments can be applied to software development—like from the auto industry."

Moreover, if something is piling up—bug reports, tests or whatever—that is waste, he said. "When you look at open source software , it’s amazing how much waste there is."

Another practice to follow is to amplify learning, Shuttleworth said. "In the free software community, the most productive people are not the specialists. It’s the ones who know a little bit about everything."

Another agile practice that can be adopted for open source development is "decide late, deliver early," Shuttleworth said. "Decide late freaks me out but it works. You do a little bit and deliver it. You have smaller chunks of work moving fast through the system."

Also, another lesson from agile development is to "build integrity in. Don’t try to add it after," Shuttleworth said. "See the whole—cultivate a shared result. It is not done until it’s completely done. This leads you to avoid handoffs, which is good because handoffs are one of the ways that things tend to pileup."

For his part, Shuttleworth said that he and his developers "look at the agile world for some of the key values—but in the community environment, maybe we can go beyond that."

For instance, pair programming is a core practice of agile development methods. However, in the open-source community, where developers are spread out all over the world, pair programming in the classic sense is difficult. "We value knowledge and interest over co location or sitting together," Shuttleworth said.

Another instance where open source takes a different tack from agile is with the notion of having an in-house customer. "We value community collaboration over internal development," Shuttleworth said. "That’s better than having a customer in house."

Meanwhile, the process of iteration—or iterative development—which is popular in the agile world, "requires a certain amount of planning," Shuttleworth said. "We value continuous integration over continuous iteration. That fits nicely into the free software world where you don’t know what’s coming in from the community."

As for quality, "we want it for every build, not just every release," he said.

As far as the pace of delivery, cadence is key," Shuttleworth said. "We need to go from ‘‘release early and often" to "this cycle—we added X." Every month we roll out new code, and beta users get code every day."

It is also important to know where you stand in terms of bugs, features and ideas, he said. "It’s amazing how many projects don’t have bug trackers."

On the issue of branching and merging code, Shuttleworth said developers "must keep the trunk pristine, keep releases flowing and release on demand." He said branching and merging is the key practice that enables the team to create cadence.

Meanwhile, Shuttleworth encourages code reviews, which accelerate learning, maintain quality and verify test coverage. He also said he is a stickler for automated testing, including unit testing, integration testing and utilization testing, as well as pre-commit testing. He joked that a pre-commit test is like the saying: I see you knocking but you can’t come in. "If any code fails the pre-commit test, the commit can’t happen," Shuttleworth said. "If you have pre commit testing, you can’t argue. It’s a way to make sure the trunk is always releasable."

Finally, he said the product is the platform, and "if it’s hard to hack they won’t come back," he said.

Shuttleworth said Canonical continues to draw from the agile and open source communities, which are not mutually exclusive, for ways to improve its development practices.

Meanwhile, from the Agile 2008 conference, according to a report released by Rally Software Development Aug. 4, development teams utilizing agile practices were on average 37 percent faster delivering their software to market and increased their teams’ productivity by 16 percent.

The report, entitled "The Agile Impact Report: Proven Performance Metrics from the Agile Enterprise," also concluded that agile teams were able to maintain normal defect counts despite significant schedule compression.

Commissioned by Rally Software Development in May and conducted through July by research firm QSM Associates, the study measured the real-life performance of 26 agile development projects against plan based or waterfall industry averages in three key areas: productivity, time to market and defects.

"Current market data on the agile industry often falls short of examining the true impact that agile practices have on teams throughout the entire lifecycle," said Tim Miller, Rally’s CEO. "We wanted to go beyond adoption rates or estimating project improvements and provide software organizations with proven metrics—supported by actual project examples—that accurately demonstrate how agile development projects measure up against plan based or waterfall projects."

By adopting agile practices, the companies measured in the report were able to produce large scale enterprise software in four to 12 months, compared to the one year to 13 months that it takes a typical organization to deliver comparable software.