Fedora 22 is out and it’s again the most quality release we’ve ever released. Our quality assurance is improving and on the developer side, we’re also trying to do our best heavily using ABRT retrace server to prioritize bugs that affect many users. Unfortunately while the quality of releases itself is improving, the quality of updates that follow the release is not.
There are still too many regressions. I’ve installed Fedora on computers of my relatives and they’re happy with it, but sadly I can’t let them do updates themselves because there is still a high risk that they might end up with a broken system. I update their systems myself and always check whether everything is working when I pay them a visit. If we want to attract a larger user base, average users can’t be afraid to update their systems.
IMHO our current updates setup doesn’t ensure the required quality. It’s pretty much a “one-size-fits-all” approach. The kernel, the most critical part of the system, needs the same number of + karmas as some small unimportant, self-contained utility. Updates of critical components get to users too quickly without much of testing. I’ve got updates-testing repo enabled, but whenever I find a (critical) regression it’s very often too late because the update already got +3 karma and made it to the stable updates. Yeah, I already have the “Missed the train” badge 🙂
While Bodhi is too fast for standard updates, it’s too slow for critical security fixes. Especially in older supported releases (F20 now). There are not many testers willing to test updates there. The active community are usually early adopters who jump on new releases early and a several-month-old release is history to them. Then security updates just get stuck in Bodhi waiting for stable karmas.
What to do with it? I truly believe we need batch updates. One pack of updates, say, once a month. We would collect updates in updates-testing for 3-4 weeks, then freeze it for a week, so that even the latest updates have some time to be properly tested (I can imagine the pack of updates gets some more structured testing like our releases do for example). This way, individual updates would get much more time to be tested and the monthly update could be tested as a whole. I believe it would improve the quality of updates and users would not be under the fire of updates (it’s actually one of frequent complaints that there are too many updates in Fedora).
I don’t see a lot of downsides there. Who’d like to get updates as soon as possible could still enable updates-testing. This actually could build an even bigger community of update testers which would again help improve the quality of updates.
Any security updates? It’s clear that they can’t wait for a month to reach the users. They will need their own process. But I think it’s clearer and clearer that they will need their own process in the current setup as well. Maybe pulling in the security team which would evaluate proposed security updates and if they approve them as critical they will get into some fast track?
Brno hat 
Delaying would not matter if things are not getting tested? Question is how do you get more tester. If a fix has been released up upstream and us stopping it for another month doesn’t sound so great.
I’m not saying things don’t get tested, I’m saying there is not enough time to test them. As I said in the article, when I get to test some updated package and find a critical regression, it’s already in stable going to thousands of users who can’t often cope with such breakages.
Very interesting post.
Well, the problem is that often the most strange and serious problems are hardware specific, so they are hard to spot. Indeed, a more structured serie of updates could improve the process of testing, so I think that your suggestions should be discussed further in the appropriate places. I’d also think that subteams of QA (for example, testers of F20, testers focused on Rawhide, testers responsible for security…) might also improve the whole coverage against different releases and focus better our time.
//Giulio (juliuxpigface)
Sorry for the late reply, I just got back from vacation. A few points:
1. +3 for the autopush threshold is not a hard-coded value and is not actually related to any update policy; it’s simply a default value which was more or less pulled out of a hat. Maintainers can and often *should* change it when submitting updates, including setting it to 0 to disable autopush entirely. Kernel updates in fact have the value set to 0 (at least recently) and the maintainers are careful about pushing them stable manually. I can’t recall which, but IIRC there’s another commonly-used component whose maintainer usually sets the threshold to 5. It would probably be a good idea for the Firefox maintainers to set the threshold for their updates much higher than +3, it would be worth suggesting that to them directly.
On the other hand, I often set the threshold to 2 or even 1 for small updates to obscure things I own which I know are never likely to get three items of feedback. It’s configurable, and it’s expected that maintainers change it.
2. So far as what karma is actually *required* for an update to go stable, we do actually have different requirements for different types of packages. The requirements for ‘critical path’ packages – https://fedoraproject.org/wiki/Critical_path_package – are rather stronger than for other packages. These requirements can always be changed and we could even add more granularity beyond ‘critpath’ and ‘non-critpath’, but the principle is already established. See https://fedoraproject.org/wiki/Updates_Policy .
3. It’s been known for a long time that Bodhi could do with many improvements, and Bodhi 2.0 – including lots of these – has been under development for almost as long. It’s just not done yet. It seems to always be scheduled for release Real Soon Now…
I’m not really opposed to batch updates at all, though.