I would like to see the following bugs fixed during 2011, the first randomly prevent users from logging off, the second randomly presents an “ugly” unexpected appearance.
https://bugs.launchpad.net/ubuntu/lucid/+source/gnome-panel/+bug/439448
https://bugs.launchpad.net/ubuntu/+source/gnome-settings-daemon/+bug/574296
Both are responsible for a disruptive desktop experience for many users. I also wish that the effort for the new “consistent user experience for desktop” does not keep or increase our current inability to fix such severe open source problems.
Thursday, December 30, 2010
Thursday, November 11, 2010
GetDeb mirror pool status
I was able to identify and fix the python threading deadlocking problem, mirror-selector is fully functional now.
I have also added some statistics which allow us to check mirrors health and traffic distribution, you can check it at:
http://archive.getdeb.net/status/
I have also added some statistics which allow us to check mirrors health and traffic distribution, you can check it at:
http://archive.getdeb.net/status/
Tuesday, November 9, 2010
Help with Python threading deadlock
Hello,
some months ago I have started developing the mirror-selector daemon.
The main work is done, it is capable to run for a few hours serving some thousand requests but it then get's into a thread deadlock scenario (the web server stops responding, gdb shows all the threads waiting on a semaphore lock).
This was my first project using multi-threading with Python and I am having a bad time finding the bug. I have recently introduced some debug code that I hope will print the stacktrace for every thread and give me an hint on the cause. It is most likely related Queue management.
The mirror-selector code is not that complex, it's available from bzr "bzr branch lp:mirror-selector", if you are experienced with Python threading and can spare a few minutes reviewing the code for the possible cause, it would be helpful.
Thanks
some months ago I have started developing the mirror-selector daemon.
The main work is done, it is capable to run for a few hours serving some thousand requests but it then get's into a thread deadlock scenario (the web server stops responding, gdb shows all the threads waiting on a semaphore lock).
This was my first project using multi-threading with Python and I am having a bad time finding the bug. I have recently introduced some debug code that I hope will print the stacktrace for every thread and give me an hint on the cause. It is most likely related Queue management.
The mirror-selector code is not that complex, it's available from bzr "bzr branch lp:mirror-selector", if you are experienced with Python threading and can spare a few minutes reviewing the code for the possible cause, it would be helpful.
Thanks
Monday, October 11, 2010
GetDeb/PlayDeb repositories for Ubuntu 10.10 are now open
We had another great Ubuntu release, the GetDeb/PlayDeb repository setup for the 10.10 is done and packages should start landing on the next days. For those which prefer to keep on the LTS release we will keep updating the packages for Lucid for at least the current release.
Friday, July 23, 2010
GetDeb: Archive traffic distribution
GetDeb user base increased exponentially since it was started on 2006. The migration to a proper APT repository while providing important benefits is also a big technical challenge. The GetDeb and PlayDeb repositories are presently configured by more than 30k users, providing tens of Gigabytes from our mirror pool during traffic peaks.
On 2007 we had a single server and the traffic was unaffordable, we have gathered some mirrors and we have developed a php script which was responsible for validating files availability on the candidate mirrors and then redirect the users to them (using http redirect). This script was poorly developed but sufficient for a long time.
Before moving to APT the file requests were human originated from web clicks, now this scrip is massively used by the automatic system upgrades, it's original faults have now a much serious impact. It needs to be replaced.
I have checked existing solutions for mirror distribution:
APT mirror: method - APT supports a specific mirror: method which dynamically obtains a mirror from an URL, however it's transaction based, the same archive will be used for all requests after an initial retrieval. This means that on the beginning of transaction it should get the url of a mirror which provides all the files required by the subsequent operations. For GetDeb this is a major limitation, since we have very frequent updates (somtimes hourly) most of the mirrors would be unavailable for mirror selection because they would be out of synch, even if they do have the packages for that specific transaction there is no way to know in advance. This issue is not present with http redirects, we always return the packages index from the master server, files will be obtained from individual mirrors as long they match the master server version, regardless of the overall mirror status.
Mirrorbrain - Mirrorbain is used by mainstream solutiosn like OpenSuse's build service and OpenOffice so it was a strong candidate. After some research I have found that it detects file availability by using a database which must be kept current using a mirror scan tool which does a full mirror scan (file info: size, last modified). While this maybe great for most scenarios I don't think it is as efficient as doing on demand mirror check, our slowest mirror took >10m for a full scan, we would need large intervals increasing the risk of redirection to a failed mirror.
mirror-selector - Because I have a strong believe on the technical merit of the on demand scan I have decided to implement a mirror selection system from scratch using Python.
The utility/project name is "mirror-selector" it runs as a standalone HTTP Server whose only purpose is to handle static file GET requests, check the availability from a local directory (it must be run on a local mirror) and then redirect to an available mirror after checking that an exact copy of the file is available remotely.
The http server uses a fixed size thread pool, each web client request is handled on it's own http server thread. When mirror-selector starts a thread is started for each mirror, each mirror thread provides an input queue which maybe used by any http server thread. With this architecture all requests related to a unique mirror are handled on a single thread, this allows to easily reuse the same TCP connection by using HTTP 1.1 Keep-Alive for multiple requests. The caching facility is also simpler to implement because it works on a per thread basis.
The code is available at launchpad: bzr branch lp:mirror-selector (check the README to test it), it should be considered as alpha.
GetDeb's/PlayDeb's main archive pool was already switched to mirror-selector, we may intermittently swap to the legacy selector as serious problems maybe still be found.
To check if it's available and some stats:
http://archive.getdeb.net/status/
On 2007 we had a single server and the traffic was unaffordable, we have gathered some mirrors and we have developed a php script which was responsible for validating files availability on the candidate mirrors and then redirect the users to them (using http redirect). This script was poorly developed but sufficient for a long time.
Before moving to APT the file requests were human originated from web clicks, now this scrip is massively used by the automatic system upgrades, it's original faults have now a much serious impact. It needs to be replaced.
I have checked existing solutions for mirror distribution:
APT mirror: method - APT supports a specific mirror: method which dynamically obtains a mirror from an URL, however it's transaction based, the same archive will be used for all requests after an initial retrieval. This means that on the beginning of transaction it should get the url of a mirror which provides all the files required by the subsequent operations. For GetDeb this is a major limitation, since we have very frequent updates (somtimes hourly) most of the mirrors would be unavailable for mirror selection because they would be out of synch, even if they do have the packages for that specific transaction there is no way to know in advance. This issue is not present with http redirects, we always return the packages index from the master server, files will be obtained from individual mirrors as long they match the master server version, regardless of the overall mirror status.
Mirrorbrain - Mirrorbain is used by mainstream solutiosn like OpenSuse's build service and OpenOffice so it was a strong candidate. After some research I have found that it detects file availability by using a database which must be kept current using a mirror scan tool which does a full mirror scan (file info: size, last modified). While this maybe great for most scenarios I don't think it is as efficient as doing on demand mirror check, our slowest mirror took >10m for a full scan, we would need large intervals increasing the risk of redirection to a failed mirror.
mirror-selector - Because I have a strong believe on the technical merit of the on demand scan I have decided to implement a mirror selection system from scratch using Python.
The utility/project name is "mirror-selector" it runs as a standalone HTTP Server whose only purpose is to handle static file GET requests, check the availability from a local directory (it must be run on a local mirror) and then redirect to an available mirror after checking that an exact copy of the file is available remotely.
The http server uses a fixed size thread pool, each web client request is handled on it's own http server thread. When mirror-selector starts a thread is started for each mirror, each mirror thread provides an input queue which maybe used by any http server thread. With this architecture all requests related to a unique mirror are handled on a single thread, this allows to easily reuse the same TCP connection by using HTTP 1.1 Keep-Alive for multiple requests. The caching facility is also simpler to implement because it works on a per thread basis.
The code is available at launchpad: bzr branch lp:mirror-selector (check the README to test it), it should be considered as alpha.
GetDeb's/PlayDeb's main archive pool was already switched to mirror-selector, we may intermittently swap to the legacy selector as serious problems maybe still be found.
To check if it's available and some stats:
http://archive.getdeb.net/
Saturday, July 10, 2010
Something is wrong with the software publishing plan for Ubuntu
Lately I have been watching a couple of threads related to Ubuntu packaging and publishing whichs makes me feel something is wrong.
With all respect, I believe Matt was probably dreaming when he came up with the "We’ve packaged all of the free software" title. There are plenty of applications not properly archive maintained or even packaged due to the lack of human resources.
Then Jorge on this article narrowed the issue to the the centralized versus distributed model. It's a bit concerning the argument which involves "iPhone’s app store", will Ubuntu app store follow iPhone's restrictive strategy because *some* people like it ?
Opinions apart, there will be a new process for applications inclusion which is great but which I am afraid will be commercially oriented. If Ubuntu does not have the resources to handle the current community oriented package inclusion process for development releases, REVU I really don't see how it will be able to handle post release inclusions unless there is revenue involved.
With all respect, I believe Matt was probably dreaming when he came up with the "We’ve packaged all of the free software" title. There are plenty of applications not properly archive maintained or even packaged due to the lack of human resources.
Then Jorge on this article narrowed the issue to the the centralized versus distributed model. It's a bit concerning the argument which involves "iPhone’s app store", will Ubuntu app store follow iPhone's restrictive strategy because *some* people like it ?
Opinions apart, there will be a new process for applications inclusion which is great but which I am afraid will be commercially oriented. If Ubuntu does not have the resources to handle the current community oriented package inclusion process for development releases, REVU I really don't see how it will be able to handle post release inclusions unless there is revenue involved.
Tuesday, July 6, 2010
Python and OpenOffice - Hello World
Recently I had the need to automate data gathering into an openoffice calc heet, after some frustration playing with the OO macro language (OOBasic) I did some research on how to interact with OO from Python. There is some documentation spread over wikis and forums but not that much so I am going to share my learning here.
This is a simple hello world script which will fill the "A1" Cell from the current active sheet with "Hello World" in bold.
Please note that I will be using regular python scripts launched from the terminal and which will interact with OO using a UNO socket bridge.
First you will need to start oocalc in listenting mode (to accept UNO connections), from the terminal:
That is all for the first lesson :)
This is a simple hello world script which will fill the "A1" Cell from the current active sheet with "Hello World" in bold.
Please note that I will be using regular python scripts launched from the terminal and which will interact with OO using a UNO socket bridge.
First you will need to start oocalc in listenting mode (to accept UNO connections), from the terminal:
oocalc "-accept=socket,host=localhost,port=8100;urp;StarOffice.ServiceManager" -norestore -nofirststartwizard -nologoNow just execute python and try the following code:
from uno import getComponentContext
from com.sun.star.awt.FontWeight import BOLD
# Connect to soffice using UNO
localContext = getComponentContext()
resolver = localContext.ServiceManager.createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", localContext)
context = resolver.resolve("uno:socket,host=localhost,port=%d;urp;StarOffice.ComponentContext" % 8100)
# Get desktop service
desktop = context.ServiceManager.createInstanceWithContext("com.sun.star.frame.Desktop", context)
# Get current document
document = desktop.getCurrentComponent()
# Get the active sheet
sheet = desktop.getCurrentComponent().getCurrentController().getActiveSheet()
# Get the "A1" cell reference
cell = sheet.getCellRangeByName("A1")
# Set a string
cell.setString('Hello World')
# Change to bold
cell.CharWeight = BOLD
That is all for the first lesson :)
Friday, April 30, 2010
GetDeb: Ubuntu 10.04 kick-off
The GetDeb and PlayDeb repositories for Ubuntu Lucid provide packages including more than 100 applications and 100 games, this was the first time we had most of the previous release packages already available on the new release day. The credit goes to Christoph Korn which tested/fixed most of the packages, thanks also to all the others which helped.
We have recently added change logs generation to our build system, when using update-manager you will be able to check the change logs before deciding to upgrade to a getdeb package.
We have recently added change logs generation to our build system, when using update-manager you will be able to check the change logs before deciding to upgrade to a getdeb package.
Tuesday, April 20, 2010
When "recovery" mode fails to boot
*** Don't try this unless you understand what you are doing ***
The recent changes to the boot process with the integration of the graphical boot loader and logger (Plymouth) and the continuous development of upstart and associated jobs brought a significant improvement to Ubuntu's Lucid boot experience.
Because the the code is new and is rapidly changing there are some good chances of bugs which may lead to an non bootable system.
Some of the bugs found (promptly fixed, or being worked) on the development stage shown that if you have an issue related to plymouth/mountall the rescue option on the boot menu will not work.
Rescue mode as implemented now depends on plymouth/mountall being started in the first place.
To work around this you will need to be able to boot without automatically starting upstart, you can do this by appending using the init kernel option, from the boot menu, press e for edit, search for the linux line, append the init=/sbin/sulogin, press ctrl-x to boot. If you don't get a root prompt then something was wrong during kernel load/initialization, or while the root filesystem was mounted (removing the quiet option from the linux kernel line might provide some hint on the problem).
So now we are in a root shell, what's next ? On my case I am going to set the console keyboard layout: loadkeys pt .
We can't yet start services, we are running from an sulogin process and we need upstart to be able to start services. We will just backup and disable the upstart services first:
The console will just hang because nothing was started, except our virtual terminals 8 and 9, now let's swtch to VT8 with with CTRL-ALT-F8.
Now list the the available services: initctl list, a bit boring, only our sulogins, let's put the regular jobs in place: cp -p /etc/init.orig/* /etc/init .
Try initctl list again, a long list of services now, which you can start with "start service".
Please note Upstart is an event driven job management system, starting a service will most likely trigger other services which depend on events emitted by the first one. My next research will be on how to avoid the chain reaction, or if not possible request such a feature :)
I hope you don't need this instructions :)
The recent changes to the boot process with the integration of the graphical boot loader and logger (Plymouth) and the continuous development of upstart and associated jobs brought a significant improvement to Ubuntu's Lucid boot experience.
Because the the code is new and is rapidly changing there are some good chances of bugs which may lead to an non bootable system.
Some of the bugs found (promptly fixed, or being worked) on the development stage shown that if you have an issue related to plymouth/mountall the rescue option on the boot menu will not work.
Rescue mode as implemented now depends on plymouth/mountall being started in the first place.
To work around this you will need to be able to boot without automatically starting upstart, you can do this by appending using the init kernel option, from the boot menu, press e for edit, search for the linux line, append the init=/sbin/sulogin, press ctrl-x to boot. If you don't get a root prompt then something was wrong during kernel load/initialization, or while the root filesystem was mounted (removing the quiet option from the linux kernel line might provide some hint on the problem).
So now we are in a root shell, what's next ? On my case I am going to set the console keyboard layout: loadkeys pt .
We can't yet start services, we are running from an sulogin process and we need upstart to be able to start services. We will just backup and disable the upstart services first:
# Remount the root fs in read-writeNow we can start upstart: exec /sbin/init .
mount -orw,remount /
# Backup the upstart jobs config
cp -a /etc/init /etc/init.orig
# Nothing will start
rm /etc/init/*
# Add 2 console virtual terminals
echo start on startup > /etc/init/sulogin8.conf
echo exec openvt -c 8 >> /etc/init/sulogin8.conf
echo start on startup > /etc/init/sulogin9.conf
echo exec openvt -c 9 >> /etc/init/sulogin9.conf
The console will just hang because nothing was started, except our virtual terminals 8 and 9, now let's swtch to VT8 with with CTRL-ALT-F8.
Now list the the available services: initctl list, a bit boring, only our sulogins, let's put the regular jobs in place: cp -p /etc/init.orig/* /etc/init .
Try initctl list again, a long list of services now, which you can start with "start service".
Please note Upstart is an event driven job management system, starting a service will most likely trigger other services which depend on events emitted by the first one. My next research will be on how to avoid the chain reaction, or if not possible request such a feature :)
I hope you don't need this instructions :)
Sunday, April 18, 2010
Upgrading to Lucid with Virtualbox USB support renders the system unbootable
Last friday Mina Shokry joined in#Ubuntu+1 reporting an unbootable system after upgrading to Lucid. Even booting with rescue mode it was hanging without any error message, apparently it was after mounting the file systems (guessing by the "mounted filesystem" message).
After several hours trying all sort of boot options we are able to identify the problem.
The /proc/bus/usb on fstab which was previously required for USB support in VirtualBox mount was failing, however upstart/mountall (not sure where is the fault) did not provide an error and simply stopped the boot process.
The bug for the mount issue is bug 565109 .
I have also filed a bug for upstart, bug 565890, simply making it verbose when using rescue would allow to easily identify the problematic task.
If you use VB's USB support make sure you remove the /proc/bus/usb entry from your fstab.
After several hours trying all sort of boot options we are able to identify the problem.
The /proc/bus/usb on fstab which was previously required for USB support in VirtualBox mount was failing, however upstart/mountall (not sure where is the fault) did not provide an error and simply stopped the boot process.
The bug for the mount issue is bug 565109 .
I have also filed a bug for upstart, bug 565890, simply making it verbose when using rescue would allow to easily identify the problematic task.
If you use VB's USB support make sure you remove the /proc/bus/usb entry from your fstab.
Monday, April 12, 2010
Update Manager will display 3rd party change logs
There was an old bug reported on update-manager about it's lack of support for non official repositories change logs display. This was something already requested on getdeb and it benefits most people which use PPAs, checking the change logs usually provides a good hint on the upgrade risk.
I had proposed a possible solution in the past, today I have check with Michael Vogt that the proposal was ok. I got positive feedback so this was the time to help with some code.
I don't have much experience with python-apt so the research over the weekend using winpdb (the python debuger) was fundamental. I have found the function responsible for the changelog file retrieval, get_changelog() at UpdateManager/Core/MyCache.py and added the change log URI construction based on the package origin server, later Michael fixed it using the ArchiveURI from the package.
The patch introduced a new feature so it required a Feature Freeze approval, it was granted quickly, it is a small patch providing a perceived benefit.
Please note this feature depends on the changelog files being available on the server, at ARCHIVE_URI/changelogs/... (pool structure). The repository maintainers need to implement their own changelog extraction utilities. GetDeb has already implemented it, however it will only be available once we upgrade our build server to 10.04 (there are some other building related changes blocking).
A task was also submitted to Soyuz so PPA's should support it in the future.
I had proposed a possible solution in the past, today I have check with Michael Vogt that the proposal was ok. I got positive feedback so this was the time to help with some code.
I don't have much experience with python-apt so the research over the weekend using winpdb (the python debuger) was fundamental. I have found the function responsible for the changelog file retrieval, get_changelog() at UpdateManager/Core/MyCache.py and added the change log URI construction based on the package origin server, later Michael fixed it using the ArchiveURI from the package.
The patch introduced a new feature so it required a Feature Freeze approval, it was granted quickly, it is a small patch providing a perceived benefit.
Please note this feature depends on the changelog files being available on the server, at ARCHIVE_URI/changelogs/... (pool structure). The repository maintainers need to implement their own changelog extraction utilities. GetDeb has already implemented it, however it will only be available once we upgrade our build server to 10.04 (there are some other building related changes blocking).
A task was also submitted to Soyuz so PPA's should support it in the future.
Saturday, April 10, 2010
Crashes with Ubuntu 10.04 beta 2
I have been using Ubuntu 10.04 since beta 1 and it has been quite stable until this week.
Yesterday on my laptop openoffice simply topped launching, running oowriter from the terminal did not reported any error but nothing happened. I was in a rush to finish a job so I was unable to debug the problem further so I simply rebooted.
During boot the root filesystem was found corrupted, the graphical boot just kept looping changing the message on the screen from "Checking disk blah blah" to "key X por blahblah". I was unable to do recovery, next step, reboot in recovery mode.
Still not good, now it was just looping on the text console, I was unable to read the output.
Another reboot, this time I kept pressing the key that was supposed to bring the recovery console, this time I was able to get into the recovery shell after some iterations of the message loop. I was able to run fsck on the root filesystem and successfully boot.
Today I have booted my desktop (also running Lucid), was about to do some draft html for a status page, decide to install "quanta" for it.
During the install I just got a white screen with X hard locking, well time for another reboot.
I was not sure about the quanta package install status, I have found that quanta was installed by quanta.desktop file was truncated with 0 bytes.
Later I have found that other quanta dependency packages where corrupted
I have filled bugs 559915 and 559915.
Anyone with such bad experiences lately ? I am a bit concerned about having such critical failures on 2 completely different systems.
Yesterday on my laptop openoffice simply topped launching, running oowriter from the terminal did not reported any error but nothing happened. I was in a rush to finish a job so I was unable to debug the problem further so I simply rebooted.
During boot the root filesystem was found corrupted, the graphical boot just kept looping changing the message on the screen from "Checking disk blah blah" to "key X por blahblah". I was unable to do recovery, next step, reboot in recovery mode.
Still not good, now it was just looping on the text console, I was unable to read the output.
Another reboot, this time I kept pressing the key that was supposed to bring the recovery console, this time I was able to get into the recovery shell after some iterations of the message loop. I was able to run fsck on the root filesystem and successfully boot.
Today I have booted my desktop (also running Lucid), was about to do some draft html for a status page, decide to install "quanta" for it.
During the install I just got a white screen with X hard locking, well time for another reboot.
I was not sure about the quanta package install status, I have found that quanta was installed by quanta.desktop file was truncated with 0 bytes.
Later I have found that other quanta dependency packages where corrupted
I have filled bugs 559915 and 559915.
Anyone with such bad experiences lately ? I am a bit concerned about having such critical failures on 2 completely different systems.
Friday, April 9, 2010
Installing a Debian schroot with debfactory
I have recently had the need to teste a package on Debian Lenny, it was a good reason to add support for Debian on the debfactory schroot_build.py script.
Durint testing I have found bug 557316, you will need to add http://ftp.debian.org/debian to /etc/apt-cacher-ng/backends_debian before proceeding.
To create and test on a Debian Lenny schroot just run:
Durint testing I have found bug 557316, you will need to add http://
To create and test on a Debian Lenny schroot just run:
bzr branch lp:debfactory && sudo debfactory/bin/schroot_build.py -c /home/chroot -p debian -d lenny -a i386 && schroot -c lenny.i386 -p
Tuesday, April 6, 2010
GetDeb/PlayDeb: Ubuntu 10.04 Lucid Lynx Repository Available
The repository for Ubuntu 10.04 repository is now available.
Lucid is an important release (LTS status) and we have a significant number of packages/updates which will not be available on the official repositories, with such in mind we are now building/testing only for Lucid.
Lucid is an important release (LTS status) and we have a significant number of packages/updates which will not be available on the official repositories, with such in mind we are now building/testing only for Lucid.
Tuesday, January 5, 2010
Some questions and answers about GetDeb
Today I have found this blog post which raises some interesting questions about GetDeb.
"who runs it" - A community of volunteers, list of individuals which can upload to the "-testing" repository is available from https://launchpad.net/~getdeb-uploaders/+members . A more restricted subset can copy packages from -testing to the general repository.
"What releases do they package?" - The general policy is to provide the current stable version, there are exceptions, we provide development releases when recommended by program authors.
"Are program authors involved in any way?" - It depends, on some cases package requests are received from the program authors, on other cases we just grab the authors source and there is no interaction.
"Are the programs modified before they land in the repository?" - In general no, on some rare cases we do apply building/integration fixes, on those cases we send the fixes to the author's also.
There's GetDeb, but I'm not using it yet. I've got questions that the GetDeb site doesn't have answers for: who runs it? What releases do they package? Are program authors involved in any way? Are the programs modified before they land in the repository? As an issue of trust, I'm more willing to add repositories like Banshee's official PPA to Software Sources over a third-party I know nothing about.The answers:
"who runs it" - A community of volunteers, list of individuals which can upload to the "-testing" repository is available from https://launchpad.net/~getdeb-uploaders/+members . A more restricted subset can copy packages from -testing to the general repository.
"What releases do they package?" - The general policy is to provide the current stable version, there are exceptions, we provide development releases when recommended by program authors.
"Are program authors involved in any way?" - It depends, on some cases package requests are received from the program authors, on other cases we just grab the authors source and there is no interaction.
"Are the programs modified before they land in the repository?" - In general no, on some rare cases we do apply building/integration fixes, on those cases we send the fixes to the author's also.
Subscribe to:
Posts (Atom)