Nic0's Sphere

Clint, Command Line Library for Python

2012-01-05T00:00:00+01:00

This library is the kind I like, the kind which make your life easier, perfect for someone like me, who uses Python all the time. So, what can Clint do for you? Well, there are a bunch of tools that might come handy when you do some command line based programming, especially quick scripts. If you use Python as your primary langage, CLI stuff is always useful.

Colors

Having some colored output can be nice, but I don’t want to use some ncurses interface to have a nice display. A common request is for text color to be green for an expected result, and red otherwise. The highest ranked answer from this stackoverflow’s question gives some'\033[93m', which it does the works, but isn’t lovely, or even pythonic.

Let’s see the simplest example possible:

from clint.textui import colored

print colored.red('some warning message')
print colored.green('nicely done!')

How intuitive was that! At least, I don’t need to check on the documentation every time I want a red message!

If you only need one word in different color, it’s as simple as a concatenation, here’s a quick example:

from clint.textui import colored

print 'I love ' + colored.yellow('pyt') + colored.blue('hon')

Indentation

One of the main feature of Clint, is to handle nicely indentations. It even managed nested indentation, a quote system, allow you to set some prefix chars, like quotes in emails for example. Here a quick example:

from clint.textui import colored, indent, puts

with indent(3, quote=colored.red(' >')):
    puts ('some random text')
    puts ('another text')
    with indent(3, quote=colored.green(' |')):
        puts('some more nested identation')
        puts('cool isn\'t?')

Once executed, the output gives:

> some random text
> another text
>  | some more nested identation
>  | cool isn't?

Looks good without breaking a sweat!

Arguments

There are ways to handle arguments, like the module argparse, or even sys.args. However, Clint wants to be as simple and useful as possible. It allows you to make distinction between the arguments passed:

args.all a simple list of every arguments passed.
args.flags list only flags, meaning with --foo or -f passed, excluding everything else that doesn’t contain a dash.
args.files this one could be very useful, it detects files in the arguments line, even with * wildcard. So for example you have in the directory foo.py and bar.py, entering something like *.py will return in the args.files a list with those two files.
args.not_files is every thing that wasn’t detected by the above argument.
args.grouped help you to managed grouped arguments.

For a redistributed program that I take time over, I would probably use argparse, as it’s more common, and gives you a nice help output. But for a quick homemade script, that only need to take few arguments, I really think this could be an alternative. Either way, it worth having a look at.

Progress bar, files and pipes

Clint comes with more tools, which are easy to use. Let’s look at a progress bar with Clint:

from time import sleep
from random import random
from clint.textui import progress

if __name__ == '__main__':
    for i in progress.bar(range(100)):
        sleep(random() * 0.2)

    for i in progress.dots(range(100)):
        sleep(random() * 0.2)

[########################        ] 75/100

Clint handles pipes, eg. cat foo.txt | bar.py, or resources like handling files IO.

Documentation and Conclusion

This post is not a review of every possible utility that Clint has to offer, but more of a quick overview, and memo for myself. I really think Clint could come in handy from time to time. Unfortunately, there is a lack of documentation, not much more than the README from the github repository, as the /doc/ directory doesn’t contain much. The best place to start is to directly check example codes provides with the library.

Another thing that could bother you is, so far, Clint only works with python 2.x. Python 3 compatibility is include in the todo list, as is documentations.
Edit about Python3: Not true anymore, thanks to Kenneth Reitz, whose done a release today with Python3 support!

I hope you enjoyed this quick review of Clint. Anyway, it’s in my toolbox now.

Changes on This Blog

2012-01-02T00:00:00+01:00

Since I started blogging, this is the biggest change I’ve done. So I felt like I own you an explanation. The main reason was to separate the English content from the French one. Well, at least, it’s what I’m telling myself, as I know I changed the blog engine for Jekyll, just because I like the way it works. I’m aware that it might sounds unreasonable to change of CMS like that, but I’m blogging for my pleasure, spending lots of time with, so it worth to change for a better system, that suite better my way about computers.

About French Content

First time I said in this blog that I wanted to write in english was in this post, it was more than 9 months ago. It took me some time to find the courage to do so, as I started only a couple of months ago.

Anyway, I moved the content to this link, I had to manage some redirections, as it isn’t a good idea to leaves 404 pages. I probably won’t use it that much now. I certainly only publish the Hacker Weekly News series, I did think about to stop it, but I like this series, helps me to classify information, and keep track of it. So I’ll should carry on the series on the French part of this site. So, if you still want follow the Hacker Weekly News series, you should add this feed on your reader.

The main site, of this post, keep the same feed as before, so no need to change anything.

About Jekyll

If you never heard about Jekyll, is a static website generator, the main idea is to generate the entire site in plain HTML, with obviously some JavaScript, but no server side language such as PHP. I never really liked hack into Wordpress, at the opposite to Jekyll, witch is much easier to get into. You just need some Ruby background. I hesitated with blogofile as it’s written in Python.

I won’t explain why Jekyll, or any static website generator, are nice to use, there is already lots of blog post around the web.

About Github Pages

I did want try this solution, Jekyll was build to works with Github pages, the idea is simply to push on the repository, and Github does the generation of the site. The problem is Github doesn’t allow any plugins. Fair enough, I can understand that they don’t want any arbitrary code to be run on them server, but it loose one of the main interest of it. No plugins, no hack, no interest, simple as that. So I’ll host myself, with some kind of rsync over ftp, seems like lftp does something like that with the mirror mode.

I would think about Github pages if I haven’t used a plugin to generate all my redirection.

I still have some stuffs and improvements to take care of.

That’s all for now, see you soon with some real blog posts!

PhotoShow, Web gallery without SQL

2011-12-04T00:00:00+01:00

When it comes to web gallery, I usually choose Piwigo, a safe choice for a stable PHP gallery.

Last week, I heard about a new project, called PhotoShow, a nice looking web gallery. A demo website is available. Here a quick functionalities review.

Easy installation, get latest tarball on github, or simply a git clone, set two lines from the configuration file, and you are ready to go.
Structure based on file system and directory, easy to manages and organizes photos and structures.
Generate thumbnails and 800*600 images automatically with phpthumb.
Two based directories, one with real photos, one with generated ones.
Drag n' Drop to upload and manages the website.
No SQL bases needed, it only based on the file system.
Nice looking jQuery effects.
Work fine without JavaScript.
Manage public and private albums.
Comments system.

First, I have to warn you that PhotoShow still on a early stage of his developpment. It seems stable enough, the project should improve quickly in a near future, with probably some new features. So perhaps it isn’t ready for production, but should be fine for a personal use.

Installation and configuration are straight forward, the easiest is use git clone, it would be simple to keep the gallery update with upstream by doing a git pull. You need to edit the conf.ini file, to set both photos and generated photos pathname. Once it done, you only need to answer the form to set your name/password account, and you’re done.o

git clone https://github.com/thibaud-rohmer/PhotoShow.git

Some would appreciate the drag n' drop system, I personally liked the disposition of thumbnails, compact as google galleries, managed well different size to find the optimal display, allow you to quickly see every photos of an album.

I will keep an eye on this project for sure.

At last, the code can be found on github, under the GNU GPLv3.

Manage automatically two dictionaries with Firefox

2011-12-03T00:00:00+01:00

When your native language isn’t English, and when, like me, you spend time writing around the Internet, between comments from some news sites, blog posts and programming question-answer site, you have to deal with at least two languages, English, and whatever language you use at home. Unfortunately, I have trouble with grammar and spelling^1. I probably have done what everyone else with bad spelling would do in this case, add to Firefox a spell checker. It does what I want, but as I write across websites, I need to switch the spell checker from French to English, certainly more often that I even notice.

Switching of dictionaries is annoying, but not enough to really do something about it. When you’re switching it, you think that perhaps you should try to google about it if there is any kind of shortcut or whatever would do the trick and save you some time, but you forget as soon as you switched the dictionary, and your mind focus on the answer/comment you’re writing. Same thing for asking to some geek friends, quickly forgotten to ask, every time.

So I’d carry this problem who isn’t really one, and by chance, I found the perfect solution for what I’m asking for. It could be obvious to some persons, but after all, I’m not switching of dictionary for that long time ago. The answer I found is no more that a Firefox plugin, called dictionary switcher. This plugin recognize automatically the language you are taping of, and adapt the dictionary in function of it, few more options on it to do more precisely what you want. With a name link that, I could find it earlier, but the thing is I didn’t toke time to look for.

Short blog post, for a quick trick, maybe it could be useful to someone else.

The missing newsletter of Arch Linux

2011-12-02T00:00:00+01:00

Most of Linux distributions comes with newsletters, a short summary of what happen around distributions and the community himself. Interestingly enough, Arch Linux doesn’t have anymore one, as far as I know. It wasn’t always like this, here a quick review of what was done so far.

y apologies if any informations was inaccurate.

The newsletter

Feb 2004 – Jul 2009

As the Arch community is growing, some of you may never heard about this old newsletter, start with weekly issue, quickly became a twice a month release, finished as a monthly issue. The information provide was good, but as I became an Arch user around this time, I haven’t the opportunity to read lots of them. For information, you still can read old archives on this link.

Arch Linux Newsletters Archives

I’m certainly not the only one, but I miss this newsletter.

Arch User Magazine

Apr 2009 – Jun 2009

Unofficial project initiate by Ghost1227, in a paper magazine style, old issues can be grab in the same link as the Newsletters Archives. We can still read the bbs thread for the first released issue. Three issues was made, and it will be merge with the official newsletter, as the author doesn’t have the time to carry on the project as he wanted.

Arch Linux Magazine

Aug 2009 – Mar 2010

The official newsletter, by Kensai, and the Arch User Magazine, by Ghost1227, will join their force together to provide a magazine available in both format html (newsletter format) and pdf (scribus/magazine format). Here is the bbs thread for nostalgic, about the first released announce by Kensai. I proudly translate issues with others French contributors in my language back this time. Unfortunately, a lack of contribution, and certainly of time as well, ended the magazine after 4 releases. But meanwhile, Kensai left for personal reasons.

Rolling Release

Sep 2010 – May 2011 (Until now?)

Rolling Release is an official website that aimed to succeed the Arch Linux agazine, as it’s an e-zine, no need to have a regular issue, or even spend time with scribus to make a nice looking pdf. Here the forum announce of the official launch of the site. Technically, the website look nice, allows comments. Even someone, jdarnold, started for few weeks a weekly issue on this site. The website has nice posts for a while, but hasn’t published anything since May 2011, without any official reason that I’m aware of, other than the lack of time I guess.

The missing newsletter

All this news was great informations, I’m very thankful for all the time that contributors spends on this, on their own time. But what’s now? I know what you think, “you want one? Do It Yourself!”, it’s absolutely true if I was following Arch Linux actuality enough, and a more fluent English.

What I do know is that this kind of initiatives often come from one man, no more, after all, Arch Linux came from a one man project as well. So, maybe if you are close to the community, following forum or mailing list, you might consider start some kind of newsletter. It doesn’t need to be great, long, with lots informations. Few links in a paper, with a short description, a couple of tricks or applications taken from the forum, it’s all it need to start something, I’m sure the community would be thankful for this, well I’ll be thankful for sure.

Tyrs 0.6.0, ncurses Twitter client, first release with Urwid

2011-12-01T00:00:00+01:00

Tyrs is a Twitter and Identi.ca client based on ncurses. This major release is the first one using the urwid library. It will improve greatly the user interface.

Edit a tweet: Previous release had a poorly way to edit tweet, I knew it, and wasn’t especially proud of it and always wanted to do it again from scratch. With urwid, it was easier to manage it. So now, it’s more like an IRC input bar to write tweet. We can do what we usually want, such as go back with arrow keys.
I had feedback as the unicode wasn’t perfect, editing a tweet with 2-bits long Chinese characters could sometime break the application. Fortunately, urwid has an excellent utf-8 support, so the overall management of unicode should be improve, and shouldn’t be an issue anymore.
More generally, a better display with less bugs.

In many way the interaction with user is improve, and in many details. This release still in early stage of development of the urwid interface, this mean it’s usable, but some functionalities aren’t yet implements, and will be done soon. I always follow the release early, release often plan, and even everything isn’t ready yet, I believe the improvement worth a release.

Some stuff that need to be done in further releases:

Configuration, lot of configuration was about the display, it will be the point of the next release.
Better default shortcut, I’m not so happy about the default shortcut, I’ve started with some, and added while I was doing the implementation, but, I guess it could be remap in much better way, even if it customizable in the configuration file.
Display without boxes, as I’m not a huge fan of line, I’ll probably do three default layout, coming soon.
Few more functionalities such as lists, information about user…

If you are using Arch Linux, the installation through the AUR package is easy:

yaourt -S tyrs
# Or the latest git revision
yaourt -S tyrs-git

You may find the latest code in the github repository, and find tarballs in the pypi page of the project.

I hope you’ll enjoy it. As usually, any feedback is much appreciated.

HTML output screenshot with Urwid ncurses library

2011-11-30T00:00:00+01:00

Urwid is a ncurses library for Python, you may check older posts if you’re not familiar about it. So far, we’ve seen some useful functionalities and tricks, to run urwid, therefor, it’s time to see something (almost) useless but geek, and in the spirit of ncurses programs, as urwid allow to take “screenshots” of the application, with a HTML output.

The problem with the reference guide of urwid, it’s that is not so easy to figure out how to get things done as we want. This corresponds to the screen capture part of the reference guide. But the easiest way to get started it’s to check source code, and not the real code of urwid, but the code who actually generate the tutorial of urwid, they do use this functionality for this part. Check in the tarball, docgen_tutorial.py to get examples.

Let’s take our own example, based on the code we’ve done in earlier post, for reminder, we obtain a list of item (couple title/description), in a urwid.Frame that allow to divide the space in three parts, header (the selected item), the body (list of the items), the footer (edit the description of current item). It was where we was last time. So let’s add a screenshot of it.

The idea

The screenshot will be triggered by the s keystroke, the idea is to start over the main loop, and pass in arguments the sequence of keystroke wanted.

Trick about screen size

We need to pass in the screenshot the size of the screen. It will be a tuple (y, x), it could be some random numbers, but it could be useful for the screenshot, and many other functionalities, to get the real size of the screen, and their is a variable for that.

self.loop = urwid.MainLoop(self.view, palette, unhandled_input=self.keystroke)
#[...]
#This variable contain the size in a tuple (x, y)
size = self.loop.screen_size

About the output

The output will come in a variable, but after every stroke we ask, their will be a screenshot, retrieve in a list, we need to choose the good one, probably the last one.

The screenshot code

import urwid.html_fragment
# [...]
    def keystroke(self, input)
    # [...]
        if input is 's':
            init = urwid.html_fragment.screenshot_init
            collect = urwid.html_fragment.screenshot_collect
            
            init([self.loop.screen_size], [['down']*3, ['enter'], ['q']])
            MyApp()
            with open ('output.html', 'w') as f:
                for l in collect()[-1]:
                    f.write(l)

In this code, we prepare two functions.

init will be pass the sequence of keystroke, quite forward to understand.
collect retrieve the result, as it was said earlier, it gives a list, we want only the latest element.

The result it place in the file output.html that we can see with firefox.

The result

selected: item 3: Cras a magna sit amet fel...                                  
  item 0       Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 1       Cras a magna sit amet felis fringilla lobortis.                  
  item 2       Sed sollicitudin, nulla id viverra pulvinar.                     
  item 3       Cras a magna sit amet felis fringilla lobortis.                  
  item 4       Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 5       Sed sollicitudin, nulla id viverra pulvinar.                     
  item 6       Sed sollicitudin, nulla id viverra pulvinar.                     
  item 7       Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 8       Cras a magna sit amet felis fringilla lobortis.                  
  item 9       Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 10      Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 11      Sed sollicitudin, nulla id viverra pulvinar.                     
  item 12      Sed sollicitudin, nulla id viverra pulvinar.                     
  item 13      Cras a magna sit amet felis fringilla lobortis.                  
  item 14      Cras a magna sit amet felis fringilla lobortis.                  
  item 15      Cras a magna sit amet felis fringilla lobortis.                  
  item 16      Sed sollicitudin, nulla id viverra pulvinar.                     
  item 17      Sed sollicitudin, nulla id viverra pulvinar.                     
  item 18      Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 19      Lorem ipsum dolor sit amet, consectetur adipiscing elit.         
  item 20      Cras a magna sit amet felis fringilla lobortis.                  
  item 21      Cras a magna sit amet felis fringilla lobortis.                  
  item 22      Sed sollicitudin, nulla id viverra pulvinar.                     
  item 23      Cras a magna sit amet felis fringilla lobortis.

The full code

import urwid
import random
import urwid.html_fragment

class ItemWidget (urwid.WidgetWrap):

    def __init__ (self, id, description):
        self.id = id
        self.title = urwid.Text('item %s' % str(id))
        self.description = urwid.Text(description)
        self.item = [
            ('fixed', 15, urwid.Padding(urwid.AttrWrap( self.title, 'body', 'focus'), left=2)),
            urwid.AttrWrap(self.description, 'body', 'focus'),
        ]
        self.content = 'item %s: %s...' % (str(id), description[:25])
        w = urwid.Columns(self.item)
        self.__super.__init__(w)

    def selectable (self):
        return True

    def keypress(self, size, key):
        return key

class CustomEdit(urwid.Edit):

    __metaclass__ = urwid.signals.MetaSignals
    signals = ['done']

    def keypress(self, size, key):
        if key == 'enter':
            urwid.emit_signal(self, 'done', self.get_edit_text())
            return
        elif key == 'esc':
            urwid.emit_signal(self, 'done', None)
            return

        urwid.Edit.keypress(self, size, key)

class MyApp(object):

    def __init__(self):

        palette = [
            ('body','dark blue', '', 'standout'),
            ('focus','dark red', '', 'standout'),
            ('head','light red', 'black'),
            ]

        lorem = [
            'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
            'Sed sollicitudin, nulla id viverra pulvinar.',
            'Cras a magna sit amet felis fringilla lobortis.',
        ]


        items = []
        for i in range(100):
            item = ItemWidget(i, random.choice(lorem))
            items.append(item)

        header = urwid.AttrMap(urwid.Text('selected:'), 'head')
        walker = urwid.SimpleListWalker(items)
        self.listbox = urwid.ListBox(walker)
        self.view = urwid.Frame(urwid.AttrWrap(self.listbox, 'body'), header=header)

        self.loop = urwid.MainLoop(self.view, palette, unhandled_input=self.keystroke)
        urwid.connect_signal(walker, 'modified', self.update)
        self.loop.run()
   
    def update(self):
        focus = self.listbox.get_focus()[0].content
        self.view.set_header(urwid.AttrWrap(urwid.Text('selected: %s' % str(focus)), 'head'))

    def keystroke (self, input):
        if input in ('q', 'Q'):
            raise urwid.ExitMainLoop()

        if input is 'enter':
            focus = self.listbox.get_focus()[0].content
            self.view.set_header(urwid.AttrWrap(urwid.Text('selected: %s' % str(focus)), 'head'))

        if input == 'e':
            self.edit()

        if input is 's':
            init = urwid.html_fragment.screenshot_init
            collect = urwid.html_fragment.screenshot_collect
            
            init([self.loop.screen_size], [['down']*3, ['enter'], ['q']])
            MyApp()
            with open ('output.html', 'w') as f:
                for l in collect()[-1]:
                    f.write(l)

    def edit(self):
        self.foot = CustomEdit(' >> ')
        self.view.set_footer(self.foot)
        self.view.set_focus('footer')
        urwid.connect_signal(self.foot, 'done', self.edit_done)

    def edit_done(self, content):
        self.view.set_focus('body')
        urwid.disconnect_signal(self, self.foot, 'done', self.edit_done)
        if content:
            focus = self.listbox.get_focus()[0]
            focus.description.set_text(content)
        self.view.set_footer(None)

if __name__ == '__main__':
    MyApp()

Several colored string with Urwid

2011-11-23T00:00:00+01:00

OK, this one is an easy and quick stuff, that I’m almost ashame to share. But the thing is, I search for it, and I wasn’t even sure where to look at it. The reason I put it here is because I trying to put together some trick about urwid, stuff I spend some time to looking for, even if it isn’t so much time. If someone need it, he should find it easily here.

The idea is to highlight some worlds in a string. The answer was right in the front of me from the start! As it was shown in the tour.py, the very first script in the official tutorial.

palette = [
    ('main', 'dark blue', ''),
    ('hl', 'dark red', '')
]

txt = urwid.AttrWrap(urwid.Text([
    "Some random text, with an ",
    ('hl', "IMPORTANT "), 
    "word. Other boring stuff."
    ]), 'main')

This short example, witch is only a part of a code of course, shows a string with the main color, with the IMPORTANT word colored with the hl attribute. It only a list, with a tuple for different colors, color attribute first.

As I said, it isn’t a very exalting post, just a quick trick.

Footer Edit with Urwid, IRC-Client-like input

2011-11-23T00:00:00+01:00

We start with the code from Selectable List with Urwid for Python, check before the Signals with Urwid for Python for a small involvement of the code, for handling signals.

So far, we have a selectable list containing two items each (title and description). We can go through it with up and down key arrows, and select one of them with the enter key. We use urwid.Frame witch give us an useful layout divided in three parts, header, body and footer. In the last post, we used the header to display the selected item, the body to display the list, and the footer wasn’t used at all.

Now, what we want is to use the footer for an Edit widget, exactly like ncurses IRC-client such as IRSSI or weechat. When we will press the e key (for edit) an input widget will appear, taking place at the footer, once validate, with enter, or abort with escape the output will replace the description of the current selected item.

In the following screenshot, the list was generated with random description. the 4th item was replaced with this editing process, and we’re about to replace the 5th item as well. It may seem not a big deal, but it still a good exercise to gain practice with urwid.

Let’s talk about the code. The full code is at the end of the post.

ItemWidget: is almost the same as before, it used to create an item for the list, it was only change in order to set the description part, here the changing part.

self.description = urwid.Text(description)
self.item = [
    ('fixed', 15, urwid.Padding(urwid.AttrWrap( self.title, 'body', 'focus'), left=2)),
    urwid.AttrWrap(self.description, 'body', 'focus'),
]

CustomEdit: inheritance of the Edit widget, it will emit a signal done once finish editing, and will give an argument to the callback, the content of the editing, or None if it was abort with the escape key. If neither keys is pressed, the key is return and managed by the normal Edit widget.

class CustomEdit(urwid.Edit):

    __metaclass__ = urwid.signals.MetaSignals
    signals = ['done']

    def keypress(self, size, key):
        if key == 'enter':
            urwid.emit_signal(self, 'done', self.get_edit_text())
            return
        elif key == 'esc':
            urwid.emit_signal(self, 'done', None)
            return

        urwid.Edit.keypress(self, size, key)

MyApp: I moved the code in a class, most of the code is the same except for the last two method.

The edit method is directly call when the e key is pressed. It only create a CustomEdit object, that would be set to the footer, but we need to specified to the frame (view) that the footer have the focus now, otherwise it would not be editable. At last, we connect the signal, to treat the content once the edit is done.

    def edit(self):
        self.foot = CustomEdit(' >> ')
        self.view.set_footer(self.foot)
        self.view.set_focus('footer')
        urwid.connect_signal(self.foot, 'done', self.edit_done)

Once the edit is done, the edit_done function is called as a callback, with the content. Then we modify the content of the description in the list. And set back to None the footer, as we don’t want anything display anymore.

    def edit_done(self, content):
        self.view.set_focus('body')
        urwid.disconnect_signal(self, self.foot, 'done', self.edit_done)
        if content:
            focus = self.listbox.get_focus()[0]
            focus.description.set_text(content)
        self.view.set_footer(None)

That’s it, in this example this code is used to manage a list, but it could be used for a large set of applications.

Here the full code:

import urwid
import random
import urwid.html_fragment

class ItemWidget (urwid.WidgetWrap):

    def __init__ (self, id, description):
        self.id = id
        self.title = urwid.Text('item %s' % str(id))
        self.description = urwid.Text(description)
        self.item = [
            ('fixed', 15, urwid.Padding(urwid.AttrWrap( self.title, 'body', 'focus'), left=2)),
            urwid.AttrWrap(self.description, 'body', 'focus'),
        ]
        self.content = 'item %s: %s...' % (str(id), description[:25])
        w = urwid.Columns(self.item)
        self.__super.__init__(w)

    def selectable (self):
        return True

    def keypress(self, size, key):
        return key

class CustomEdit(urwid.Edit):

    __metaclass__ = urwid.signals.MetaSignals
    signals = ['done']

    def keypress(self, size, key):
        if key == 'enter':
            urwid.emit_signal(self, 'done', self.get_edit_text())
            return
        elif key == 'esc':
            urwid.emit_signal(self, 'done', None)
            return

        urwid.Edit.keypress(self, size, key)

class MyApp(object):

    def __init__(self):

        palette = [
            ('body','dark blue', '', 'standout'),
            ('focus','dark red', '', 'standout'),
            ('head','light red', 'black'),
            ]

        lorem = [
            'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
            'Sed sollicitudin, nulla id viverra pulvinar.',
            'Cras a magna sit amet felis fringilla lobortis.',
        ]


        items = []
        for i in range(100):
            item = ItemWidget(i, random.choice(lorem))
            items.append(item)

        header = urwid.AttrMap(urwid.Text('selected:'), 'head')
        walker = urwid.SimpleListWalker(items)
        self.listbox = urwid.ListBox(walker)
        self.view = urwid.Frame(urwid.AttrWrap(self.listbox, 'body'), header=header)

        loop = urwid.MainLoop(self.view, palette, unhandled_input=self.keystroke)
        urwid.connect_signal(walker, 'modified', self.update)
        loop.run()
    
    def update(self):
        focus = self.listbox.get_focus()[0].content
        self.view.set_header(urwid.AttrWrap(urwid.Text('selected: %s' % str(focus)), 'head'))

    def keystroke (self, input):
        if input in ('q', 'Q'):
            raise urwid.ExitMainLoop()

        if input is 'enter':
            focus = self.listbox.get_focus()[0].content
            self.view.set_header(urwid.AttrWrap(urwid.Text('selected: %s' % str(focus)), 'head'))

        if input == 'e':
            self.edit()

    def edit(self):
        self.foot = CustomEdit(' >> ')
        self.view.set_footer(self.foot)
        self.view.set_focus('footer')
        urwid.connect_signal(self.foot, 'done', self.edit_done)

    def edit_done(self, content):
        self.view.set_focus('body')
        urwid.disconnect_signal(self, self.foot, 'done', self.edit_done)
        if content:
            focus = self.listbox.get_focus()[0]
            focus.description.set_text(content)
        self.view.set_footer(None)

if __name__ == '__main__':
    MyApp()

Hope it can help someone.

Signals with Urwid for Python

2011-11-22T00:00:00+01:00

Urwid, a Ncurses library for Python, handle a signal system, witch is nice for a ncurses library. You may want to check others posts on urwid if needed. We’ll see two examples here.

A first example

Let’s start with the code from the last post, for reminder, it does a selectable list, with SimpleListWalker, and display the selected item every time Enter is pressed. It’s a simple code, that we’ll extend a little.

What we want is to every time we move from the list, the selection is update to the header, showing the correct item, and we want have it working with a signal. Through, it would has other way to get it work, but it’s a way to start.

The reference about signals isn’t so easy to get into, maybe the best example from the tutorial is Events and listbox widget, but in this example, the signal is connected to an Edit Widget, here, we doesn’t have such thing, and we want use the list instead. If we try with the change argument, it would says to us that list doesn’t have such signal. So let’s find out witch signal a list has by reading the code source.

class ListWalker(object):
    __metaclass__ = signals.MetaSignals
    
    signals = ["modified"]

    def __hash__(self): return id(self)

    def _modified(self):
        signals.emit_signal(self, "modified")

Fine, as the SimpleListWalker extends to ListWalker, it seems that this widget use the modified signal, so let’s use this one for our example. Here the interesting part, but if you want the entire code, who is basically the same as the last post, you can find it in a gist.

    items = []
    for i in range(100):
        item = ItemWidget(i, random.choice(lorem))
        items.append(item)

    header = urwid.AttrMap(urwid.Text('selected:'), 'head')
    walker = urwid.SimpleListWalker(items)
    listbox = urwid.ListBox(walker)
    view = urwid.Frame(urwid.AttrWrap(listbox, 'body'), header=header)

    def update():
        focus = listbox.get_focus()[0].content
        view.set_header(urwid.AttrWrap(urwid.Text('selected: %s' % str(focus)), 'head'))

    loop = urwid.MainLoop(view, palette, unhandled_input=keystrock)
    urwid.connect_signal(walker, 'modified', update)
    loop.run()

Importants remarques:

The signal is connected to the SimpleListWalker, so the walker variable, and not to the listbox, as ListBox extends BoxWidget doesn’t respond to the modified signal.
The update function correspond at the same function when we pressed Enter in the last post.

That’s it, nothing complicated, we run it, and it works as expected, every time we move, the header is updated with the signal system.

What about others signals?

Their is no signal list in the documentation, so let’s find out some more signal with the documentation. We’ll looking for signals = ['foo'] in the code, it might give us a clue.

$ ack 'signals = \['
display_common.py
703:    signals = [UPDATE_PALETTE_ENTRY, INPUT_DESCRIPTORS_CHANGED]
 
vterm.py
1317:    signals = ['closed', 'beep', 'leds', 'title']
 
widget.py
719:    signals = ["change"]
 
wimp.py
104:    signals = ["change"]
422:    signals = ["click"]
 
listbox.py
37:    signals = ["modified"]

We find the the modified signal as shown in the first code in this post. And without surprise, widget.py contain a change signal, exactly as it was shown with the Edit Widget in the tutorial. The 719 line corresponding to the Edit class. So:

Edit (widget.py): change
ListWalker (listbox.py): modified
CheckBox (wimp.py): change
Button (wimp.py): click

vterm.py is more specific and exist only on the latest 1.0.0 version.

Custom signal from scratch

Let’s go deeper, and create a signal from scratch. The example will as simple as possible. We create an object, containing a function that raise a signal, on the other hand, an hello world that would change his message when it will receive the signal. I found another example from the mailing list but a bit more confusing maybe.

Here the simplest example that I could came with.

import urwid
import urwid.signals

class Sig (object):
    __metaclass__ = urwid.signals.MetaSignals
    signals = ['custom']

    def send_signal(self):
        urwid.emit_signal(self, 'custom')

def main():

    txt = urwid.Text("Hello World")
    fill = urwid.Filler(txt, 'top')
    sig = Sig()

    def update():
        """Callback once the signal is received"""
        fill.body = urwid.Text("Text Updated")

    def show_or_exit(input):
        if input in ('q', 'Q'):
            raise urwid.ExitMainLoop()
        #We press the 'u' letter (as in update)
        if input is 'u':
            sig.send_signal()

    #connection between the object, the signal name and the callback
    urwid.connect_signal(sig, 'custom', update)
    loop = urwid.MainLoop(fill, unhandled_input=show_or_exit)
    loop.run()

if __name__ == '__main__':
    main()

First thing, don’t forget the import. Now, we create the Sig object, it only purposes it to send a signal once we ask for by calling the send_signal function. Every line is important, but easily understandable.

The main function isn’t more difficult. I added some comments in case. The most important, it’s the callback to the update function and how it’s connected with:

urwid.connect_signal(sig, 'custom', update)

That’s all for now, I hope it’ll give you some more very basics example to get started with signals and urwid, as the documentation isn’t always easy. To be continued…

Selectable List with Urwid for Python

2011-11-20T00:00:00+01:00

Urwid is a ncurses library for Python, after a short introduction of urwid, I said I will keep update some training code. As urwid isn’t so easy library to start with and examples aren’t find all around the Internet, I through it could be interesting (for myself at least) to have some more code.

The complete code can be view at the end of this post

We want something very common, a simple list of items (title + description), and select one item as we go through the list. To understand easier what we want, here a screenshot of the result.

The main idea, is to create a selectable widget, with everything we want in it, and create a SimpleListWalker from a list of this widget.

First, let’s see this custom widget.

class ItemWidget (urwid.WidgetWrap):

    def __init__ (self, id, description):
        self.id = id
        self.content = 'item %s: %s...' % (str(id), description[:25])
        self.item = [
            ('fixed', 15, urwid.Padding(urwid.AttrWrap(
                urwid.Text('item %s' % str(id)), 'body', 'focus'), left=2)),
            urwid.AttrWrap(urwid.Text('%s' % description), 'body', 'focus'),
        ]
        w = urwid.Columns(self.item)
        self.__super.__init__(w)

    def selectable (self):
        return True

    def keypress(self, size, key):
        return key

Selectable method

To make a list, it have to be selectable, otherwise when you scroll, it will behave like a web browser, it will scroll the screen as soon as you press the down key, instead of go through every item of the screen before. For example, by default, urwid.Text isn’t a selectable item, and can be overwrite.

Keypress method

This is required, but in this example, we won’t handle keypress itself, so we simply return the key, and will be handle in the main loop.

Init method

What we do here is to prepare a widget that is a combination of other widgets, so we prepare it, and call it parent constructor in the last line.

Some more explanation:

        self.id = id
        self.content = 'item %s: %s...' % (str(id), description[:25])

This is some basic stuff, the content attribute is more to retrieve information after it would be selected.

 w = urwid.Columns(self.item)

First, I skip the explanation for self.item as it better to keep in mind what we want, so for every item, we display it as a two columns layout. so it will be align nicely.

            ('fixed', 15, urwid.Padding(urwid.AttrWrap(
                urwid.Text('item %s' % str(id)), 'body', 'focus'), left=2)),

There is a lot in this one.

fixed and 15, this will tell to the Columns widget that we want this column to be fixed to 15, as we want a smaller column for the title than the description. We affect at the column 15 cols width.
urwid.Padding and left=2, this tell we don’t want the list start right on the left, so we give it two cols space.
urwid.AttrWrap and body and focus, usual color wrap, with a second one that specify witch palette color do we use when the item have the focus, note that it could be an other name.
urwid.Text doesn’t need much more explanation.

urwid.AttrWrap(urwid.Text('%s' % description), 'body', 'focus'),

This is almost the same as the other one. This one is easier and doesn’t need more explanation.

That’s it for the ItemWidget, It’s ready and we only need to construct a list from this.

def main ():

    palette = [
        ('body','dark cyan', '', 'standout'),
        ('focus','dark red', '', 'standout'),
        ('head','light red', 'black'),
        ]

    lorem = [
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
        'Sed sollicitudin, nulla id viverra pulvinar.',
        'Cras a magna sit amet felis fringilla lobortis.',
    ]

Here’s go some basic stuff, the very classic palette, and a lorem generator that will give some random description.

    def keystroke (input):
        if input in ('q', 'Q'):
            raise urwid.ExitMainLoop()

        if input is 'enter':
            focus = listbox.get_focus()[0].content
            view.set_header(urwid.AttrWrap(urwid.Text(
                'selected: %s' % str(focus)), 'head'))

We handle keystrokes in the main method, the important thing is we get back the content of the focus element of the list of widget, and set the content of the header with it.

    items = []
    for i in range(100):
        items.append(ItemWidget(i, random.choice(lorem)))

this is where we construct the list, some basic stuff.

    header = urwid.AttrMap(urwid.Text('selected:'), 'head')
    listbox = urwid.ListBox(urwid.SimpleListWalker(items))
    view = urwid.Frame(urwid.AttrWrap(listbox, 'body'), header=header)
    loop = urwid.MainLoop(view, palette, unhandled_input=keystroke)
    loop.run()

And finally, we put everything together, by creating the SimpleListWalker.

Here the code, without my disturbing comments:

import urwid
import random

class ItemWidget (urwid.WidgetWrap):

    def __init__ (self, id, description):
        self.id = id
        self.content = 'item %s: %s...' % (str(id), description[:25])
        self.item = [
            ('fixed', 15, urwid.Padding(urwid.AttrWrap(
                urwid.Text('item %s' % str(id)), 'body', 'focus'), left=2)),
            urwid.AttrWrap(urwid.Text('%s' % description), 'body', 'focus'),
        ]
        w = urwid.Columns(self.item)
        self.__super.__init__(w)

    def selectable (self):
        return True

    def keypress(self, size, key):
        return key

def main ():

    palette = [
        ('body','dark cyan', '', 'standout'),
        ('focus','dark red', '', 'standout'),
        ('head','light red', 'black'),
        ]

    lorem = [
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
        'Sed sollicitudin, nulla id viverra pulvinar.',
        'Cras a magna sit amet felis fringilla lobortis.',
    ]
    
    def keystroke (input):
        if input in ('q', 'Q'):
            raise urwid.ExitMainLoop()

        if input is 'enter':
            focus = listbox.get_focus()[0].content
            view.set_header(urwid.AttrWrap(urwid.Text(
                'selected: %s' % str(focus)), 'head'))

    items = []
    for i in range(100):
        items.append(ItemWidget(i, random.choice(lorem)))

    header = urwid.AttrMap(urwid.Text('selected:'), 'head')
    listbox = urwid.ListBox(urwid.SimpleListWalker(items))
    view = urwid.Frame(urwid.AttrWrap(listbox, 'body'), header=header)
    loop = urwid.MainLoop(view, palette, unhandled_input=keystroke)
    loop.run()

if __name__ == '__main__':
    main()

Their are certainly others ways to do this, but as I’m learning Urwid, I didn’t find this information so easily, so I though I could be useful to someone else who will want have a try with Urwid.

This list, even if it’s a very basic one, is a common thing, as lots application need some kind of selectable list. I probably publish some more exercises of this kind as I’ll go through my learning of Urwid.

Urwid for Python, a ncurses library

2011-11-19T00:00:00+01:00

This, is the first part of a six posts series on Urwid.

I really like ncurses programs, it’s certainly what I use the most for an every day usage. The curses module from Python provides some elements to build an application, it’s what I used for my last program Tyrs. Unfortunately, the module doesn’t provide any widgets or advances elements, usually instead of reinvents the wheel, we use a library. As far as I know, Python have only two libraries for ncurses.

PyCDK is some alpha stage library, who doesn’t seem to be maintain anymore. It’s originally a binding of the CDK library for C. I tried it, but I didn’t manage to get it work, maybe I was unlucky. Anyway, I would not advise this library.
Urwid is a serious library, who’s come with lots of features and widgets, and will be the subject of this post.

Unfortunately, their isn’t any others libraries that those two.

It’s always interesting when starting consider a library to know if their are others programs using it too, a list is provide by the website, with some interesting programs, such as wicd-curses, hachoir and the Debian “reportbug” tool.

The most interesting pieces of example to get start with come from this page, a set of small programs with screenshot.

Urwid come with lots of features and widget, allow you to display a nice layout easily, editing text, 256 colors, utf-8, check box, advance list management, mouse event and much more. The documentation and tutorial is well documented.

Urwid is maybe not so easy to get start with if you’re not used to play with library, and maybe it’s why it isn’t more popular but have lots of potential. Urwid has became mature, and after seven years of development as reach the 1.0.0 release few months ago.

Here is a traditional Hello World, with a quit keystroke handling.

import urwid

txt = urwid.Text("Hello World")
fill = urwid.Filler(txt, 'top')

def exit(input):
    if input in ('q', 'Q'):
        raise urwid.ExitMainLoop()

loop = urwid.MainLoop(fill, unhandled_input=exit)
loop.run()

I will soon write some more posts about Urwid, with codes and more technical. This is only a short introduction of Urwid. I hope this post will give you enough reason to give Urwid a try.

Tyrs a Microblogging Client based on Ncurses

2011-11-06T00:00:00+01:00

Tyrs is a microblogging client, supporting Twitter and Status.net (identi.ca), it’s based on console using the NCurses module from Python. The release of the 0.5.0 version is a good excuse to introduce Tyrs.

Tyrs aims to get a good interaction with a fairly intuitive interface that can provide support ncurses. Tyrs tries also not to impose choices to the user by providing a configuration file for features, display and defaults keys. You don’t like borders, space between tweets, or colors? It’s all configurable in an INI-like file.

It’s really for the fun I started Tyrs, as I’m not a developer. I chose to start this project because I use it, as well as educational purpose. Moreover, contrary to what you should believe, I am not a big Twitter user, just a little regular user, wanting to have pleasure to write a program. The first lines are from May

The license used is GPLv3, its development is carried out on a repository on Github, the code source is also available in the pypi page. Tyrs is Free Software.

I first used other clients, GUI or CLI, but the CLI alternatives doesn’t correspond specifically to my expectations. There are mainly:

BitleBee, is an application to be added to IRC client such as irssi or weechat. I never was a big fan of bitlebee.
TTYtter, is the application I was using before writing Tyrs. Very complete and mature. His approach does not include however ncurses, and therefore, although it is easily controllable from the keyboard, it doesn’t have the interactive that have ncurses software.

Since then, I had the opportunity to discover other microblogging clients, but generally without ncurses support. According to my tastes, I through that there was a lack to find my favorite ncurses Twitter client, and so, fit my console application collection that I use every days.

Here a screenshot (full screen view) to visualize what we are talking about, it isn’t my timeline, but someone’s public timeline.

![Screenshot of Tyrs] (http://pix.toile-libre.org/upload/img/1320604662.png)

Let’s go a little deeper into of Tyrs with an overview of the main features.

Timelines
- Home: Used to view your tweets timeline, similar to what you have on your home page of the site.
- Mention: Includes tweets containing your nickname.
- Direct: Includes messages sent privately.
- Favorites: Displays tweets who has been bookmarked.
- Other …

The Timelines are navigable with the arrow keys or with ‘j’ and ‘k’. We could be compared to buffers of a ncurses IRC client.

This way of separate timelines is quite close to what can be seen in the microblogging classic clients with a graphical interface. It’s probably a reason why the interface is intuitive even if it’s a console application.

Indicator of activities

As we can display only one timeline, the indicator of activity indicates whether new notices have appeared in other timelines, this detail avoids having to navigate between timelines only to check for new incomings. A counter can also quantify the activity. This place can also view the timeline currently selected. Location is at the top right.

Standard features:
- OAuth authentication;
- Follow / Unfollow, ie subscribe to a feed of persons;
- Search for a particular keyword;
- Favorites;
- Direct messages;
- Opening direct URLs;
Various
- Retweets Counter: This is a detail, but after using it a while, I must say that I am used to. The purpose is to indicate the number of times a tweet has been relayed (retweeted). It is therefore a possibility to visualize in real time, popularity of a status.
- Support URL shortener, so far, the main services are supported ur1.ca, bit.ly and goo.gl.
- Proxy support through the configuration file. This is one of the latest feature added.
- Filter system for tweets that do not contain URLs, the system that can filter by name. It’s a way to reduce noise.
- Waterline to identify the last tweet read in the last use, it is re-adjusted if necessary by just a key strock.
- View a thread to get the conversation history. The answers to a tweet are viewable by an envelope icon (✉).
- Ability to start several instances of Tyrs, passing as argument ‘-a account’.

The number of keys shortcut can be confusing at first, the most common is displayed at the bottom of the screen, that can be switched off, and a help screen is available by pressing ‘h’.

As mentioned in the introduction, the main idea is to leave the choice up to the user, and most behavior is customizable through a configuration file, that is also generated at startup to ease your first customization.

Tyrs has been translated into French and Spanish, so this may seem anecdotal, but it isn’t for me because is a real example of what I like in free software, opportunities for contributions and exchange of humans from all horizons, not speaking a word of Spanish, but it’s indeed a contribution straight from Venezuela, which is given to Tyrs. Other aids to correct my poor English thanks to jasonwryan from Arch Linux, or proposals for patches and bug reports and the packaging for distribution, including Frugalware. Contributions and help which are always appreciated, especially because it’s a small project.

If I am passionate about programming, I also have another focus is to keep a blog. I mention here is not to make an advertisement, but because I think the topics about Tyrs(unfortunately in French) are indicative of the state of mind in which I came to write Tyrs, Speaking of this little program is the first time early in its design, while the feature was only able to read his instructions without being able to to send. There are some form of narrative developments and problems sometimes encountered in darkness, discovering ncurses for Python.

For installation, more details are on the documentation page, a package is available for Arch Linux and Frugalware, in the case of Arch Linux, you simply do:

yaourt -S tyrs

Tyrs are mainly tested with Python 2.7, I’m not sure of the result obtained with earlier versions of Python. However if you have this version of Python and python-pip installed you can do:

pip install tyrs

Tyrs is not perfect, far from it, as seen above, support for Python prior to 2.7 is not really tested. Another black, is editing a record, I’m really not satisfied with the system before, I think I would see to rewrite that part in-depth, something I wanted to do for a long time, but always rejected. Certainly other bugs, depending of the moon and distribution of butterflies.

I opened a thread on the Arch Linux Forum.

Free Software often come with no warranty but I did my best not to introduce bugs. I’m interested in all returns, regardless of the form, comments, bug reports, patches. Most important thing is I really enjoyed write Tyrs.

Some usefull links to learn more about Tyrs:

Why AUR is part of the Arch Linux success

2011-11-01T00:00:00+01:00

If you usually follow some blogs about Linux, especially those on Arch Linux, there is a common word that you are come across often, AUR, acronym of Arch User Repository. You might wonder what does it mean, what does it imply for the distribution, and why it’s so popular for the Arch Linux community. If you asking yourself those questions, this post is for you.

First, you should keep these two characteristics in mind.

Everybody may contribute to AUR.
It’s really easy to contribute to AUR.

AUR is a kind of repository totally public, open to whoever want to contribute to let some resources available to anyone. No ceremonials, agreement or long ritual initiation to submits a package to AUR.

It’s as easy to contribute to AUR as to use it, with some AUR helpers such as yaourt or others alternatives. Install a package from AUR isn’t more difficult than install one with the official package manager pacman, manager such as apt-get for Debian or Ubuntu. Dependencies and research are easily managed that way. Even a vote system allow credibility about security and code source of a package.

The main idea about packaging for AUR is to write some installation directives in a file called PKGBUILD, by simply follow some easy packaging’s rules. That’s what AUR contain, only directives that gives instruction to the helper on how to build the package. There aren’t binary packages in AUR.

The fact that a PKGBUILD is easy to write increase the idea that everyone can contribute to AUR given that the repository is open to everyone. If you managed to install Arch Linux by yourself, you should be able to find out how to write PKGBUILD. Of these two ideas, easy to contribute, and for everyone, resulting implications very beneficial for Arch Linux.

For example, a person who have some practice with it, can make a script or a program available within few minutes (which was the case for example to offer hacker-top and reddit-top a bit on a whim). If more time was needed, would I have done it? probably not. This is also beneficial for the user, so you may easily enjoy interesting programs that would not be known however.

This simplification of the provision also encourages some exotic applications, with for example some very recent drivers that would not be incorporated otherwise. But also with the latest programming frameworks in vogues. Knowing that a framework can gain very big reputation in a short time. So latest framework can be available easily and quickly to users. Most importantly, it allows them to be updated, This is important if the evolution is rapid. For example, if I try Node.js, I do not want the version of last year, but the latest. Same for Ruby on Rails, I do not want to watch a version of Ruby, which may not be compatible with the latest Rails. This last example taken from a real example, when we saw on the forums concerns about running Rails with version 3, especially for Debian. All installed cleanly with a packet manager, not by hand as we see often advisable in such cases.

All this convenience, and the non-formality could even be quantified. The website, front-end of AUR, announces 32,000 packages available for archers, it’s more than 29,000 packages debian announced on their home page. So I know that is not the same and that is not far from being a troll, but it is a fact that Arch Linux has a community and a number of developers much less important than Debian, and is undeniable, for this system, Arch Linux is able to offer to his users almost as many packages.

From the maintainer’s point of view, those concerned with the [community] repository, they can easily demote a package in AUR so it was officially supported, without worrying to much about consequences, associated with the fact that Arch Linux only supports two architectures, not dragging dozen architecture to their credit. these two points I believe are the key to their reactivity, and can keep a small core team (no more than two dozen core-dev, without TU’s). A responsiveness of a small team to a rolling release enough stable to interest a median users of Linux.

AUR also allows a developer to easily share its application, whatever the language chosen, each language generally offers opportunities, such as pip (python) gem (ruby), it is still nice to have an opportunity to ensure the smooth running of installation on its own distribution. Simplify installation and distribution of software that is not known and still under development, is a good way to promote an application. Furthermore, PKGBUILD allows be easily coupled to a manager for review as git or svn, allowing for easier returns on development issues by an user.

Among all these points, everyone’s winner, from the median user to developers. I hope that with these explanations, far from being exhaustive, the success of the AUR for Arch Linux is more clear implications that can be blurred at first.

Without AUR, Arch Linux would no longer be the same.

Edit: More comments on Hacker News, or on Reddit.

Comparison of major Linux distributions forums

2011-08-30T00:00:00+02:00

It could be useful to visualize the registrations of the major Linux distribution forums, and gives an idea of the health of each one. This is what it will be done here. A previous article(French) describe step by step the procedure used to retrieve registrations date from forums, based on scrapy to crawl the forum and pylab to draw graphs.

There is two kind of graphics, sums of registrations per months, and per day. The first one is better to visualize average, second one give more accurate graphs. As the last month (august) isn’t complete, there is a fall in last spot for months graphs. To collects data, I took my time, make sure it was accurate, check between number of the file entry, and what it’s actually said in the forum. In doubt I run tests twice.

Bellow is a list of forums that I goes through here. They are major forum for each distribution, witch I added some French one to compared the accuracy.

Arch Linux
- archlinux.org
- archlinux.fr
Ubuntu
- ubuntu.org
- ubuntu-fr.org
Debian
- debian.org
- debian.fr
Fedora
Linux Mint
PCLinuxOs
Puppy
Gentoo
Sabayon
Chakra
Mandriva
OpenSuse
Conclusion

Arch Linux

Archlinux (en)

Website: http://bbs.archlinux.org
Members: 37,331
1st registration: 2002-08-12

Archlinux (fr)

Website: http://forums.archlinux.fr
Members: 3,040
1st registration: 2006-08-11

Ubuntu

Ubuntu (en)

Website: http://ubuntuforums.org
Members: 1,416,637
1st registration: 2004-10-07

This site doesn’t have any memberlist, this option was deactivate, so I needed to go through every members profile. This wouldn’t be a problem if the forum hasn’t 1.4 million members. So I took only one of every ten members, based on the id number. This give 140,000 members crawled. So, the graphic is not the exact reality, but close enough I’m sure.

The last raise is surprising, and doesn’t appear on the French one, It might be some spam, such as with the Debian one (we see this one right after). It not easy to be sure, as they don’t provide a user list.

Ubuntu (fr)

Website: http://forum.ubuntu-fr.org/
Members: 171,126
1st registration: 2004-12-17

Debian

Debian (en)

Website: http://forums.debian.net/
Members: 37,567
1st Registration: 2004-02-15

This forum has recently lots of spam user, they aim to have a backlink in the member list. The list contain lots of not Linux related links.

Debian (fr)

Website: https://www.debian-fr.org/
Members: 9,859
1st registration: 2003-02-28

Fedora

Fedora (en)

Website: http://forums.fedoraforum.org/
Members: 172,622
1st registration: 2004-01-26

Fedora (fr)

Site: http://forums.fedora-fr.org
Membres: 20,837
1ère inscription: 2002-01-19

Linux Mint

Website: http://forums.linuxmint.com/
Members: 53,972
1st registration: 2006-11-15

PCLinuxOs

Website: http://www.pclinuxos.com/forum/
Members: 6,194
1st registration: 2006-01-18

Puppy

Website: http://murga-linux.com/puppy/
Members: 18032
1st registration: 2005-05-04

Gentoo

Website: http://forums.gentoo.org/
Members: 133,317
1st registration: 2002-04-08

Sabayon

Website: http://forum.sabayon.org/
Members: 27,869
1st registration: 2005-10-14

OpenSuse

Website: http://forums.opensuse.org/
Members: 61,223
1st registration: 2007-06-25

Mandriva

Website: http://forum.mandriva.com/en/
Members: 67,460
1st registration: 2001-12-16

Chakra

Website: http://chakra-project.org/bbs/
Members: 4,614
1st registration: 2008-12-15

Conclusion

I started this mainly to test scrapy with a real example, but curiosity push me to conduct those tests for major Linux distribution. I really thinks this reflect the reality, of each distributions popularity. It’s interesting to compare results with DistroWatch, specially as the provide result from past years.

Few distributions missing, as CentOS, Knoppix, Frugalware, but I had some difficulties to retrieve results from them forum. And Frugalware as not enought members to make result, less than one hundred, due to a very young forum.

As I said last time, yes, Arch Linux doing well. I was surprise with the rise of Linux Mint, confirm with the result from DistroWatch, between first and second place with Ubuntu. I’m surprise too than Fedora didn’t get a rise when Ubuntu turn to Unity, as Arch Linux had. And I’m glade that Gentoo come back.

At last, a small ranking of the health of the Linux distributions:

Good health: Arch Linux, Linux Mint, PCLinuxOs,
Stagnant: Ubuntu, Debian
Not so good: Fedora, Puppy, Sabayon, Gentoo
Bad shape: Mandriva, OpenSuse, Chakra