How to lose 5 year of life: 2° try...

The first try was when I applied a wrong configuration to grub and thereafter the laptop did not boot anymore (ok, it was some years ago...)
This time, after what seemed a normal update of my laptop, which is now running Archlinux, and a concurrent installation of cutegram, I found that I was unable do anything on the machine and I was basically locked out from my machine. Login from console give "login error" even before asking for the password and even with root. SSH did not work and also sudo or su did not work.

After a bit od digging, I was able to reboot to a shell rescue using systemd, I just needed to add the option systemd.unit=rescue.target to the kernel command line.

Once logged in, I begun to look for the problem. The first suspect was cutegram, but it turn out innocent.

After a bit of analisys of the log, I found this error:
Jun 17 23:35:05 galactica login[30384]: PAM adding faulty module: /usr/lib/security/pam_unix.so Jun 17 23:35:01 galactica login[30379]: PAM adding faulty module: /usr/lib/security/pam_access.so
which was followed by
PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libtirpc.so.1: undefined symbol: __rpc_get_default_domain

The basic problem is that I was sure that I had not updated anything related to PAM or friends. The second error, anyway, directed me to the kernelof the problem: for some reason, the libtirpc library was the wrong version (also if I still don't know why).

The first try to solve the problem was to try to run pacman -S libtirpc, which oddly enough update the library: after that the system restarted to work properly.

I still don't know what happened and I am writing this to maybe help someone else. My impression was that it was an unfortunate update timing (since I use a mirror) which for some combination of factor completed my update but at the same time not all the packages in the mirror was at the correct version.

STL Export on Bulbcalculator

BulbCalculator (in HEAD) has now the export to STL file at least as Ascii. This follow a request from a user to be able to print a model with a 3D printer.
Using this request as starting point I added some features that are likely to be used to mill the model, so the STL can also be exported as half model.

There is, for now, a limitation in this since only one half of the model is saved to the STL file.
If you have a model perfectly symmetrical you just have to mill two times the object and you have a complete bulb. Conversely, if you have an asymmetric model (I presume on the vertical plane) this leads to a small amount of work on the CAD/CAM side since you have to generate a mirrored object.

This limitation will be eliminated in a next version, but for now you should keep it in mind if you plan to print (or mill) a bulb designed with BulbCalculator.

Two new features for BulbCalculator

Following a request from a BulbCalculator user, I started the work on implementing the export of the bulb into a STL model file, to be able to print a model with a 3D printer (or to be able to mill the bulb with a milling machine :-) ).

I also started to implemente the possibility to add a hole for the keel, which is somewhat tricky since for any material you remove, you should update the bulb dimension to take care of the difference.

When I started to work on these two features, I realized thatI needed to do some preliminary works that have, as a result, optimizede the code, especially the code that draw the 3D view of the model.

In the end I now have a more real wireframe view, while I added two other view modes: triangles and a real surface. The triangles view was a preliminary work to be able to create the STL file, the other one is just for fun ;-)

Bulbcalculator ported to QT5

After some work, Bulbcalculator is finally ported to QT5, with the future 2.2.0 version. While I am working on this, I also decided to implement some other features, firs of all a recent project list and the possibility to setup the page size, like some very well known CAD programs.

I have not a date to release the 2.2.0 version, since I have not so much time to work on it, but the work is slowly going on.

Working on stl-tools

Stl-tools get some work in the last couple of days which will be the basis for future development.

The main point I nailed out is to have a structure that can support commands that needs different number of parameters.

The old implementation needs to specify the axis for all the operations, but while adding the new operation stubs, I realize that not all of the operations I'd like to implement need the axis specification and, more important, not all of them refer to an axis.

With this in mind I removed the -x option and update the mirror operation to reflect this. The new method is based on a structure (technically a python dict) which hase a "name" , which is the operation itself, and a nested structure (another python dict) which contain all the couples option-value for the operation. This give the required liberty to implement every operation without the risk to touch every part of the code every time I will need to have another option for a new operation.

While working on this, I also corrected the mirror operation definition so now the it is defined to refer to a plane instead of an axis, which is way too more logical.

The last thing done is to move all the help related function to specific class. With this I hope to have a clearer code

stl-tools operation: mirror

This want to be the first of a series of posts that will explain the operation that can be performed by stl-tools

The first, and the one which inspired me to write the software, is mirror

This was the first since I originally write stl-mirror (which later became stl-tools) to mirror a STL file generated from FREE!Ship without the necessity to use a full 3D CAD/CAM program.

Sintax:
stl-tools -i source_file.stl --axis=x|y|z [-o dest_file.stl] mirror

This operation mirror (what a surprise ;-) )the object given in the source_file along one of it's axis (X, Y or Z), writing it to a new file, so also if something is wrong, the original is not lost.

If the -o parameter is omitted, the output file will be named output[sourcefile].stl

On upgrading server software

Short story: when upgrading a production system, always check the release notes of the new version.

Long story.

After the first upgrade of the server following the release of the new Debian stable, I was no more able to send e-mail from my home pc, but I am still able to send from the server itself, which is running Debian testing, so in the following days of the Debian release, testing got a lot of upgrade as usual.

This upgrade also boost postfix to the 2.1.0 release.

Sending mail from my home pc, then result in the error:

May 11 09:28:22 localhost postfix/smtpd[17094]: connect from unknown[79.21.142.151]
May 11 09:28:24 localhost postfix/smtpd[17094]: NOQUEUE: reject: RCPT from unknown[79.21.142.151]: 554 5.7.1 
<gianlum@xxxxxxx>: Relay access denied; from=<gian@grys.it> to=<gianlum@xxxxxxxx> proto=ESMTP helo=<galactica>
May 11 09:28:24 localhost postfix/smtpd[17094]: lost connection after RCPT from unknown[79.21.142.151]
May 11 09:28:24 localhost postfix/smtpd[17094]: disconnect from unknown[79.21.142.151]

It turn out that from this release on, to be able to have a relay (from home I send mail using my server and not the provider's one), you should put the client restriction in the directive

smtpd_relay_restrictions

instead of the old

smtpd_recipient_restrictions

as always in the main.cf file

I discovered this the hard way, since I was in a hurry to send a mail, and before I can find this change, I tried a gazillion other solutions and also checked for every blacklists in the case the server and/or my provider nerwork were blocked.
So, lessons learned:

  1. Always read the release notes (or at least, check them for the important software)
  2. Do not try massive upgrade past 10.00 pm or you will end to lose sleep and do not solve anything anyway, at least until the next morning
  3. If you need to send a very urgent email, send it before trying a massive upgrade :-D

Python and profiling

While working on the Rosalind's problems, I discover a nice feature of python, the cProfile module, that produce an output like this:

         5683 function calls (5637 primitive calls) in 0.013 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1012(__init__)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:1018(load_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1025(get_filename)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1030(get_data)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1040(path_stats)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:108(_path_isfile)
        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1257(_path_importer_cache)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1274(_get_loader)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1299(find_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1346(find_loader)
        6    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1446(__enter__)
        6    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1450(__exit__)
        3    0.000    0.000    0.001    0.000 <frozen importlib._bootstrap>:1464(_find_module)
      3/2    0.000    0.000    0.004    0.002 <frozen importlib._bootstrap>:1500(_find_and_load_unlocked)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:154(new_module)
      3/2    0.000    0.000    0.005    0.002 <frozen importlib._bootstrap>:1550(_find_and_load)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1587(_handle_fromlist)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:181(__init__)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:201(acquire)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:226(release)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:266(_get_module_lock)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:280(cb)
      3/2    0.000    0.000    0.003    0.002 <frozen importlib._bootstrap>:305(_call_with_frames_removed)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:34(_relax_case)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:415(cache_from_source)
        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:486(_verbose_message)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:496(set_package_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:509(set_loader_wrapper)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:536(module_for_loader_wrapper)
      5/1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:581(_check_name_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:593(_requires_builtin_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:61(_r_long)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:643(find_module)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:654(load_module)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:700(find_module)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:74(_path_join)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:754(_open_registry)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:761(_search_registry)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:776(find_module)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:796(is_package)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:804(_bytes_from_bytecode)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:849(_load_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:86(_path_split)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:934(get_code)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:98(_path_is_mode_type)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:996(load_module)
        1    0.000    0.000    0.000    0.000 codecs.py:238(__init__)
      852    0.001    0.000    0.002    0.000 cp1252.py:18(encode)
        1    0.000    0.000    0.000    0.000 cp1252.py:22(decode)
        1    0.000    0.000    0.000    0.000 functools.py:148(__init__)
        3    0.000    0.000    0.000    0.000 functools.py:152(__hash__)
        1    0.000    0.000    0.000    0.000 functools.py:155(_make_key)
        3    0.000    0.000    0.000    0.000 functools.py:167(<genexpr>)
        1    0.000    0.000    0.003    0.003 functools.py:241(wrapper)
        1    0.000    0.000    0.000    0.000 locale.py:555(getpreferredencoding)
        1    0.000    0.000    0.003    0.003 re.py:212(compile)
        1    0.000    0.000    0.000    0.000 re.py:230(escape)
        1    0.000    0.000    0.003    0.003 re.py:264(_compile)
        1    0.002    0.002    0.013    0.013 ros_kmp.py:3(<module>)
      168    0.000    0.000    0.000    0.000 ros_kmp.py:45(<listcomp>)
      212    0.000    0.000    0.000    0.000 ros_kmp.py:5(Find)
      212    0.001    0.000    0.001    0.000 ros_kmp.py:8(Comp)
        4    0.000    0.000    0.000    0.000 sre_compile.py:178(_compile_charset)
        4    0.000    0.000    0.000    0.000 sre_compile.py:207(_optimize_charset)
        2    0.000    0.000    0.000    0.000 sre_compile.py:258(_mk_bitmap)
     12/1    0.000    0.000    0.001    0.001 sre_compile.py:32(_compile)
        2    0.000    0.000    0.000    0.000 sre_compile.py:357(_simple)
        1    0.000    0.000    0.000    0.000 sre_compile.py:364(_compile_info)
        2    0.000    0.000    0.000    0.000 sre_compile.py:470(isstring)
        1    0.000    0.000    0.001    0.001 sre_compile.py:473(_code)
        1    0.000    0.000    0.003    0.003 sre_compile.py:488(compile)
       22    0.000    0.000    0.000    0.000 sre_compile.py:51(fixup)
        9    0.000    0.000    0.000    0.000 sre_parse.py:127(__len__)
       36    0.000    0.000    0.000    0.000 sre_parse.py:131(__getitem__)
        2    0.000    0.000    0.000    0.000 sre_parse.py:135(__setitem__)
       14    0.000    0.000    0.000    0.000 sre_parse.py:139(append)
     14/3    0.000    0.000    0.000    0.000 sre_parse.py:141(getwidth)
        1    0.000    0.000    0.000    0.000 sre_parse.py:179(__init__)
      310    0.001    0.000    0.001    0.000 sre_parse.py:184(__next)
       52    0.000    0.000    0.000    0.000 sre_parse.py:203(match)
      281    0.000    0.000    0.001    0.000 sre_parse.py:209(get)
        1    0.000    0.000    0.000    0.000 sre_parse.py:222(tell)
        1    0.000    0.000    0.000    0.000 sre_parse.py:224(seek)
       25    0.000    0.000    0.000    0.000 sre_parse.py:227(isident)
        4    0.000    0.000    0.000    0.000 sre_parse.py:233(isname)
        2    0.000    0.000    0.000    0.000 sre_parse.py:284(_escape)
      6/1    0.000    0.000    0.002    0.002 sre_parse.py:340(_parse_sub)
      9/1    0.000    0.000    0.002    0.002 sre_parse.py:418(_parse)
        1    0.000    0.000    0.000    0.000 sre_parse.py:68(__init__)
        1    0.000    0.000    0.000    0.000 sre_parse.py:702(fix_flags)
        1    0.000    0.000    0.002    0.002 sre_parse.py:714(parse)
        4    0.000    0.000    0.000    0.000 sre_parse.py:73(opengroup)
        4    0.000    0.000    0.000    0.000 sre_parse.py:84(closegroup)
       12    0.000    0.000    0.000    0.000 sre_parse.py:91(__init__)
        1    0.000    0.000    0.003    0.003 string.py:15(<module>)
        1    0.000    0.000    0.000    0.000 string.py:162(Formatter)
        1    0.000    0.000    0.000    0.000 string.py:51(_TemplateMetaclass)
        1    0.000    0.000    0.003    0.003 string.py:61(__init__)
        1    0.000    0.000    0.000    0.000 string.py:73(Template)
        2    0.000    0.000    0.000    0.000 {built-in method OpenKey}
        3    0.000    0.000    0.003    0.001 {built-in method __build_class__}
        1    0.000    0.000    0.000    0.000 {built-in method _fix_co_filename}
        1    0.000    0.000    0.000    0.000 {built-in method _getdefaultlocale}
        6    0.000    0.000    0.000    0.000 {built-in method acquire_lock}
        6    0.000    0.000    0.000    0.000 {built-in method allocate_lock}
        1    0.000    0.000    0.000    0.000 {built-in method charmap_decode}
      852    0.001    0.000    0.001    0.000 {built-in method charmap_encode}
        1    0.000    0.000    0.000    0.000 {built-in method compile}
      2/1    0.000    0.000    0.013    0.013 {built-in method exec}
        6    0.000    0.000    0.000    0.000 {built-in method get_ident}
        5    0.000    0.000    0.000    0.000 {built-in method getattr}
       26    0.000    0.000    0.000    0.000 {built-in method getlower}
       13    0.000    0.000    0.000    0.000 {built-in method hasattr}
        1    0.000    0.000    0.000    0.000 {built-in method hash}
        2    0.000    0.000    0.000    0.000 {built-in method init_builtin}
        3    0.000    0.000    0.000    0.000 {built-in method is_builtin}
        1    0.000    0.000    0.000    0.000 {built-in method is_frozen}
       43    0.000    0.000    0.000    0.000 {built-in method isinstance}
1322/1319    0.000    0.000    0.000    0.000 {built-in method len}
        1    0.000    0.000    0.000    0.000 {built-in method loads}
        4    0.000    0.000    0.000    0.000 {built-in method max}
       28    0.000    0.000    0.000    0.000 {built-in method min}
        1    0.000    0.000    0.000    0.000 {built-in method open}
       20    0.000    0.000    0.000    0.000 {built-in method ord}
      214    0.002    0.000    0.004    0.000 {built-in method print}
        9    0.000    0.000    0.000    0.000 {built-in method release_lock}
        6    0.000    0.000    0.000    0.000 {built-in method stat}
      423    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        5    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
      212    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
       10    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        7    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'partition' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'read' of '_io.FileIO' objects}
        1    0.000    0.000    0.000    0.000 {method 'read' of '_io.TextIOWrapper' objects}
        4    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}
       17    0.000    0.000    0.000    0.000 {method 'rpartition' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'rsplit' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}

These informations are very usefull when the execution time matter, since looking at what function/method are called and how many time they are called, you can locate the bottle-neck and try to eliminate them.

For example, trying to find a solution for a problem on Rosalind.info the script need about 3 minute to run (with correct result). After some profiling and optimization, the time needed is down to 39 seconds. The problem in that case was to see that a certain function that I write, while giving correct results, was called way too much time and use too much time.

Optimizing this single function, both in term of numer of calls and execution time, reduce the overall execution time of about 2 minutes.

So the lesson learned here was: profile the code !
Ok, profiling the code is not always usefull, for example if a software execute fast operations in the order of milliseconds and then wait for user input that are in the order of seconds, but it is always a good way to learn something else.

Deploying Flask application

As part of my activity with a modeller group near home, I maintain the site of the group (www.footy.it, italian only) and someone suggested that it should be nice to have a list of yacht that we own/use. So flotta.footy.it was born.

The site itself is a very simple interface to search for someone that has a footy yacht near you, but it is not this the point of the article. It is developed using the Flask python micro framework and mongodb database.

After some weeks of working in the free time, I come up with a pretty decent site, at least for what it need to do, and I have the problem to deploy it, using nginx as webserver. After a bit of research, I fount that a simple way to deploy the site was to use this configuration:

#
# Flask application
#
server {
        listen       80;
        server_name  [your_site];
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $proxy_host;
        proxy_set_header  X-Real-IP  $remote_addr;
    }
}

which basically say to forward all the request on the port 80 for [yoursite] to the proxypass address, where the flask application is running.

After that, I just needed to start the application and test the site. The application was started using the nohup command, but I suppose there are some better ways to do it but I will think at this another time (for now the site is in the testing phase).

Update: the site is no longer active, but the the instruction still stand ;-)

A nightmare called akonadi...

I finally restored all my mailboxes, after an encounter with akonadi.

Background story: I am an kde user from version 3.0 and a kmail user since then. It happen that in about 10 years I accumulated some thousand of mails, about 35000, from various mailing lists and projects I follow/followed plus my personal emails. So aside all the mailing list messages, there are about one thousand personal mail that I cannot lose (yes, I backup my data of course ;-) ) and until kmail 4.6.x I never had a problem while upgrading.

Now enter kmail 4.7 and akonadi and let the nightmare begin.

For some reason the kdepim developer decide that all the personal information of the user should be saved in a database using akonadi. Nice approach, if it only works.

I see the logic of the decision: if I can store all my personal data in one place and I build a layer to manage them, every application that need to know, for example, my email address just need to call this layer and ask for it, without the necessity to reimplement some type of mechanism to store the information. (I think that akonadi is a lot more, but, as I understand, it basically do this) But you must build a layer that work really well.

After the upgrade to kdepim 4.7, the first "stable" release of the suite to use akonadi as core, I find myself with a system where I basically am not able to use kmail in the way I used to do, run it after the login and close it just before the shutdown, and luckily I just use kmail and not for work related projects.

There are a number of problems I run into, but these seems to me the worst:

  • after the upgrade from kmail 4.6.x and the conversion of the data, kmail itself crash on start. Why ? No reason, or better, wrong conversion
  • after recreating the akonadi db, things seems better, the mails are where they are supposed to be and I can read them, but wait, why the system is so sloppy ? After running "top" for 10 seconds the reason is clear: the load average is between 3 and 4, main reason the akonadi agents.
  • why akonadi download the mail even if kmail is not running ? Answer: because the pop3 agents of akonadi, by default, are active also if kmail is not running. The feature is a good idea, but IMHO it must be enabled by the user, not to be enabled by default and without telling it to the user
  • where are saved the mails ? Why it decide that is must be a certain directory that must not be visible to the users ? (something like .local/share/.local-mail-folder/.mailbox) How can I backup my mails if I don't know where they are ? Let the user decide where to put his mails is a so bad idea ?
  • ok, I deleted the old db, copied the mail in a backup folder and switched to posgresql, a better database. Now it is time to import them, but wait, where are now my emails ? Answer: is some hidden directory.
  • I recreated again my db, configured a correct local mail folder agent, but where are the mails ? Why the local agent cannot just read the files in the directory ?
  • after that, I finally imported the old mails, using a second local agent, and now I'd like put every mail in the appropriate mailboxes, right ? Wrong, just filtering about a thousand messages imply about 10 minutes of work and not always the process ends. Let imagine how many time if I filter all my mails. And just for comparison, Claws mail filter the same amount of messages with the same filter rules in about 1 minute. This can also be a problem of kmail itself, since it seems that it has something to do with the message list refreshing for each message moved
  • why akonadi need about 20 or more database connection also if there are just 5 or 6 agent configured, some of which are not running ?

These are just the worst, as I said, but there are a number of minor problems that I don't write down that are really annoying (one for all: after the conversion, kmail does not send mail anymore, until the agent are recreated)

Now, I am not in the position to tell anyone that he did and orrible work, of course the KDEpim developers are far better than me and they do something, nor I have a solution or the time to work out a solution, but reading forums give the impression that the kdepim developers made the same mistake the kde developers done with the release of version 4.0 of KDE, they release a software stack that is far from being stable and usable on the day-by-day work. (Look, after years where I was able to run kmail and make my work with it open, often on low-end machines, I find intolerable that on a decent machine I cannot do it anymore)

The downside of this "akonadi nightmare" are that I temporarily switch to Claws mail, loosing the KDE integration, and I loose about 2 weeks of my spare time to try to work out a decent solution and recover the mess.