Some new activity

Following an effort to minimize the systems I currently maintain (git, mail, www and so on) I am beginning to migrate some services from my own host to some hosted solution.

The first service to face a restructuration is git, with all my private repositories hosted on git.grys.it moved to GitLab, which offer free private repositories.
These private repositories are only the first ones moved, other will follow.

git.grys.it will not die anyway, some really personal repositories will stay on it, at least for some more time.

How to lose 5 year of life: 2° try...

The first try was when I applied a wrong configuration to grub and thereafter the laptop did not boot anymore (ok, it was some years ago...)
This time, after what seemed a normal update of my laptop, which is now running Archlinux, and a concurrent installation of cutegram, I found that I was unable do anything on the machine and I was basically locked out from my machine. Login from console give "login error" even before asking for the password and even with root. SSH did not work and also sudo or su did not work.

After a bit od digging, I was able to reboot to a shell rescue using systemd, I just needed to add the option systemd.unit=rescue.target to the kernel command line.

Once logged in, I begun to look for the problem. The first suspect was cutegram, but it turn out innocent.

After a bit of analisys of the log, I found this error:
Jun 17 23:35:05 galactica login[30384]: PAM adding faulty module: /usr/lib/security/pam_unix.so Jun 17 23:35:01 galactica login[30379]: PAM adding faulty module: /usr/lib/security/pam_access.so
which was followed by
PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libtirpc.so.1: undefined symbol: __rpc_get_default_domain

The basic problem is that I was sure that I had not updated anything related to PAM or friends. The second error, anyway, directed me to the kernelof the problem: for some reason, the libtirpc library was the wrong version (also if I still don't know why).

The first try to solve the problem was to try to run pacman -S libtirpc, which oddly enough update the library: after that the system restarted to work properly.

I still don't know what happened and I am writing this to maybe help someone else. My impression was that it was an unfortunate update timing (since I use a mirror) which for some combination of factor completed my update but at the same time not all the packages in the mirror was at the correct version.

STL Export on Bulbcalculator

BulbCalculator (in HEAD) has now the export to STL file at least as Ascii. This follow a request from a user to be able to print a model with a 3D printer.
Using this request as starting point I added some features that are likely to be used to mill the model, so the STL can also be exported as half model.

There is, for now, a limitation in this since only one half of the model is saved to the STL file.
If you have a model perfectly symmetrical you just have to mill two times the object and you have a complete bulb. Conversely, if you have an asymmetric model (I presume on the vertical plane) this leads to a small amount of work on the CAD/CAM side since you have to generate a mirrored object.

This limitation will be eliminated in a next version, but for now you should keep it in mind if you plan to print (or mill) a bulb designed with BulbCalculator.

Two new features for BulbCalculator

Following a request from a BulbCalculator user, I started the work on implementing the export of the bulb into a STL model file, to be able to print a model with a 3D printer (or to be able to mill the bulb with a milling machine :-) ).

I also started to implemente the possibility to add a hole for the keel, which is somewhat tricky since for any material you remove, you should update the bulb dimension to take care of the difference.

When I started to work on these two features, I realized thatI needed to do some preliminary works that have, as a result, optimizede the code, especially the code that draw the 3D view of the model.

In the end I now have a more real wireframe view, while I added two other view modes: triangles and a real surface. The triangles view was a preliminary work to be able to create the STL file, the other one is just for fun ;-)

Bulbcalculator ported to QT5

After some work, Bulbcalculator is finally ported to QT5, with the future 2.2.0 version. While I am working on this, I also decided to implement some other features, firs of all a recent project list and the possibility to setup the page size, like some very well known CAD programs.

I have not a date to release the 2.2.0 version, since I have not so much time to work on it, but the work is slowly going on.

Working on stl-tools

Stl-tools get some work in the last couple of days which will be the basis for future development.

The main point I nailed out is to have a structure that can support commands that needs different number of parameters.

The old implementation needs to specify the axis for all the operations, but while adding the new operation stubs, I realize that not all of the operations I'd like to implement need the axis specification and, more important, not all of them refer to an axis.

With this in mind I removed the -x option and update the mirror operation to reflect this. The new method is based on a structure (technically a python dict) which hase a "name" , which is the operation itself, and a nested structure (another python dict) which contain all the couples option-value for the operation. This give the required liberty to implement every operation without the risk to touch every part of the code every time I will need to have another option for a new operation.

While working on this, I also corrected the mirror operation definition so now the it is defined to refer to a plane instead of an axis, which is way too more logical.

The last thing done is to move all the help related function to specific class. With this I hope to have a clearer code

stl-tools operation: mirror

This want to be the first of a series of posts that will explain the operation that can be performed by stl-tools

The first, and the one which inspired me to write the software, is mirror

This was the first since I originally write stl-mirror (which later became stl-tools) to mirror a STL file generated from FREE!Ship without the necessity to use a full 3D CAD/CAM program.

Sintax:
stl-tools -i source_file.stl --axis=x|y|z [-o dest_file.stl] mirror

This operation mirror (what a surprise ;-) )the object given in the source_file along one of it's axis (X, Y or Z), writing it to a new file, so also if something is wrong, the original is not lost.

If the -o parameter is omitted, the output file will be named output[sourcefile].stl

On upgrading server software

Short story: when upgrading a production system, always check the release notes of the new version.

Long story.

After the first upgrade of the server following the release of the new Debian stable, I was no more able to send e-mail from my home pc, but I am still able to send from the server itself, which is running Debian testing, so in the following days of the Debian release, testing got a lot of upgrade as usual.

This upgrade also boost postfix to the 2.1.0 release.

Sending mail from my home pc, then result in the error:

May 11 09:28:22 localhost postfix/smtpd[17094]: connect from unknown[79.21.142.151]
May 11 09:28:24 localhost postfix/smtpd[17094]: NOQUEUE: reject: RCPT from unknown[79.21.142.151]: 554 5.7.1 
<gianlum@xxxxxxx>: Relay access denied; from=<gian@grys.it> to=<gianlum@xxxxxxxx> proto=ESMTP helo=<galactica>
May 11 09:28:24 localhost postfix/smtpd[17094]: lost connection after RCPT from unknown[79.21.142.151]
May 11 09:28:24 localhost postfix/smtpd[17094]: disconnect from unknown[79.21.142.151]

It turn out that from this release on, to be able to have a relay (from home I send mail using my server and not the provider's one), you should put the client restriction in the directive

smtpd_relay_restrictions

instead of the old

smtpd_recipient_restrictions

as always in the main.cf file

I discovered this the hard way, since I was in a hurry to send a mail, and before I can find this change, I tried a gazillion other solutions and also checked for every blacklists in the case the server and/or my provider nerwork were blocked.
So, lessons learned:

  1. Always read the release notes (or at least, check them for the important software)
  2. Do not try massive upgrade past 10.00 pm or you will end to lose sleep and do not solve anything anyway, at least until the next morning
  3. If you need to send a very urgent email, send it before trying a massive upgrade :-D

Python and profiling

While working on the Rosalind's problems, I discover a nice feature of python, the cProfile module, that produce an output like this:

         5683 function calls (5637 primitive calls) in 0.013 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1012(__init__)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:1018(load_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1025(get_filename)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1030(get_data)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1040(path_stats)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:108(_path_isfile)
        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1257(_path_importer_cache)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1274(_get_loader)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1299(find_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1346(find_loader)
        6    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1446(__enter__)
        6    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1450(__exit__)
        3    0.000    0.000    0.001    0.000 <frozen importlib._bootstrap>:1464(_find_module)
      3/2    0.000    0.000    0.004    0.002 <frozen importlib._bootstrap>:1500(_find_and_load_unlocked)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:154(new_module)
      3/2    0.000    0.000    0.005    0.002 <frozen importlib._bootstrap>:1550(_find_and_load)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1587(_handle_fromlist)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:181(__init__)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:201(acquire)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:226(release)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:266(_get_module_lock)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:280(cb)
      3/2    0.000    0.000    0.003    0.002 <frozen importlib._bootstrap>:305(_call_with_frames_removed)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:34(_relax_case)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:415(cache_from_source)
        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:486(_verbose_message)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:496(set_package_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:509(set_loader_wrapper)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:536(module_for_loader_wrapper)
      5/1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:581(_check_name_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:593(_requires_builtin_wrapper)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:61(_r_long)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:643(find_module)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:654(load_module)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:700(find_module)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:74(_path_join)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:754(_open_registry)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:761(_search_registry)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:776(find_module)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:796(is_package)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:804(_bytes_from_bytecode)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:849(_load_module)
        4    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:86(_path_split)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:934(get_code)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:98(_path_is_mode_type)
        1    0.000    0.000    0.004    0.004 <frozen importlib._bootstrap>:996(load_module)
        1    0.000    0.000    0.000    0.000 codecs.py:238(__init__)
      852    0.001    0.000    0.002    0.000 cp1252.py:18(encode)
        1    0.000    0.000    0.000    0.000 cp1252.py:22(decode)
        1    0.000    0.000    0.000    0.000 functools.py:148(__init__)
        3    0.000    0.000    0.000    0.000 functools.py:152(__hash__)
        1    0.000    0.000    0.000    0.000 functools.py:155(_make_key)
        3    0.000    0.000    0.000    0.000 functools.py:167(<genexpr>)
        1    0.000    0.000    0.003    0.003 functools.py:241(wrapper)
        1    0.000    0.000    0.000    0.000 locale.py:555(getpreferredencoding)
        1    0.000    0.000    0.003    0.003 re.py:212(compile)
        1    0.000    0.000    0.000    0.000 re.py:230(escape)
        1    0.000    0.000    0.003    0.003 re.py:264(_compile)
        1    0.002    0.002    0.013    0.013 ros_kmp.py:3(<module>)
      168    0.000    0.000    0.000    0.000 ros_kmp.py:45(<listcomp>)
      212    0.000    0.000    0.000    0.000 ros_kmp.py:5(Find)
      212    0.001    0.000    0.001    0.000 ros_kmp.py:8(Comp)
        4    0.000    0.000    0.000    0.000 sre_compile.py:178(_compile_charset)
        4    0.000    0.000    0.000    0.000 sre_compile.py:207(_optimize_charset)
        2    0.000    0.000    0.000    0.000 sre_compile.py:258(_mk_bitmap)
     12/1    0.000    0.000    0.001    0.001 sre_compile.py:32(_compile)
        2    0.000    0.000    0.000    0.000 sre_compile.py:357(_simple)
        1    0.000    0.000    0.000    0.000 sre_compile.py:364(_compile_info)
        2    0.000    0.000    0.000    0.000 sre_compile.py:470(isstring)
        1    0.000    0.000    0.001    0.001 sre_compile.py:473(_code)
        1    0.000    0.000    0.003    0.003 sre_compile.py:488(compile)
       22    0.000    0.000    0.000    0.000 sre_compile.py:51(fixup)
        9    0.000    0.000    0.000    0.000 sre_parse.py:127(__len__)
       36    0.000    0.000    0.000    0.000 sre_parse.py:131(__getitem__)
        2    0.000    0.000    0.000    0.000 sre_parse.py:135(__setitem__)
       14    0.000    0.000    0.000    0.000 sre_parse.py:139(append)
     14/3    0.000    0.000    0.000    0.000 sre_parse.py:141(getwidth)
        1    0.000    0.000    0.000    0.000 sre_parse.py:179(__init__)
      310    0.001    0.000    0.001    0.000 sre_parse.py:184(__next)
       52    0.000    0.000    0.000    0.000 sre_parse.py:203(match)
      281    0.000    0.000    0.001    0.000 sre_parse.py:209(get)
        1    0.000    0.000    0.000    0.000 sre_parse.py:222(tell)
        1    0.000    0.000    0.000    0.000 sre_parse.py:224(seek)
       25    0.000    0.000    0.000    0.000 sre_parse.py:227(isident)
        4    0.000    0.000    0.000    0.000 sre_parse.py:233(isname)
        2    0.000    0.000    0.000    0.000 sre_parse.py:284(_escape)
      6/1    0.000    0.000    0.002    0.002 sre_parse.py:340(_parse_sub)
      9/1    0.000    0.000    0.002    0.002 sre_parse.py:418(_parse)
        1    0.000    0.000    0.000    0.000 sre_parse.py:68(__init__)
        1    0.000    0.000    0.000    0.000 sre_parse.py:702(fix_flags)
        1    0.000    0.000    0.002    0.002 sre_parse.py:714(parse)
        4    0.000    0.000    0.000    0.000 sre_parse.py:73(opengroup)
        4    0.000    0.000    0.000    0.000 sre_parse.py:84(closegroup)
       12    0.000    0.000    0.000    0.000 sre_parse.py:91(__init__)
        1    0.000    0.000    0.003    0.003 string.py:15(<module>)
        1    0.000    0.000    0.000    0.000 string.py:162(Formatter)
        1    0.000    0.000    0.000    0.000 string.py:51(_TemplateMetaclass)
        1    0.000    0.000    0.003    0.003 string.py:61(__init__)
        1    0.000    0.000    0.000    0.000 string.py:73(Template)
        2    0.000    0.000    0.000    0.000 {built-in method OpenKey}
        3    0.000    0.000    0.003    0.001 {built-in method __build_class__}
        1    0.000    0.000    0.000    0.000 {built-in method _fix_co_filename}
        1    0.000    0.000    0.000    0.000 {built-in method _getdefaultlocale}
        6    0.000    0.000    0.000    0.000 {built-in method acquire_lock}
        6    0.000    0.000    0.000    0.000 {built-in method allocate_lock}
        1    0.000    0.000    0.000    0.000 {built-in method charmap_decode}
      852    0.001    0.000    0.001    0.000 {built-in method charmap_encode}
        1    0.000    0.000    0.000    0.000 {built-in method compile}
      2/1    0.000    0.000    0.013    0.013 {built-in method exec}
        6    0.000    0.000    0.000    0.000 {built-in method get_ident}
        5    0.000    0.000    0.000    0.000 {built-in method getattr}
       26    0.000    0.000    0.000    0.000 {built-in method getlower}
       13    0.000    0.000    0.000    0.000 {built-in method hasattr}
        1    0.000    0.000    0.000    0.000 {built-in method hash}
        2    0.000    0.000    0.000    0.000 {built-in method init_builtin}
        3    0.000    0.000    0.000    0.000 {built-in method is_builtin}
        1    0.000    0.000    0.000    0.000 {built-in method is_frozen}
       43    0.000    0.000    0.000    0.000 {built-in method isinstance}
1322/1319    0.000    0.000    0.000    0.000 {built-in method len}
        1    0.000    0.000    0.000    0.000 {built-in method loads}
        4    0.000    0.000    0.000    0.000 {built-in method max}
       28    0.000    0.000    0.000    0.000 {built-in method min}
        1    0.000    0.000    0.000    0.000 {built-in method open}
       20    0.000    0.000    0.000    0.000 {built-in method ord}
      214    0.002    0.000    0.004    0.000 {built-in method print}
        9    0.000    0.000    0.000    0.000 {built-in method release_lock}
        6    0.000    0.000    0.000    0.000 {built-in method stat}
      423    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        5    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
      212    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
       10    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        7    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'partition' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'read' of '_io.FileIO' objects}
        1    0.000    0.000    0.000    0.000 {method 'read' of '_io.TextIOWrapper' objects}
        4    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}
       17    0.000    0.000    0.000    0.000 {method 'rpartition' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'rsplit' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}

These informations are very usefull when the execution time matter, since looking at what function/method are called and how many time they are called, you can locate the bottle-neck and try to eliminate them.

For example, trying to find a solution for a problem on Rosalind.info the script need about 3 minute to run (with correct result). After some profiling and optimization, the time needed is down to 39 seconds. The problem in that case was to see that a certain function that I write, while giving correct results, was called way too much time and use too much time.

Optimizing this single function, both in term of numer of calls and execution time, reduce the overall execution time of about 2 minutes.

So the lesson learned here was: profile the code !
Ok, profiling the code is not always usefull, for example if a software execute fast operations in the order of milliseconds and then wait for user input that are in the order of seconds, but it is always a good way to learn something else.

Deploying Flask application

As part of my activity with a modeller group near home, I maintain the site of the group (www.footy.it, italian only) and someone suggested that it should be nice to have a list of yacht that we own/use. So flotta.footy.it was born.

The site itself is a very simple interface to search for someone that has a footy yacht near you, but it is not this the point of the article. It is developed using the Flask python micro framework and mongodb database.

After some weeks of working in the free time, I come up with a pretty decent site, at least for what it need to do, and I have the problem to deploy it, using nginx as webserver. After a bit of research, I fount that a simple way to deploy the site was to use this configuration:

#
# Flask application
#
server {
        listen       80;
        server_name  [your_site];
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $proxy_host;
        proxy_set_header  X-Real-IP  $remote_addr;
    }
}

which basically say to forward all the request on the port 80 for [yoursite] to the proxypass address, where the flask application is running.

After that, I just needed to start the application and test the site. The application was started using the nohup command, but I suppose there are some better ways to do it but I will think at this another time (for now the site is in the testing phase).

Update: the site is no longer active, but the the instruction still stand ;-)