Two new features for BulbCalculator

Following a request from a BulbCalculator user, I started the work on implementing the export of the bulb into a STL model file, to be able to print a model with a 3D printer (or to be able to mill the bulb with a milling machine :-) ).

I also started to develop the possibility to add a hole for the keel, which is somewhat tricky since for any material you remove, you should update the bulb dimension to take care of the difference.

When I started to work on these two features, I realized thatI needed to do some preliminary works that have, as a result, optimized the code, especially the code that draw the 3D view of the model.

In the end I now have a more real wireframe view, while I added two other view modes: triangles and a real surface. The triangles view was a preliminary work to be able to create the STL file, the other one is just for fun ;-)

Two new features for BulbCalcularor

Following a request from a BulbCalculator user, I started the work on implementing the export of the bulb into a STL model file, to be able to print a model with a 3D printer (or to be able to mill the bulb with a milling machine :-) ).

I also started to implemente the possibility to add a hole for the keel, which is somewhat tricky since for any material you remove, you should update the bulb dimension to take care of the difference.

When I started to work on these two features, I realized thatI needed to do some preliminary works that have, as a result, optimizede the code, especially the code that draw the 3D view of the model.
In the end I now have a more real wireframe view, while I added two other view modes: triangles and a real surface. The triangles view was a preliminary work to be able to create the STL file, the other one is just for fun ;-)

Working on stl-tools

Stl-tools get some work in the last couple of days which will be the basis for future development.

The main point I nailed out is to have a structure that can support commands that needs different number of parameters.

The old implementation needs to specify the axis for all the operations, but while adding the new operation stubs, I realize that not all of the operations I’d like to implement need the axis specification and, more important, not all of them refer to an axis.

With this in mind I removed the -x option and update the mirror operation to reflect this. The new method is based on a structure (technically a python dict) which hase a “name” , which is the operation itself, and a nested structure (another python dict) which contain all the couples option-value for the operation. This give the required liberty to implement every operation without the risk to touch every part of the code every time I will need to have another option for a new operation.

While working on this, I also corrected the mirror operation definition so now the it is defined to refer to a plane instead of an axis, which is way too more logical.

The last thing done is to move all the help related function to specific class. With this I hope to have a clearer code

stl-tools operation: mirror

This want to be the first of a series of posts that will explain the operation that can be performed by stl-tools

The first, and the one which inspired me to write the software, is mirror

This was the first since I originally write stl-mirror (which later became stl-tools) to mirror a STL file generated from FREE!Ship without the necessity to use a full 3D CAD/CAM program.


Sintax:
stl-tools -i source_file.stl –axis=x|y|z [-o dest_file.stl] mirror

This operation mirror (what a surprise ;-) )the object given in the source_file along one of it’s axis (X, Y or Z), writing it to a new file, so also if something is wrong, the original is not lost.

If the -o parameter is omitted, the output file will be named output_[source_file].stl

Python and profiling

While working on the Rosalind’s problems, I discover a nice feature of python, the cProfile module, that produce an output like this:

         
5683 function calls (5637 primitive calls) in 0.013 seconds
Ordered by: standard name   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.000    0.000    0.000    0.000 :1012(__init__)
    1    0.000    0.000    0.004    0.004 :1018(load_module)
    4    0.000    0.000    0.000    0.000 :1025(get_filename)
    1    0.000    0.000    0.000    0.000 :1030(get_data)
    1    0.000    0.000    0.000    0.000 :1040(path_stats)
    1    0.000    0.000    0.000    0.000 :108(_path_isfile)
    5    0.000    0.000    0.000    0.000 :1257(_path_importer_cache)
    1    0.000    0.000    0.000    0.000 :1274(_get_loader)
    1    0.000    0.000    0.000    0.000 :1299(find_module)
    4    0.000    0.000    0.000    0.000 :1346(find_loader)
    6    0.000    0.000    0.000    0.000 :1446(__enter__)
    6    0.000    0.000    0.000    0.000 :1450(__exit__)
    3    0.000    0.000    0.001    0.000 :1464(_find_module)     
    3/2    0.000    0.000    0.004    0.002 :1500(_find_and_load_unlocked)
    1    0.000    0.000    0.000    0.000 :154(new_module)
    3/2    0.000    0.000    0.005    0.002 :1550(_find_and_load)
    1    0.000    0.000    0.000    0.000 :1587(_handle_fromlist)
    3    0.000    0.000    0.000    0.000 :181(__init__)
    3    0.000    0.000    0.000    0.000 :201(acquire)
    3    0.000    0.000    0.000    0.000 :226(release)
    3    0.000    0.000    0.000    0.000 :266(_get_module_lock)
    3    0.000    0.000    0.000    0.000 :280(cb)
    3/2    0.000    0.000    0.003    0.002 :305(_call_with_frames_removed)
    4    0.000    0.000    0.000    0.000 :34(_relax_case)
    2    0.000    0.000    0.000    0.000 :415(cache_from_source)
    5    0.000    0.000    0.000    0.000 :486(_verbose_message)
    2    0.000    0.000    0.000    0.000 :496(set_package_wrapper)
    2    0.000    0.000    0.000    0.000 :509(set_loader_wrapper)
    1    0.000    0.000    0.004    0.004 :536(module_for_loader_wrapper)
    5/1    0.000    0.000    0.004    0.004 :581(_check_name_wrapper)
    2    0.000    0.000    0.000    0.000 :593(_requires_builtin_wrapper)
    2    0.000    0.000    0.000    0.000 :61(_r_long)
    3    0.000    0.000    0.000    0.000 :643(find_module)
    2    0.000    0.000    0.000    0.000 :654(load_module)
    1    0.000    0.000    0.000    0.000 :700(find_module)
    3    0.000    0.000    0.000    0.000 :74(_path_join)
    1    0.000    0.000    0.000    0.000 :754(_open_registry)
    1    0.000    0.000    0.000    0.000 :761(_search_registry)
    1    0.000    0.000    0.000    0.000 :776(find_module)
    2    0.000    0.000    0.000    0.000 :796(is_package)
    1    0.000    0.000    0.000    0.000 :804(_bytes_from_bytecode)
    1    0.000    0.000    0.004    0.004 :849(_load_module)
    4    0.000    0.000    0.000    0.000 :86(_path_split)
    1    0.000    0.000    0.000    0.000 :934(get_code)
    1    0.000    0.000    0.000    0.000 :98(_path_is_mode_type)
    1    0.000    0.000    0.004    0.004 :996(load_module)
    1    0.000    0.000    0.000    0.000 codecs.py:238(__init__)
    852    0.001    0.000    0.002    0.000 cp1252.py:18(encode)
    1    0.000    0.000    0.000    0.000 cp1252.py:22(decode)
    1    0.000    0.000    0.000    0.000 functools.py:148(__init__)
    3    0.000    0.000    0.000    0.000 functools.py:152(__hash__)
    1    0.000    0.000    0.000    0.000 functools.py:155(_make_key)
    3    0.000    0.000    0.000    0.000 functools.py:167()
    1    0.000    0.000    0.003    0.003 functools.py:241(wrapper)
    1    0.000    0.000    0.000    0.000 locale.py:555(getpreferredencoding)
    1    0.000    0.000    0.003    0.003 re.py:212(compile)
    1    0.000    0.000    0.000    0.000 re.py:230(escape)
    1    0.000    0.000    0.003    0.003 re.py:264(_compile)
    1    0.002    0.002    0.013    0.013 ros_kmp.py:3()
    168    0.000    0.000    0.000    0.000 ros_kmp.py:45()
    212    0.000    0.000    0.000    0.000 ros_kmp.py:5(Find)
    212    0.001    0.000    0.001    0.000 ros_kmp.py:8(Comp)
    4    0.000    0.000    0.000    0.000 sre_compile.py:178(_compile_charset)
    4    0.000    0.000    0.000    0.000 sre_compile.py:207(_optimize_charset)
    2    0.000    0.000    0.000    0.000 sre_compile.py:258(_mk_bitmap)
    ù12/1    0.000    0.000    0.001    0.001 sre_compile.py:32(_compile)
    2    0.000    0.000    0.000    0.000 sre_compile.py:357(_simple)
    1    0.000    0.000    0.000    0.000 sre_compile.py:364(_compile_info)
    2    0.000    0.000    0.000    0.000 sre_compile.py:470(isstring)
    1    0.000    0.000    0.001    0.001 sre_compile.py:473(_code)
    1    0.000    0.000    0.003    0.003 sre_compile.py:488(compile)
    22    0.000    0.000    0.000    0.000 sre_compile.py:51(fixup)
    9    0.000    0.000    0.000    0.000 sre_parse.py:127(__len__)
    36    0.000    0.000    0.000    0.000 sre_parse.py:131(__getitem__)
    2    0.000    0.000    0.000    0.000 sre_parse.py:135(__setitem__)
    14    0.000    0.000    0.000    0.000 sre_parse.py:139(append)
    14/3    0.000    0.000    0.000    0.000 sre_parse.py:141(getwidth)
    1    0.000    0.000    0.000    0.000 sre_parse.py:179(__init__)
    310    0.001    0.000    0.001    0.000 sre_parse.py:184(__next)
    52    0.000    0.000    0.000    0.000 sre_parse.py:203(match)
    281    0.000    0.000    0.001    0.000 sre_parse.py:209(get)
    1    0.000    0.000    0.000    0.000 sre_parse.py:222(tell)
    1    0.000    0.000    0.000    0.000 sre_parse.py:224(seek)
    25    0.000    0.000    0.000    0.000 sre_parse.py:227(isident)
    4    0.000    0.000    0.000    0.000 sre_parse.py:233(isname)
    2    0.000    0.000    0.000    0.000 sre_parse.py:284(_escape)
    6/1    0.000    0.000    0.002    0.002 sre_parse.py:340(_parse_sub)
    9/1    0.000    0.000    0.002    0.002 sre_parse.py:418(_parse)
      1    0.000    0.000    0.000    0.000 sre_parse.py:68(__init__)
      1    0.000    0.000    0.000    0.000 sre_parse.py:702(fix_flags)
      1    0.000    0.000    0.002    0.002 sre_parse.py:714(parse)
      4    0.000    0.000    0.000    0.000 sre_parse.py:73(opengroup)
      4    0.000    0.000    0.000    0.000 sre_parse.py:84(closegroup)
     12    0.000    0.000    0.000    0.000 sre_parse.py:91(__init__)
      1    0.000    0.000    0.003    0.003 string.py:15()
      1    0.000    0.000    0.000    0.000 string.py:162(Formatter)
      1    0.000    0.000    0.000    0.000 string.py:51(_TemplateMetaclass)
      1    0.000    0.000    0.003    0.003 string.py:61(__init__)
      1    0.000    0.000    0.000    0.000 string.py:73(Template)
      2    0.000    0.000    0.000    0.000 {built-in method OpenKey}
      3    0.000    0.000    0.003    0.001 {built-in method __build_class__}
      1    0.000    0.000    0.000    0.000 {built-in method _fix_co_filename}
      1    0.000    0.000    0.000    0.000 {built-in method _getdefaultlocale}
      6    0.000    0.000    0.000    0.000 {built-in method acquire_lock}
      6    0.000    0.000    0.000    0.000 {built-in method allocate_lock}
      1    0.000    0.000    0.000    0.000 {built-in method charmap_decode}
    852    0.001    0.000    0.001    0.000 {built-in method charmap_encode}
      1    0.000    0.000    0.000    0.000 {built-in method compile}
    2/1    0.000    0.000    0.013    0.013 {built-in method exec}
      6    0.000    0.000    0.000    0.000 {built-in method get_ident}
      5    0.000    0.000    0.000    0.000 {built-in method getattr}
     26    0.000    0.000    0.000    0.000 {built-in method getlower}
     13    0.000    0.000    0.000    0.000 {built-in method hasattr}
      1    0.000    0.000    0.000    0.000 {built-in method hash}
      2    0.000    0.000    0.000    0.000 {built-in method init_builtin}
      3    0.000    0.000    0.000    0.000 {built-in method is_builtin}
      1    0.000    0.000    0.000    0.000 {built-in method is_frozen}
     43    0.000    0.000    0.000    0.000 {built-in method isinstance}
     1322/1319    0.000    0.000    0.000    0.000 {built-in method len}
      1    0.000    0.000    0.000    0.000 {built-in method loads}
      4    0.000    0.000    0.000    0.000 {built-in method max}
     28    0.000    0.000    0.000    0.000 {built-in method min}
      1    0.000    0.000    0.000    0.000 {built-in method open}
     20    0.000    0.000    0.000    0.000 {built-in method ord}
    214    0.002    0.000    0.004    0.000 {built-in method print}
      9    0.000    0.000    0.000    0.000 {built-in method release_lock}
      6    0.000    0.000    0.000    0.000 {built-in method stat}
    423    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      5    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
    212    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
      1    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
     10    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      1    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
      7    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
      2    0.000    0.000    0.000    0.000 {method 'partition' of 'str' objects}
      1    0.000    0.000    0.000    0.000 {method 'read' of '_io.FileIO' objects}
      1    0.000    0.000    0.000    0.000 {method 'read' of '_io.TextIOWrapper' objects}
      4    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}
     17    0.000    0.000    0.000    0.000 {method 'rpartition' of 'str' objects}
      2    0.000    0.000    0.000    0.000 {method 'rsplit' of 'str' objects}
      1    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}

These informations are very usefull when the execution time matter, since looking at what function/method are called and how many time they are called, you can locate the bottle-neck and try to eliminate them.

For example,  trying to find a  solution for a problem on Rosalind.info the script need about 3 minute to run (with correct result). After some profiling and optimization, the time needed is down to 39 seconds. The problem in that case was to see that a certain function that I write, while giving correct results,  was called way too much time and use too much time.

Optimizing this single function, both in term of numer of calls and execution time, reduce the overall execution time of about 2 minutes.

So the lesson learned here was: profile the code !  
Ok, profiling the code is not always usefull, for example if a software execute fast operations in the order of milliseconds and then wait for user input that are in the order of seconds, but it is always a good way to learn something else.

Deploying Flask application

As part of my activity with a modeller group near home, I maintain the site of the group (www.footy.it, italian only) and someone suggested that it should be nice to have a list of yacht that we own/use. So flotta.footy.it was born.\n

The site itself is a very simple interface to search for someone that has a footy yacht near you, but it is not this the point of the article. It is developed using the Flask python micro framework and mongodb database.

After some weeks of working in the free time, I come up with a pretty decent site, at least for what it need to do, and I have the problem to deploy it, using nginx as webserver. After a bit of research, I fount that a simple way to deploy the site was to use this configuration:

#
# Flask application
#
server {
          listen       80;
          server_name  [your_site];
          location / {
            proxy_pass http://127.0.0.1:5000;
            proxy_set_header Host $proxy_host;
            proxy_set_header  X-Real-IP  $remote_addr;
          }
}

which basically say to forward all the request on the port 80 for [your_site] to the proxy_pass address, where the flask application is running.
After that, I just needed to start the application and test the site. The application was started using the nohup command, but I suppose there are some better ways to do it but I will think at this another time (for now the site is in the testing phase).

Update: the site is no longer active, but the the instruction still stand ;-)

A nightmare called akonadi...

I finally restored all my mailboxes, after an encounter with akonadi.

Background story: I am an kde user from version 3.0 and a kmail user since then. It happen that in about 10 years I accumulated some thousand of mails, about 35000, from various mailing lists and projects I follow/followed plus my personal emails. So aside all the mailing list messages, there are about one thousand personal mail that I cannot lose (yes, I backup my data of course ;-) ) and until kmail 4.6.x I never had a problem while upgrading.

Now enter kmail 4.7 and akonadi and let the nightmare begin.

For some reason the kdepim developer decide that all the personal information of the user should be saved in a database using akonadi. Nice approach, if it only works.

I see the logic of the decision: if I can store all my personal data in one place and I build a layer to manage them, every application that need to know, for example, my email address just need to call this layer and ask for it, without the necessity to reimplement some type of mechanism to store the information. (I think that akonadi is a lot more, but, as I understand, it basically do this) But you must build a layer that work really well.

After the upgrade to kdepim 4.7, the first “stable” release of the suite to use akonadi as core, I find myself with a system where I basically am not able to use kmail in the way I used to do, run it after the login and close it just before the shutdown, and luckily I just use kmail and not for work related projects.

There are a number of problems I run into, but these seems to me the worst:


  • after the upgrade from kmail 4.6.x and the conversion of the data, kmail itself crash on start. Why ? No reason, or better, wrong conversion

  • after recreating the akonadi db, things seems better, the mails are where they are supposed to be and I can read them, but wait, why the system is so sloppy ? After running “top” for 10 seconds the reason is clear: the load average is between 3 and 4, main reason the akonadi agents.

  • why akonadi download the mail even if kmail is not running ? Answer: because the pop3 agents of akonadi, by default, are active also if kmail is not running. The feature is a good idea, but IMHO it must be enabled by the user, not to be enabled by default and without telling it to the user

  • where are saved the mails ? Why it decide that is must be a certain directory that must not be visible to the users ? (something like .local/share/.local-mail-folder/.mailbox) How can I backup my mails if I don’t know where they are ? Let the user decide where to put his mails is a so bad idea ?

  • ok, I deleted the old db, copied the mail in a backup folder and switched to posgresql, a better database. Now it is time to import them, but wait, where are now my emails ? Answer: is some hidden directory.

  • I recreated again my db, configured a correct local mail folder agent, but where are the mails ? Why the local agent cannot just read the files in the directory ?

  • after that, I finally imported the old mails, using a second local agent, and now I’d like put every mail in the appropriate mailboxes, right ? Wrong, just filtering about a thousand messages imply about 10 minutes of work and not always the process ends. Let imagine how many time if I filter all my mails. And just for comparison, Claws mail filter the same amount of messages with the same filter rules in about 1 minute. This can also be a problem of kmail itself, since it seems that it has something to do with the message list refreshing for each message moved

  • why akonadi need about 20 or more database connection also if there are just 5 or 6 agent configured, some of which are not running ?


These are just the worst, as I said, but there are a number of minor problems that I don’t write down that are really annoying (one for all: after the conversion, kmail does not send mail anymore, until the agent are recreated)

Now, I am not in the position to tell anyone that he did and orrible work, of course the KDEpim developers are far better than me and they do something, nor I have a solution or the time to work out a solution, but reading forums give the impression that the kdepim developers made the same mistake the kde developers done with the release of version 4.0 of KDE, they release a software stack that is far from being stable and usable on the day-by-day work. (Look, after years where I was able to run kmail and make my work with it open, often on low-end machines, I find intolerable that on a decent machine I cannot do it anymore)

The downside of this “akonadi nightmare” are that I temporarily switch to Claws mail, loosing the KDE integration, and I loose about 2 weeks of my spare time to try to work out a decent solution and recover the mess.

First BugsEverywhere contribution

Let’s start.

My first BE contribution was the port of a patch Chris Ball send me when I asked for the possibility to add a command to list all the target in the repository.

After my mail to the list Chris was kind enough to send me a little patch to implement the option, but this patch was never merged in the repository. So after some time, I ported the patch to the last revision and send it to the list.

My idea was to have a command sintax like

be target list

that show all the target present in the repository, with or without associated bug.

The patch was accepted, with one minor modification to the sintax so now the command is

be target --list

since this is now the new sintax of the be commands

Template system for BugsEverywhere completed

With the last commit (and push) I’ve completed the html export of BugEverywhere

After the suggestions from W.Trevor King (who also give me a good insight about bzr workflow) I added two other options for the command (–export-template and –export-dir-template).

The first one simply export the default template in the default directory, the second one let you to specify a custom directory as target for the export. This default template can be used as starting point to write a custom template for the export.

Trevor also added two options to let you to personalize the title and the index header of the export (–title and –index-header).

At this point, if no bugs are found, I think the export is complete.

New template system for be html

Just pushed to my branch the new template system for the ‘html’ command of BugEverywhere.

Now there is a default template hardcoded in the command, and a new option (-t|–template-dir template_directory) that can be used to pass a different template for the html export.

The template itself is a simplified version of the previuos one, and with the -t option is no more necessary to modify the source code to modify the look of the output, but just put in one directory 4 files:


  • style.css

  • index_file.tpl

  • detail_file.tpl

  • comment_section.tpl


The tpl files are just normal html files, and of course the css file is a normal css file ;-)

As a bonus, there is also a verbose option to have a complete output of the command execution, very useful if the command is executed with a cron job.