Rewrite of wxCatalogue for GTK
27 March 2021Almost from the start a major problem I had with doing electronics was keeping track of components I had sourced so I wrote wxCatalogue to help me with record what I had and where, and a year later I rewrote it to take account of how I used the program in practice. However a major part of wxCatalogue was using drap-and-drop to arrange component into hierarchical categories, and this experience was being ruined due to issues with the wxDataViewCtrl widget it was based around. In short visual hints of where a row is dropped were not being reflected in what the application was receiving. I did not think the deficiencies with what for this application is the essential component would be fixed on any reasonable time-scale and as a result I decided to reimplement the application using GTK.
Since wxCatalogue is not an appropriate name for something that does not actually use wxWidgets, I looked for a new name that does not make reference to the underlying toolkit and settled on Leichreonach. It started out as a straight clone but later on it diverged due to different selection of controls in GTK versus wxWidgets. The source code is available from my Bitbucket account.
Brief history of wxCatalogue
The first version of wxCatalogue was as much an exercise in getting back into Python and wxPython as producing a useful tool, since I had not used either to any real extent for years and this resulted in a few implementation decisions that were done for initial expedience and would quickly become sources of pain. From the outset I decided to have a tree-based classification system but otherwise had little idea from the outset of how to present the other information. The result was functional and proved the central idea of classification but its inflexibilities soon became apparent.
Later on I came across the data-view control which was a mix between a tree and a list, and felt it would be a much better interface that would allow for much quicker eye-ball scanning so I started on a new version that was based around it. At the same time I also decided to switch to using an SQL database back-end because I wanted the ability to easily manipulate the data and show it in alternative forms such as grouping by storage location, as well as target the interface much more at how I used the tool in practice. I made a start using Python 3 and wxPython Phoenix but later switched to C++ because of some features that had just made it into wxWidgets but would probably not filter down to wxPython for some time.
The drag-and-drop support never felt quite right and would remain a persistent embarrassment that was a demotivation to work on its other areas. It was still very useful as 95% of the time I was using it to look up information rather than record it, but in the end got fed up of waiting for wxWidgets to sort itself out.
The role of wxWidgets
I think I first heard of wxWidgets back in 2000 when it was still known as wxWindows but the earliest actual use of it that I remember was writing a long-abandoned Lighttpd configuration tool using wxPython. It was 2010 when I really took to wxWidgets during the building of a prototype video transmission product, where for one reason or another I decided to rewrite an interface I initially made using GTK2, and found wxWidgets so much nicer for the task. I would then use it for one or two Windows-based internal tools, one of which was possibly a licence key generator. I did try various other tool-kits over the years since but in the long-run wxWidgets is the one I kept coming back to, especially after about 2014 or 2015 when it was clear that I probably would not use either the Win32 or WinCE APIs again.I switched from wxPython to the underlying wxWidgets because of what I viewed as the lag time before changes that had just landed in the latter would appear in the former, and it is the only serious C++ program I have written in something like 20 years. By this point portability was not a real aim, not least because wxDataViewCtrl seemed to be broken under Windows anyway, although I did see merit in the way wxWidgets did at least hide what I perceived as some of GTK's grubbiness. In any case wxWidgets and wxFormBuilder were more than enough for anything I was doing. The trouble was that while wxWidgets was fixing a lot of issues, wxDataViewCtrl seemed to be an area of relative neglect and I had waited long enough for it to be fixed. I did make a half-hearted effort of seeing if I could fix it myself but in the end I decided for the same effort I may as well just use the underlying GTK controls directly. The rest is basically history and having written a proof-of-concept of the drag-and-drop interface I actually wanted, it was not long before I decided to do a complete rewrite using GTK.
The rewrite in GTK
When I wrote wxCatalogue v2.0 I made many changes based on actual experience, and as a result the GUI itself was spot-on so the reimplementation using GTK started out as more or less a clone of it. Initially the rewrite was an experiment to see whether GTK's TreeView could be made to do what I wanted it to, and this then evolved into a complete replacement. However the underlying implementation involved some quite significant changes, some of which were required due to differences in API, but most were based in the experience from the previous wxWidgets-based versions.
There are a few things I still want to polish off, such as the choice of icons and some of the pop-up alert dialogs, but functionally Leichreonach is at the stage where I can start using it instead of wxCatalogue.
Database structure
While the structure of the classification tree was specified by the categories table, items from the separate components table are also inserted into this tree. With the wxWidgets implementation where there is just get parent and get children functions where the latter always returns a list of all child nodes this is not a major problem, but GTK instead has get-previous & get-next for fetching siblings, and a get-nth-child for child nodes. GTK also requires conversion between a node reference object and a path which is basically a list of indices from the root node. Having got things working with just the categories table I soon realised that also pulling in from the components table would introduce a lot of edge cases, so I decided that I would replace the categories table with a nodes table that also contained components. Since this broke database compatibility I did consider taking the opportunity to make other changes to database schema, such ascomponent to part and idManufact to idManu, but in the end decided to leave things as-is since I saw nothing to gain.
I even looked at reversing the restriction on stock values but in practice free-form values were something I did not miss.
Database business logic
In both previous versions there was extensinve use ofSELECT .. IF id IS NULL followed by SELECT .. IF id=? for the non-NULL case, rather than using the single SELECT .. IF id IS ? — this was due to finding out that the former does not work if the parameter passed to SQLite is None but not realising that “IS” is a token that had all the functionality of “=” rather than IS NULL being a specific sequence.
Dealing with these cases separately meant a lot of extra code and complexity.
In addition much greater use was made of helper functions to reduce the amount of boiler-plate SQLite code, and with the exception of the Gtk.TreeModel for the main display all database access is wrapped up within a database object.
C++ vs. Python
What originally took something like 2-3 months of work to create was reimplemented in the space of about a week, and ended up taking about half amount of program code in terms of lines, and a third in terms of characters. While some of this is due to reimplementation requiring very little time for things like design and architecture, a significant part of this is down to Python being a substantially better language for the type of data manipulation that needs to be done in the background. For instance, consider this piece of C++ thats runs through an iterator:for(wxDataViewItemArray::iterator iterChild = listChilds.begin(); iterChild != listChilds.end(); iterChild++) { ...
I would be taking guesses as to why C++ does things this way, but the equivilent Python snippet would be like the following, and the difference is stark:
for iterChild in listChilds: ...
Much of the time iterChild will be a tuple that needs manipulation, but since C++ basically does not support natively what takes a few characters in Python takes a few lines in C++.
Then there is no need to deal with explicit containers such as wxVariant, which to be fair to wxWidgets is done away with in wxPython.
This is partly why it was a significant length of time before conversion Python scripts kept alongside were ported to C++ and integrated into the main application, and I have doubts whether I will write another C++ program of this size.
I think wxCatalogue may well be the only signficiant C++ program I have written in the last 20 years.
JSON
The JSON export in wxCatalogue v2 was more of a database dump, since it retained the database id numbers rather than constructing proper human-readable records. Yes it takes up less space using indices than fully spelled out names, and an eyeballing did reveal some mis-imported entries that had data in the wrong field, but the importer has to remap them anyway so there is not much to show for the loss of readability especially since space is not really a concern here. In Python JSON conversion is basically a one-liner between text and Python-native data-structures, whereas in C++ it requires a lot of work which is why JSON import never made it in, so this time round I am much more likley to add data-transfer features.New impression of GTK
Finding information about GTK is a headache because you end up having to Google search for the widget name, which usually brings up either the underlying C-based API or the Python2 PyGTK which uses a different naming scheme for its constants, and since issues with documentation is something I noted in the past this is a bit of a disappointment. Things might be different had I used the bleeding-edge Glade 3.38.1 alongside GTK 4.0.2 but the system I mostly used for writing ElecCatalogue had Glade 3.22.1 and GTK3, and what I have seen of them does not quite convince me that I should invest in switching to them from wxFormBuilder and wxWidgets. Quite a few of wxWidgets' controls I was expecting including ones wxCatalogue used are actually wxWidgets-specific rather than being built on top of GTK controls, and overall drag-drop support is the only thing that GTK really bought to the table.Having said that I got all the functionality I wanted working and all within a fairly short space of time so by any measure switching to it has been a success for this project. I am planning on doing a partial port of the Python code to C as it was the C-based API that was the target of my ire when I developed my prejudices against GTK, and I really need to see how things really are these days. However there needs to be an actual reason to use GTK in favour of wxWidgets as a general preference but since it is not often that I write programs with a graphical interface it is hard to come up with any concrete criteria for choosing one tool-kit over another.