Porting wxLED to Python 3
01 June 2019I have been using Python for about a decade, and with Python 2 due to be end-of-life January 2020 I needed to decide whether to invest time and effort in Python 3 or to abandon Python entirely for something else. All indications are that the support infrastructure for Python 2 such as PIP is also being shut down, so sticking with an unsupported Python 2 will not really be an option. Differences between Python 2 and Python 3 have been a major pain-point for years, to the extent that it is only in the last 2-3 years that Python 3 has actually overtaken Python 2 in terms of popularity. As an experiment I decided to convert my recently-released wxLED as a first real taste of Python 3.
While porting wxLED was on the whole a good experience, it did not cover a major use-case of mine: Protocol prototyping. Mercifully in Python 3.5 a new feature I happened to spot is percent formatting for bytes/bytearrays, which to me made the difference between giving Python 3 a chance and investing in an alternative. In the absence of any formatting functionality — either percent-formatting or .format() — having to convert between byte-sequences and Unicode would have been unacceptably painful for protocol work.
Slackware support
Slackware 14.2 does not include Python 3, quite likely because the decision on what to have as “core” packages may have been taken as long ago as 2015, and back then rightly or wrongly Python 3 would simply not have been considered a priority. I suspect that the AlienBob repository only started carrying it recently, as when seeing if I could just copy a virtualenv install over from my desktop I noticed that the laptop had a SlackBuild install of Python 3.5.2 that would have dated from the latter part of 2016. The Slackware-current development branch does include Python 3, so it will be bundled with Slackware 15 when the latter is finally released.History of using Python
At some point towards the end of my Ph.D, most likely in early-2009 as that would correspond with when I wrote my now-abandoned LyteCFG, I decided that I would concentrate on two programming languages: C and Python. C was always going to be one of them and Python was chosen for both object-orientation and scripting. By this time I would have decided Perl was too messy for my liking so Python's clean nature was attractive, and I was in two minds whether to get serious with C++. Although Python 3.0 came out in December 2008 it was botched and from what I gather Python 3.1 released in February 2009 also had issues, so Python 3 would not have been on my radar. The first Python 3 release I remember seeing was Python 3.2 which was released February 2011.Professional use
I used Python for some experimental and testing purposes in late-2010 and 2011 but the vast bulk of my professional use of Python was in 2013-2015 when I was an official Python Developer and the company this was with specifically opted to use Python 2 rather than Python 3. I cannot remember the specifics of the supposed problems it caused, but it was due to Python 3 using Unicode — my suspicion is that since the huge proportion of the code-base was doing stuff with ASCII-based configuration snippets, Unicode was of zero practical benefit but it came with non-zero cost. Whatever the reason Python 2 was what my experience consisted of, and I left that line of work before the big migration from Python 2 to Python 3 had really kicked off in the industry. Having gone back to C programming my professional use of Python in the subsequent years has typically been small scripts making use of Scapy to either fabricate or examine network packets — technically continued commercial use, but one that is on the outer bounds of acceptability to claim.Subsequent personal use
Although I used Python for various odd tasks such as the test script for my I2C-related electronics projects, the lion's share of my Python programming since 2015 has been wxCatalogue and wxLED which were both wxPython-based applications started in the latter part of 2018. I did not realise it at the time but wxPython only got Python 3 support with the pip-based wxPython Phoenix in early-2018, so if I seriously considered using Python 3 at the time I would not have got very far. My Python working-knowledge is clearly not going to be as as sharp compared to when I was using Python full-time, but I did feel that I was referring back to online documentation for information that ought to be second-nature.Possible future professional use
The former job title of Python Developer is a magnet for recruitment agencies but given the follow-up rate it is clear that actual companies take one look and decide otherwise, and in any case I have long decided that the type of projects I want to do as a career are not the ones that are typically advertised as Python roles. It is all about Python 3 these days and considering the experience level of jobs appropriate for the current stage in my career, the technology is not longer really a skill worth emphasising. As with C# the ramp-up I would require is unlikely to be one that a future company would accept.The porting experience
Converting wxLED to Python 3 was a lot quicker and easier than I expected, although this was perhaps to be expected as the huge majority of API usage is wxPython GUI toolkit rather than Python itself. A lot of the “new” way of doing things in Python 3 were back-ported to Python 2.7 which were subsequently considered best-practice to use, so the mass removal of deprecated features did not have as much of an effect as expected. From what I read Python 3.5 took account of what were notably painful for migrations to Python 3, and v3.5 was released after I became a significantly less serious user of Python.Environment setup
A small issue but a little trickery is needed to install a bootstrap PIP and virtualenv for both Python 2 and Python 3. My personal choice is to keep Python 2 as default Python in order to avoid breaking my existing Python programs, and this is the most expedient distribution-independent way of doing it. The paranoid might want to check file hashes.curl -O https://bootstrap.pypa.io/get-pip.py sudo python3 get-pip.py sudo pip3 install virtualenv virtualenv ~/virtenv-py3
To install wxPython into the non-root virtualenv the following commands can be used. This involves building wxPython Phoenix from sources, which is quite quick on my desktop but is a strain for my laptop.
. ~/virtenv-py3/bin/activate pip3 install pathlib2 pip3 install -U wxPython
Print statements
In Python 3print became a built-in function rather than a special key-word, which in practical terms meant that it now needed brackets. A simple change but given how often it is used this is something that takes a lot of getting used to. I used %s/print \(.*\)/print(\1)/gc within GVim to automatically convert all my code to add them, which given that I only used it for debugging purposes, did not require any follow-up fixes.
Integer divisions
In Python 3 unless the explicit rounding-down division operator of// is used rather than the traditional /, a division is automatically converted to a floating-point type, which in the past would have required something like float(x)/float(y) to get. In wxLED the resulting value is almost always used as input to something like range() that does not accept the float type, but in this case a simple %s/ \/ / \/\/ /gc fixed them all up. There is a clear design decision here that is aimed at people using Python for scientific computations at the expense of people like me who are rarely interested in fractions.
Lists replaced with iterable objects
The.keys() and .values() methods of a dictionary now returns iterable objects that does not support indexing rather than a list of values, which I am guessing is done for efficiency reasons as most of the time an array lookup is not required. If indexing is required then a list has to be obtained by using the dictionary as a parameter to list() — this seems part of a larger trend away from using lists as intermediate values.
Removal of file() for opening files
For some reason file() has been replaced with open() for creating/opening a file, which although is a simple change is one that I do not really understand the benefit of as the former is more intuitive as a file object creator. Some reading up on file() versus open() convinces me this was a bad decision since the latter was at one time the deprecated way of opening a file.
Binary data pain-points
In Python 2 strings are sequences of bytes and as a result they are interchangable, but in Python 3 strings are Unicode. There are good motives for this bit when I read the “what doesn't work” part of a Python 2 to Python 3 porting guide, I knew it would be something that for a significant portion of my use-cases could well be a source of significant pain. The example Python 2 code below is somewhat contrived, but it demonstrates how how strings and bytes mix freely:with open("256bytes.dat","rb") as fp: data = fp.read(256) msg = "In: {0}".format( data[0x41:0x44] ) print(msg) hdr = "DATA {0}\r\n".format(10) pkt = hdr + data[0:10]
It is assumed 256bytes.dat is bytes 0x00-0xff in ascending order.
In Python 3 such hard-and-fast mixing is no longer permitted, as bytes and strings have been separated and moving between then requires encode/decode steps. For Python 3 the equivalent code would be:
with open("256bytes.dat","rb") as fp: data = fp.read(256) msg = "In: {0}".format( data[0x41:0x44].decode('utf-8') ) print(msg) hdr = "DATA {0}\r\n".format(10) pkt = bytes(hdr,'utf-8') + data[0:10]
When working with protocols such as RFC2326 which by their nature are a mix of ASCII and binary data, having to do such conversions reminds me why I ditched Java. In the example above it looks like minor inconvenience but when implementing a practical protocol it becomes a major irritation, and when the purpose of using a language in the first place is expedient prototyping work which is then used as a reference for production business logic, such irritation is not acceptable. Thankfully formatting of byte arrays (i.e. ASCII text) was added in Python 3.5 via percent formatting.