python-for-android 0.4 released, now available on PyPI

We’ve just officially released python-for-android 0.4, and pushed it to PyPI for the first time!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

This release is the culmination of all the work over the last year to replace Kivy’s old Android toolchain with something more flexible and useful for other projects. Major features added in this time include the fully Python toolchain itself, support for SDL2 and other bootstraps, (experimental) python3 support via the CrystaX NDK, multiple architecture support, and many general improvements to the backend. Many thanks to all the contributors who have made this possible!

From now on we intend to move to regular versioned releases rather than the previous rolling master branch. Short term targets for the next release include bringing the python3 build up to full functionality and stability, and some argument restructuring to make command line usage simpler and clearer.

As of this release, you can now install python-for-android with simply:

pip install python-for-android

For full instructions and further information, see the python-for-android documentation.

Posted in android, Kivy, Python | Comments Off on python-for-android 0.4 released, now available on PyPI

Kivy at PyCon

Jacob Kovac, Kivy core developer and creator of the KivEnt game engine, is at PyCon 2016.

Click here or see below to watch his talk, Revitalizing Python Game Development: Packaging, Performance, and Platforms.

Posted in kivent, Kivy, Python | Comments Off on Kivy at PyCon

Android apps with Python, Flask and a WebView

python-for-android has just gained support for a new webview app interface, an alternative to the existing SDL2 or Pygame backends. Under this mode of operation the app gui consists entirely of a browser window directed to open a webpage on localhost, and the Python backend can then run any web framework (I tested with Flask, but others like Bottle or even Django should work), serving this website and managing the app backend.

Example Flask app running on Android

This idea is not itself new; I think SL4A has supported a kind of webview interface for some time and certainly does so now, and we’ve previously seen users running web servers alongside Kivy. The difference to other projects is that apps can take advantage of python-for-android’s relatively extensive toolchain including python3.5 support, the ability to build popular libraries like numpy, support for multiple architectures, and access to the Android API via PyJNIus or Plyer rather than SL4A.

In the image of my testing app above, each of the vibration and orientation buttons sends a request to a Flask url that calls the Android API with PyJNIus to achieve the desired result.

Building a webview app

You can use the webview backend by adding --bootstrap=webview to your python-for-android command line (see the documentation for more details), or including webviewjni in your --requirements argument list. Note that this is incompatible with using SDL or Kivy because the webview bootstrap does not start or manage an OpenGL context. If for any reason you want to run a web server alongside a Kivy app, this is possible but you’ll need to use a different bootstrap and manage the webview yourself via PyJNIus from your Kivy code.

You should also add your chosen web framework to the --requirements argument, or include it your app directory so that it will be imported locally. If there isn’t a recipe for it and it’s a pure Python module, make sure you also add its Python dependencies as these aren’t automatically included right now (letting pip resolve dependencies causes issues when they include compiled modules that must be built separately). python-for-android now includes a recipe for Flask that automatically installs its dependencies (jinja2, werkzeug, markupsafe, itsdangerous and click), so you only need to add flask to the requirements in that case.

Technical details

It turns out that very little hackery is necessary to make a webview type app work. The APK seems to need the INTERNET permission to use a WebView, but Android is very happy for the Python code to run a web server with no further problems.

Making PyJNIus work required a little extra work, as it previously relied on the now-absent SDL to access a pointer to the current JNIEnv. This was fairly simple to fix by using only the relevant code from SDL2 - the important parts are only a small fraction of what SDL provides, as SDL has to worry about all the app input and output going via JNI. For now, python-for-android just patches PyJNIus before building it, but now that there are three different ways to get the JNIEnv on Android this will need addressing somehow in PyJNIus itself.

Posted in android, Kivy, Python | Comments Off on Android apps with Python, Flask and a WebView

Kivy Android app showcase

A natural question when people hear about Kivy as a way to create Android apps in Python is…what can you do with it? Is it performant enough for games, can you call the Android APIs, do all apps look the same? One of the best resources for these kinds of question are existing apps, and in this post I’ll give a quick impression of three of my favourites. This is obviously highly subjective, but I’m focusing in particular on features of technical interest, apps that push Kivy beyond what’s normal to show what it capable of.

If you’re interested in other examples, there’s a fairly extensive (but far from exhaustive) list on the Kivy wiki, including winners of our programming contests and many contributions from users. If you’d like to make your own apps in Python, check out Kivy (which also runs on Windows, Linux, OS X and iOS) and python-for-android (which can also package non-Kivy Python apps).

Boardz

You can download Boardz here.

I’ve put Boardz first because it’s my single favourite Kivy app. It’s actually a work in progress (and in fact hasn’t been updated for a while), but is already a fun game showcasing some of Kivy’s more impressive performance potential.

Boardz homescreen and gameplay

Boardz homescreen (left) and gameplay (right). The black ring on the right is the input circle controlling rider posture.

Boardz is a snowboarding physics game; you control your snowboarder by touching the screen, then moving your finger with respect to its initial position to control your posture; quick movements throw your weight around and can cause you to jump, spin, or fall over, while just positioning the rider differently helps you to pick up speed or navigate barriers. The objective of the game is to get to the end of each stage, with different obstacles including slopes and jumps, collapsing structures, falling rocks, or even multidirectional gravity and rocket boosters. You can fail if your head collides with another object with too much force, or if you simply get stuck and can no longer reach the finish.

What’s immediately impressive is that all this runs well as a Python powered game running on a smartphone. It achieves this by being built using the KivEnt game engine developed by Jacob Kovac, one of Kivy’s core developers. This entity based system lets you write game code in Python but internally is highly optimised in Cython, using Kivy’s OpenGL API extremely efficently as well as interfacing with the popular Chipmunk Physics engine.

Boardz wipeout failure and ad example

Boardz wipeout failure by fatal collision (left), and an ad (right).

Boardz betrays its in-progress nature in other ways; you can see in the above screenshots that its UI isn’t very polished, and in this sense it’s the worst of the apps I’m showing here. However, it makes up for this with its surprisingly engaging gameplay, and a breadth of entertaining features not showcased here, including leader boards, racing your ghost, and different riders with different physics attributes.

A final technical feature interesting to Kivy app developers is that Boardz includes ad integration. Regardless of your feelings about ads themselves, the ability to use them is a major feature enquiry from new Kivy users. The problem here is that integrating with a normal ad provider normally requires adding to the Java components of your app, which it may not be immediately obvious how to do from Python. There are actually a number of resources for this nowadays, with a key point being that python-for-android tries to make it easy to include extra Java code, with which you can interact from Python using Pyjnius. KivEnt’s implementation, pictured above, is a nice demonstration.

Kognitivo

You can download Kognitivo here.

Kognitivo is perhaps the single most polished Kivy app on the Play store, being relatively complex, extensively customised, and exhibiting a number of nice Android API interactions. It is also (deservedly) possibly the most popular Kivy app on Google Play.

Kognitivo introduction and main screens

Kognitivo tutorial (left), game instructions (centre) and homescreen (right).

Kognitivo is a brain training and performance monitoring app. The basic structure is to perform a series of simple exercises intended to test different aspects of cognitive performance, being rated on accuracy and speed, and with the results compiled over time in order to detect and act on trends.

As I’ve said already, the nice thing about Kognitivo is its huge amount of polish. It is extensively themed such that no trace of the Kivy defaults remains, runs extremly smoothly, and includes many nice animation tweaks (unfortunately not captured in screenshots) to feel responsive and active.

Kognitivo gameplay, Android notification and in-app purchase

Kognitivo training game (left), Android notification (centre) and in-app purchase option (right).

On the technical side, Kognitivo exhibits a number of features not normally included in Kivy applications but possible through interaction with the Android API via some Java code and/or the aforementioned Pyjnius. These include notifications, interaction with your calendar, and in-app purchases.

The author of Kognitivo, Sergey Cheparev, has his own writeup of Kognitivo’s development on his blog, including discussion of the advantages and disadvantages of Kivy development, and of the experience of putting together all these features. This is a great resource on its own; I don’t agree with some of the author’s criticisms and some of Kivy’s features have been improved since then, but it’s an excellent overview of the experience. Most of all, he enthusiastically captures some of my own reasons for finding Python on Android interesting:

I think the most beautiful thing in it is to use the almightiness of [Python]’s frameworks. I used sqlalchemy and sqlite as a backend, and it worked like a charm! Python is the most powerfull language because of it’s frameworks, you can even start Django on your smartphone! It’s amazing! Or twisted for asynchro communication with server. Or nltk for in-app natural language processing. Or maybe you want make a mobile equations solver with scipy and numpy. This makes all the dreams come true.

Barly

You can download Barly here, or visit its own website.

Barly is the most recent of these apps to appear on Google Play. I’ve chosen to include it as a nice example of pulling off its concept quite well while making good use of Kivy; like Kognitivo the app is themed very differently to Kivy’s defaults (though it doesn’t look like a normal Android app either), and is generally well put together.

Barly homescreen, palate options, and beer example

Barly homescreen (left), palate options (centre) and a beer search result (right).

To quote its Google Play blurb, Barly is ‘your personal beer expert’. It provides a convenient interface to browse beers via data from standard popular websites, and according to your description of your own palate. Barly’s most interesting feature is the ability to take a picture of a beer menu and have it automatically detect what beers are listed, followed by downloading information about them to help you choose. That kind of image analysis has to be tricky, but actually didn’t perform badly when I tested it.

The interface to this functionality is quite nice, switching to the Android camera app to get the image before uploading it to a server for processing (during which you can input your preferences). This functionality is possible with Pyjnius as mentioned previously, but actually in this case is an API also exposed in pure Python by Plyer (another Kivy sister project, wrapping platform-specific APIs in a Python frontend). Not all APIs can be conveniently exposed this way, and actually Barly may not even be using this particular method, but it’s a good example of functionality that can be achieved with Kivy in a particularly cross-platform way.

Beyond this, Barly does not make such wide use of the Android API or unusual Kivy features, but nor does it try to; it is a nice example of a complete and self-contained Kivy app using the power of Python for an unusual and interesting goal.

Posted in android, Kivy, Python | Comments Off on Kivy Android app showcase

python-for-android now supports Python 3 APKs

It’s been a long time coming, but we can finally make the announcement… python-for-android now supports Python 3 Android apps! This naturally includes Kivy, but also should work for anything else you can package with python-for-android, such as apps made with PySDL2. Using Python 3 remains experimental for now, it works but doesn’t yet perform all possible optimisations and hasn’t been as widely tested as Python 2. However, there should be no extra application requirements (beyond actually being written for Python 3), and the remaining issues and optimisations are being worked on.

Overview

python-for-android is a packaging tool for turning Python applications into Android APKs. It was originally created to make apps with the very cross-platform Kivy graphical framework (though it didn’t arise in a vacuum, I think it built in particular on previous work by the Ren’Py project). However, the original python-for-android had flaws including being quite inconvenient to modify for non-Kivy apps (several other projects seem to have used modified versions, but each was performing similar changes), hard to extend for multiple architecture support, hard to extend internally (both in general and from the perspective of new contributors, as much of the toolchain was a big shell script), and only supported building apps with Python 2.

We recently completed and released a fully revamped version of python-for-android aimed at fixing all of these problems, as discussed in several previous posts on this blog (originally here). Almost all of the original goals are now complete, with Python 3 support the major missing one until now, though not for lack of trying. Some technical details and basic instructions for testing the new support are given below, and you can also see the online documentation for further information.

Our Python 3 support depends on the prebuilt Python distributions provided with the CrystaX NDK, a drop-in replacement for Google’s own Android NDK with many fixes and improvements. The technical details of this choice are given below, and we’ll try to further support a locally-built Python 3 option in the future. Thanks to the CrystaX team for making it so (relatively) easy!

Technical details

python-for-android works by bundling a Python interpreter, compiled for Android devices and architectures (usually arm, though other choices are supported), into an Android APK. The APK includes a simple Java bootstrap application, which mostly starts a Python script via JNI. The script then runs essentially as normal, almost all of the Python standard library is present and works fine, and python-for-android supports including other modules or non-Python dependencies. Pure Python modules will mostly work without special treatment, though things requiring compilation need a special recipe. Many common modules such as numpy and sqlalchemy are supported this way. Following its revamp python-for-android is now designed to support multiple kinds of java bootstrap, but the current main support is for GUI apps via Pygame (for Kivy’s old Android support) or SDL2 (both for Kivy and for anything else that can use it); SDL2 also now does much of the heavy lifting of handling events etc. itself, via its own Android support.

The main problems with compiling and including Python are first that it must be patched to compile (as Android’s libc doesn’t support some things very well or at all), and second that it must be unpacked and started on the device via its C API. The second point is fiddly but ultimately not that different to working this way on the desktop. The first is (in my opinion) harder because it needs some understanding of Python’s internals, of Android’s limitations, of appropriate fixes, and of how to test and debug these problems.

Such patches have been made by a number of different people for different Android versions, and I believe there has been activity on Python itself to fix some issues (including this current issue to make Python build natively on Android). For Python 2, I think python-for-android’s original patches came from here, though extended with further modifications. However, the main thing holding up my efforts to get Python 3 working was the inability to find a similar working patchset; I tried a few sources, achieving a working compilation using SL4A’s python3 patches, but I couldn’t get Python working on the device. I’m sure this was my own technical mistakes, since other projects do have it working, but it’s what was holding up the feature.

I eventually resolved this by using the new Python on Android support from the CrystaX NDK. As mentioned above, this is a drop-in replacement for Google’s own NDK (the Native Development Kit providing compilers etc for targeting Android with non-Java code), including many improvements to the build environment. As of version 10.3.0, they provide prebuilt Python packages for Android on all architectures - and all of their NDK improvements mean that Python no longer even needs patching for this compilation to work. Python is provided as a zipped standard library (Python can automatically load modules from zip files), and a folder of the compiled components like ctypes (as it’s hard to dynamically load from zips). From the perspective of python-for-android, supporting Python 3 means modifying the build to load CrystaX’s prebuilt components (both in the Android project structure and in python-for-android’s support for compiling other modules), and modifying the C initialisation code for Python 3. This takes some work, but all told wasn’t very hard and the Python bundles worked with no issues, so we owe a lot to CrystaX; thanks again.

I’d still like to come back to the issue of local python3 compilation; CrystaX’s versions are fine, but I’ve learned a lot from making them work, and have a much better idea of what I may have been doing wrong. However, the focus for now will be on resolving the remaining issues with what’s already working.

Building apps with Python 3

Building Python 3 APKs is only supported in the revamped python-for-android toolchain which was merged to the master branch a while ago. It can be installed and used as described in its documentation. If you use Buildozer, it currently does not support this new toolchain, though tito has been working on this. There is also the new restriction that you must (for obvious reasons) use the CrystaX NDK, which can be obtained here; simply refer to its filepath when setting the NDK directory, and everything else should work automatically.

To build for Python 3, add the python3crystax recipe to the requirements option, e.g. --requirements=python3crystax,kivy. It should mostly work automatically with existing recipes, though at this stage there may be bugs or problems with a few, and some will need modification. The exact syntax above may also change in the future as the Python 3 support becomes better integrated, but not significantly.

There’s also one big change whose importance I’m not sure about; the Python 3 mechanism doesn’t currently build a local Python 3 to use as a hostpython, instead using the system python. This means that you must have python3.5 (3.4 may also work) installed locally in order for python-for-android to build Python 3 APKs. This will be fixed soon, adding a hostpython3 recipe to avoid weird bugs with system-specific differences, but you need to bear it in mind for now.

Future work

For now, this Python 3 support remains experimental. I anticipate no major issues, but it’s internally a quite different method to the Python 2 support and needs further work to duplicate some of the old optimisations, and undoubtedly to fix bugs in the toolchain that will appear as it stabilises. Amongst other things, Python 3 shared libraries are not currently collected and merged (with Python 2 we did this originally to get around an Android limit but also for optimisation), python files are not precompiled to bytecode (it can make a big loading speed difference), and some features of the old pygame bootstrap have not yet been implemented in SDL2. All this and more will come in the future, but shouldn’t be hard to add now that the toolchain all works.

The SDL2 bootstrap is also missing a few features that users of the old toolchain will be used to, like the splash screen image and at least one Kivy-specific function. These too are being actively worked on, especially as more people start to move their apps to SDL2.

I’ve also phrased this as Python 2 built locally vs Python 3 from CrystaX, but actually CrystaX also supports Python 2.7 and I hope to add this option in the near future. As discussed in the technical details, it also should absolutely be possible to have a local Python 3 build, which I’d like to eventually come back to.

Posted in android, Kivy, Python | Comments Off on python-for-android now supports Python 3 APKs

Kivy 1.9.1 released

We’ve just released a new stable version of Kivy, version 1.9.1. You can see the changelog on the mailing list announcement, and download the new version from the Kivy website or via your package manager.

This is mainly a bugfix and tidying release following the major version 1.9 last year, but includes many bugfixes, smaller new features, and improvements to our surrounding infrastructure across almost 1000 new commits from over 70 different contributors.

One major improvement for Windows users is that we now have a fully working installation method using pip and wheels for both Kivy and its non-python binary dependencies, rather than our older standalone kivy distribution. This should make it easy to install Kivy in any existing Python installation. OS X distribution has also seen improvement, including better support for working with homebrew.

We’ve also improved app packaging particularly on OS X, with a new packaging method that should be easier than pyinstaller (though pyinstaller is still supported), a buildozer backend for OS X packaging (now buildozer works with Android, iOS and OS X!), and generally improved and updated documentation for the packaging process. The documentation for Windows and Linux packaging has similarly been updated, and the new packaging methods and buildozer support will hopefully be added for these in the future.

Packaging for Android with python-for-android is not tied to the Kivy update schedule in the same way, but has been seeing significant improvements and updates in the last few months, including a full revamp of the toolchain and support for many new features, which you can see in several of the recent previous posts on this blog.

In the future, we’re heading towards Kivy 2.0, which we’ve had in mind for a while to be a major release with some big new features and potentially removal of some long-deprecated components. We aren’t sure on the timescale for this yet, but if it takes too long there will be other minor releases first. For other updates, watch this blog or the standard Kivy support channels.

Thanks to all our contributors, and enjoy the new release!

Posted in Kivy, Python, release | Comments Off on Kivy 1.9.1 released

python-for-android status update

It’s been a while since Kivy’s python-for-android project was revamped, so here’s a quick status update.

There have since been well over 200 commits from 15 different contributors, cleaning up the missing pieces of the new toolchain and adding new features that weren’t previously possible. Thanks to everyone who has contributed.

These fixes include progress on the remaining major goals of the python-for-android revamp. In particular, compilation is now supported for multiple target architectures - in principle anything targeted by the Android SDK (i.e. ARM, ARMv7a, x86, x86_64 and MIPS options), though I’ve tested only with the ARM and x86 ones. This means that Kivy applications, or other Python projects built with these tools, can be built for devices with e.g. intel atom processors. Even without this compilation it was often possible to run Kivy apps as many devices include libhoudini, but directly targeting them means such apps should now always work. A further advantage is that Kivy apps can be built for and tested on the Android emulator, which was not previously possible.

The architecture target support does need some more work to create fully multiarch APKs (i.e. including .so files for each target, so a single APK can work on different types of device). The problem here is that we need to duplicate as little as possible, as Kivy APKs are already made large by including the python interpreter, and it is undesirable to include two or more copies of everything. Using a single python installation and loading the .so dependencies as appropriate should be possible but needs more work. However, this is not a problem if uploading an APK to a store like Google Play; in this case you can include multiple APKs, one for each arch target, and the user will receive one that is appropriate.

Another important feature that I’ve worked on, but unfortunately unsuccessfully so far, is support for python3 APKs. The problem to be solved is to patch the interpreter to compile for Android (it cannot do so by default, due to problems with the Android platform like poor locale support), to modify the python-for-android bootstrap to load it correctly (it builds things a little differently to python2), and to modify the initialisation code to have it start successfully. I’ve only partially succeeded with the first two of these; using patches from the SL4A python-for-android tools (plus extras for our own modifications to python loading) allows the interpreter to be built, but it fails during Py_Initialize when run on the device, apparently raising an exception when calculating the python install path. Work on this will continue, but it’s hard to know how long it might take to resolve this error. If you know of any other projects patching python3.4+ for Android, I’d love to heard about it to compare their methods.

Posted in android, Kivy, Python | Comments Off on python-for-android status update

python-for-android revamp replaces master

This post is to announce that the revamped python-for-android toolchain, introduced in this previous post, has now been merged into python-for-android’s master branch. This is now the master branch going forward.

The revamp project is largely (but not quite) feature complete with the old toolchain, supporting almost all the same options when building pygame-based APKs. It also supports a new and much better SDL2 backend, which Kivy will move to in the future, but which also supports other kinds of python projects such as Vispy as described in the previous post.

We’ve done our best to minimise problems arising from this change. The old toolchain (with distribute.sh and build.py) is still available in the old_toolchain branch. Issues and PRs relating to this branch are still accepted, though existing PRs will need to be retargeted or merged manually (we’ll try to do what’s easiest case by case, if necessary).

If you use buildozer, this now pulls from the old_toolchain branch and so will work exactly as before. However, you will need to install the latest version from pypi or github (at least version 0.30) for this to work. Older versions of buildozer will continue to build APKs fine with existing projects, but if you create a new one they will download the new and incompatible python-for-android master. The revamp includes a fake distribute.sh executable giving these same instructions, so if this happens the problem and solution should be clearly displayed.

The new toolchain is currently documented (temporarily) here. We’ll push the new documentation to the normal readthedocs site as soon as possible, which will also include the legacy doc for the old toolchain so nothing is lost.

In the slightly longer term, the new toolchain will receive an official release and hopefully be released itself on pypi; unlike the old toolchain, it behaves as a fully installable python module with an improved command line interface.

Posted in android, Kivy, Python | Comments Off on python-for-android revamp replaces master

Kivy – Interactive Applications and Games in Python, 2nd Edition Review

I was recently asked by the author to review the second edition of “Kivy – Interactive Applications in Python” from Packt Publishing. I had difficulty recommending the first edition mostly due to the atrocious editing – or lack thereof – that it had suffered. It really reflected badly on Packt, and since it was the only Kivy book available, I did not want that same inattention to quality to reflect on Kivy. Packt gave me a free ebook copy of this book in exchange for agreeing to do this review.

At any rate, the second edition is much improved over the first. Although a couple of glaring issues remain, it looks like it has been visited by at least one native English speaking editor. The Kivy content is good, and I can now recommend it for folks who know Python and want to get started with Kivy. The following is the review I posted to Amazon:

This second edition of “Kivy – Interactive Applications and Games in Python” is much improved from the first edition. The atrocious grammar throughout the first edition book has mostly been fixed, although it’s still worse than what I expect from a professionally edited book. The new chapters showcase current Kivy features while reiterating how to build a basic Kivy app, and the book covers an impressive amount material in its nearly 185 pages. I think this is due largely to the efficiency and power of coding in Python and Kivy, but also to the carefully-chosen projects the author selected for his readers to create. Despite several indentation issues in the example code and the many grammar issues typical of Packt’s books, I can now recommend this book for intermediate to experienced Python programmers who are looking to get started with Kivy.

Chapter one is a good, quick introduction to a minimal Kivy app, layouts, widgets, and their properties.

Chapter two is an excellent introduction and exploration of basic canvas features and usage. This is often a difficult concept for beginners to understand, and this chapter handles it well.

Chapter three covers events and binding of events, but is much denser and difficult to grok than chapter two. It will likely require multiple reads of the chapter to get a good understanding of the topic, but if you’re persistent, everything you need is there.

Chapter four contains a hodge-podge of Kivy user interface features. Screens and scatters are covered well, but gestures still feel like magic. I have yet to find a good in-depth explanation of gestures in Kivy, so this does not come as a surprise. Behaviors is a new feature in Kivy and a new section in this second edition of the book. Changing default styles is also covered in this chapter. The author does not talk about providing a custom atlas for styling, but presents an alternative method for theming involving Factories.

In chapter six the author does a good job of covering animations, and introduces sounds, the clock, and atlases. He brings these pieces together to build a version of Space Invaders, in about 500 lines of Python and KV. It ends up a bit code-dense, but the result is a fun game and a concise code base to play around with.

In chapter seven the author builds a TED video player including subtitles and an Android actionbar. There is perhaps too much attention paid to the VideoPlayer widget, but the resulting application is a useful base for creating other video applications.

Posted in grpug, Kivy, Python, tech | Comments Off on Kivy – Interactive Applications and Games in Python, 2nd Edition Review

python-for-android revamped

I’ve recently been working on a significantly revamped version of python-for-android, the Kivy-project tools that take a Python program and package it - along with any dependencies and the Python interpreter itself - into an Android APK that can be run and distributed just like a normal Android app. This rewrite is driven by the problem that although the current python-for-android is fairly robust as far as the build process goes, it’s quite set in its ways and hard to modify to make large changes such as to support backends other than Kivy.

To this end, the revamp project has several major goals:

  • Rewrite python-for-android to a fully Python toolchain that can be more easily modified and extended.
  • Support multiple bootstrap targets for different kinds of Python scrips to run on Android, starting with a new SDL2 backend for Kivy applications.
  • Support Python 3 on Android.
  • Support some kind of binary distribution, enabling easier and cross-platform distribution…this should work on Windows!
  • Be a standalone PyPI module with a more convenient and standard interface, potentially interfacing with setuptools.
  • Support (less painfully) multiple Android architectures, or fat APKs supporting more than one.

Each of these is individually a popularly requested feature, and this is a great opportunity to go for all of them at once!

I’m making this post to announce that the python-for-android revamp project has reached a usable state, with several of these goals either implemented or significantly advanced, and all of them at least made much more accessible. The core change is that all of the original toolchain has been rewritten in Python, with the initial structure based on the recent Kivy-iOS rewrite. It’s also much lighter, all heavy pygame stuff is downloaded on demand instead of built in, and is designed to be accessed by a single set of commands and the new python-for-android executable rather than by the separate invocation of different scripts in different places. I won’t go into the technical details here, but you can find the (WIP) documentation temporarily hosted here. If anyone would like to test it you can try the instructions there, but the project is in an experimental state right now and it’s likely you may encounter bugs or missing features, the current focus is ironing these out. I’m very happy to discuss these on the kivy-users mailing list or #kivy irc channel on irc.freenode.net.

Another of the core goals of this rewrite was to support multiple app backends; in Kivy’s case in particular we want to move from Pygame to a new SDL2 bootstrap, but this could also include support for other Python module backends. As of now, the new python-for-android supports the old Pygame bootstrap mostly as before, but also implements an SDL2 based PythonActivity that works very well with Kivy - highly anecdotal testing found, amongst other things, app start time reduced to 60% of what it was with Pygame. It also simplifies maintenance as SDL2’s java components fulfil the same role as those maintained in the Pygame backend, but no longer require much special treatment as functionality such as touch interaction and pausing are now accessed with the SDL2 api just as on desktop platforms.

Further, python-for-android can now build non-Kivy apps! The first project with this support built in is the Vispy scientific visualisation library. This uses the same new SDL2 backend but Kivy is not involved, and the build process does not use Cython (unlike with Kivy); instead, SDL2 and OpenGL are called entirely with ctypes, using pysdl2 and Vispy’s own gloo wrapper respectively. I didn’t even have to patch things much for this, Vispy is mostly self contained and already quite platform independent, barring a couple of possible small bugs and a hack to avoid the lack of a supported font provider on Android. Vispy also uses numpy heavily, but this was already supported by python-for-android and seems to work fine. The Vispy support is itself quite experimental and there are some performance issues that will need resolving, but it was very simple to set up with the new toolchain. Here’s a screenshot of one of Vispy’s 3D examples running on Android - there are a few small visual artifacts, but I think these are small bugs in Vispy’s OpenGL ES 2 support that the Vispy team are actively addressing:

Image of Vispy running on Android

Support for binary distribution and multiple architectures are both partially implemented but (at the time of writing) not yet working. However, the toolchain is built around them, so there should be no major issues. The initial idea with binary distribution will be to simply make available a number of prebuilt distributions (i.e. Android projects with the Python interpreter) that include common dependencies, so that when the user adds modules as requirements when calling python-for-android they can automatically be checked and an appropriate choice downloaded, with this process being transparent to the user and not requiring any special options. This should not only make many builds faster but also work on Windows, one of our most requested features but something that was not possible when the toolchain required that everything be locally compiled. Likewise, the toolchain has semi-implemented support for multiple architectures internally, but none other than armeabi are yet supported and there will be bugs to work out when more are enabled. Still, these will (fingers crossed) be things to look forward to in the relatively near future.

I should note here that this model of binary distribution is what I initially targeted as a natural extension of python-for-android’s distribution system - although we never made much of it, the first step is already to build a standalone android package which later can be distributed separately and populated with app details by a user, and the above just involves making such prebuilt distributions available to download and use automatically. I found more recently the method of the pybee project/Toga toolkit’s Python-Android-template. This is a similar idea (built by a modified python-for-android) but implemented much better as a standalone project with app details populated by cookiecutter and the packaging of the user’s Python modules taken care of using ant itself, the normal APK build and deployment interface - I didn’t know this was even possible! This means that the user can just write their Python code, drop it in place, and run the APK build, a very neat process. It should be quite easy to modify python-for-android’s dist output to easily create such templates, its dist system is functionally the same thing with a different and less standard-Android interface, and I hope that doing so could make it even more convenient for users to build and distribute different kinds of Python apps.

This leaves the elephant in the room, support for Python 3 on Android, which is perhaps the most requested feature for Kivy itself. The new toolchain makes significant progress on this in a couple of ways. The first is by removing hardcoded use of Python 2, so that now (barring remaining bugs) a Python 3 build should be well behaved once a recipe and appropriate Android patch set are added. The second is that the old Kivy Android bootstrap probably needed significant changes to support Python 3, but this is sidestepped entirely by moving to the new SDL2 backend which should have no issues supporting Python3 more or less the same as on the desktop. However, the missing link here is still actually being able to compile Python 3 for Android, and I don’t yet know how to do this. Some of Kivy’s main Python 2 patches come from this site, but this has Python 3 patches only up to 3.2.2 and it would be ideal to target 3.4 or 3.5 (and to be able to support new versions as they appear). I’ve looked around and seen a few different discussions of this, but I’m not sure what’s the best direction to try. If anyone has any information about places to find more up to date patch sets I would be very interested. I can’t make any other predictions about this as I don’t know much about the Python compilation process or what difficulties might be involved in doing the work we need.

That’s all for now. I’ll note again that this is an initial announcement of the new toolchain; I hope that people may be interested to look and try it, and it should support most of what the old toolchain does when it comes to compiling Pygame APKs, but there are likely to be bugs and missing features particularly surrounding (but not limited to) the new additions. If you’re interested in making this toolchain work with other modules or backends, or just have any questions, comments or criticisms, let us know! If you want to keep informed, watch this space, I’ll make further announcements as things develop. If there is developer interest and people start switching from the old toolchain, I hope development will speed up a lot and quickly approach proper feature parity.

tl;dr (I was told I should have one): Kivy’s python-for-android build tools have been revamped to have a better interface, build apps based on SDL2, build non-Kivy apps (currently Vispy apps), and be more extensible. Further semi-complete features include binary distribution, Windows support, and multiple architecture targets. Python 3 is brought closer but needs direct compilation work.

Posted in android, Kivy, Python | Comments Off on python-for-android revamped