Exploring how to create binary wheels for Pythonista
31 Jul 2022This is a follow-up on the previous post on how to get Pip working with Pythonista.
We ended with a working pip but didn’t have a way of installing binary packages yet (like scipy and scikit-learn).
Using the Oracle Cloud, which offers (free!) aarch64 intances, I tried to build some Python wheels for my iPhone.
sudo apt install zlib1g-dev make libssl-dev curl
git clone https://github.com/deadsnakes/python3.6
cd python3.6
./configure
make
curl -L https://bootstrap.pypa.io/pip/3.6/get-pip.py > ./get-pip.py
./python get-pip.py
./python -m pip wheel scikit-learn
After uploading it to my PyPI repository we can try to install it using pip.
Except unfortunately, this wheel is still not in the expected format.
This is related to how the wheel file format is specified in PEP 0427.
The short summary is that the platform tag can be seen in the filename:
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.
To find out what platform tags Pythonista asks for on iPhones, I added a debug line in pip/_internal/index/package_finder.py, and ran pip again.
This resulted in the following list of platform tags:
The Python wheels will need to comply to this expected platform tag.
Instead of cp36-cp36m-manylinux2014_aarch64 it looks like it needs to say cp36-cp36-darwin_21_5_0_iphone12,8.
Update: Right after finishing this post I noticed that the platform tag changed to
cp36-cp36-macosx_15_0_iphone12,8(instead of thedarwintag above). This might have been caused by an iOS update.
After running auditwheel fix on the created wheel all relevant system libraries are copied into the wheel.
Now I run the following script to modify the wheel to use the expected platform tag:
import zipfile
with zipfile.ZipFile('scipy-1.5.4-cp36-cp36m-linux_aarch64.whl', 'r') as input_wheel:
with zipfile.ZipFile('scipy-1.5.4-cp36-cp36-macosx_15_0_iphone12,8.whl', 'w',
compression=zipfile.ZIP_DEFLATED) as output_wheel:
for input_zipinfo in input_wheel.infolist():
if input_zipinfo.filename.endswith('.dist-info/WHEEL'):
output_wheel.writestr(
input_zipinfo.filename,
input_wheel.read(input_zipinfo.filename).replace(
b'cp36-cp36m-linux_aarch64',
b'cp36-cp36-macosx_15_0_iphone12,8')
)
elif input_zipinfo.filename.endswith('.dist-info/RECORD'):
output_wheel.writestr(
input_zipinfo.filename,
input_wheel.read(input_zipinfo.filename).replace(
b'.cpython-36m-aarch64-linux-gnu',
b'')
)
else:
output_wheel.writestr(
input_zipinfo.filename.replace('.cpython-36m-aarch64-linux-gnu', ''),
input_wheel.read(input_zipinfo.filename)
)
Now it is recognized by pip as suitable for the platform and installs without issue.
When trying to use the newly installed scipy however, it still can’t find the correct shared objects.
If we try to directly import this shared object using ctypes, we can see better why it will not work:
DLLs need to be Mach-O, instead of the a.out format.
But how does Pythonista include the non-standard-libraries it ships with? To find out, I made a copy of the app itself. This was quite easy to do, since Pythonista ships with Python:
Using this dump, I could determine that the extra packages like numpy and matplotlib all live in Frameworks/Py3Kit.framework/pylib/site-packages.
However, in this directory, all shared objects that normally also live there, are missing.
If we decompile the app’s Py3Kit.framework executable, we can see that it actually contains these binary Python modules that were missing in site-packages.
They are all added to the built-in Python packages, using the _PyImport_AppendInittab method available in Python’s C API.
void PYK3Interpreter::registerBuiltinModules(ID param_1,SEL param_2)
{
_PyImport_AppendInittab("speech",_PyInit_speech);
_PyImport_AppendInittab("reminders",_PyInit_reminders);
_PyImport_AppendInittab("contacts",_PyInit_contacts);
_PyImport_AppendInittab("sound",_PyInit_sound);
_PyImport_AppendInittab("linguistictagger",_PyInit_linguistictagger);
_PyImport_AppendInittab("_ui",_PyInit__ui);
_PyImport_AppendInittab("_notification",_PyInit__notification);
_PyImport_AppendInittab("_pythonista",_PyInit__pythonista);
_PyImport_AppendInittab("_keyboard",_PyInit__keyboard);
_PyImport_AppendInittab("_dialogs",_PyInit__dialogs);
_PyImport_AppendInittab("_appex",_PyInit__appex);
_PyImport_AppendInittab("_font_cache",_PyInit__font_cache);
_PyImport_AppendInittab("_scene2",_PyInit__scene2);
_PyImport_AppendInittab("console",_PyInit_console);
_PyImport_AppendInittab("_clipboard",_PyInit__clipboard);
_PyImport_AppendInittab("_photos",_PyInit__photos);
_PyImport_AppendInittab("_photos2",_PyInit__photos2);
_PyImport_AppendInittab("_webbrowser",_PyInit__webbrowser);
_PyImport_AppendInittab("_twitter",_PyInit__twitter);
_PyImport_AppendInittab("location",_PyInit_location);
_PyImport_AppendInittab("_motion",_PyInit__motion);
_PyImport_AppendInittab("keychain",_PyInit_keychain);
_PyImport_AppendInittab("_cb",_PyInit__cb);
_PyImport_AppendInittab("_canvas",_PyInit__canvas);
_PyImport_AppendInittab("_imaging",_PyInit__imaging);
_PyImport_AppendInittab("_imagingft",_PyInit__imagingft);
_PyImport_AppendInittab("_imagingmath",_PyInit__imagingmath);
_PyImport_AppendInittab("_imagingmorph",_PyInit__imagingmorph);
_PyImport_AppendInittab("_np_multiarray",_PyInit_multiarray);
_PyImport_AppendInittab("_np_scalarmath",_PyInit_scalarmath);
_PyImport_AppendInittab("_np_umath",_PyInit_umath);
_PyImport_AppendInittab("_np_fftpack_lite",_PyInit_fftpack_lite);
_PyImport_AppendInittab("_np__compiled_base",_PyInit__compiled_base);
_PyImport_AppendInittab("_np__umath_linalg",_PyInit__umath_linalg);
_PyImport_AppendInittab("_np_lapack_lite",_PyInit_lapack_lite);
_PyImport_AppendInittab("_np_mtrand",&_PyInit_mtrand);
_PyImport_AppendInittab("_np__capi",_PyInit__capi);
_PyImport_AppendInittab("_mpl__backend_agg",_PyInit__backend_agg);
_PyImport_AppendInittab("_mpl__image",_PyInit__image);
_PyImport_AppendInittab("_mpl__path",_PyInit__path);
_PyImport_AppendInittab("_mpl_ttconv",_PyInit_ttconv);
_PyImport_AppendInittab("_mpl__cntr",_PyInit__cntr);
_PyImport_AppendInittab("_mpl_ft2font",_PyInit_ft2font);
_PyImport_AppendInittab("_mpl__png",_PyInit__png);
_PyImport_AppendInittab("_mpl__delaunay",_PyInit__delaunay);
_PyImport_AppendInittab("_mpl__qhull",_PyInit__qhull);
_PyImport_AppendInittab("_mpl__tri",_PyInit__tri);
_PyImport_AppendInittab("_counter",_PyInit__counter);
_PyImport_AppendInittab("_AES",_PyInit__AES);
_PyImport_AppendInittab("_ARC2",_PyInit__ARC2);
_PyImport_AppendInittab("_ARC4",_PyInit__ARC4);
_PyImport_AppendInittab("_Blowfish",_PyInit__Blowfish);
_PyImport_AppendInittab("_CAST",_PyInit__CAST);
_PyImport_AppendInittab("_DES3",_PyInit__DES3);
_PyImport_AppendInittab("_DES",_PyInit__DES);
_PyImport_AppendInittab("_MD2",_PyInit__MD2);
_PyImport_AppendInittab("_MD4",_PyInit__MD4);
_PyImport_AppendInittab("_RIPEMD160",_PyInit__RIPEMD160);
_PyImport_AppendInittab("_SHA224",_PyInit__SHA224);
_PyImport_AppendInittab("_SHA256",_PyInit__SHA256);
_PyImport_AppendInittab("_SHA512",_PyInit__SHA512);
_PyImport_AppendInittab("_XOR",_PyInit__XOR);
_PyImport_AppendInittab("strxor",_PyInit_strxor);
_PyImport_AppendInittab("pykit_io",_PyInit_pykit_io);
return;
}
In a next post I’ll be looking into compiling the wheels with Mach-O shared libraries (or bundles as Apple calls them).