Towards a universal DRU fireplace remote using the Flipper Zero

In the manual of my gas fireplace’s remote I came across this bit of text:

Voordat het toestel in gebruik wordt genomen, moet een communicatiecode ingesteld worden tussen de afstandsbediening en de ontvanger. De code wordt willekeurig gekozen uit de 65000 codes die beschikbaar zijn. Hierdoor is de kans klein dat andere afstandsbedieningen in uw omgeving dezelfde code gebruiken en de werking van uw toestel beïnvloeden.

Translated into English, it says something like:

Before using the device, a communication code needs to be set between the remote and the receiver. This code is chosen randomly from the 65000 codes that are available. Because of this, the chances are slim that a different remote in your environment uses the same code, which would interfere with the working of your device.

The number 65000 is suspiciously close to 2^16 (65536). This means that the Mertik GV60 (the remote type) might send a “unique-enough” 2-byte identifier over the air, along with the command for the heater.

Since this remote transmits at 433.92 MHz, it can be interesting to see what the Flipper Zero makes of this signal. To do this, I used the “Read Raw” functionality in the Sub-GHz app on the Flipper.

Dumping files for two different remotes, and for four different operations (higher, lower, ignite, turn off), we end up with eight files:

• Remote0_higher.sub
• Remote0_lower.sub
• Remote0_ignite.sub
• Remote0_off.sub
• Remote1_higher.sub
• Remote1_lower.sub
• Remote1_ignite.sub
• Remote1_off.sub

Since only one of these remotes works with my fireplace, it’s safe to assume they have different identifiers. This will be nice later, if we are going to compare the signals.

Reading a bit more in the manual, it also seemed unlikely to me that there was an actual bi-directional handshake when connecting a remote to the fireplace. To pair it, you need to put the receiver in pairing mode, and press the flame higher or lower button within 20 seconds. This makes me suspect that the 2-byte identifier is hardcoded in the remote, since the remote itself does not have to be put in some kind of pairing mode.

Now we need to make sense of the Flipper Zero’s .sub-files. The documentation mentions that a raw .sub file contains timings, but does not have a lot of information beyond that:

RAW_Data, contains an array of timings, specified in micro seconds. Values must be non-zero, start with a positive number, and interleaved (change sign with each value).

Of course I am not the first person to look at those files, so I found the fzsubtk script on Github. In absence of a software license, I just read this as inspiration to make my own visualisation.

While parsing the .sub-file, I discovered something that probably shouldn’t happen when dumping these files. I had a Raw_Data line that started with a negative value, which should not be possible. Of course I have submitted this as a Github issue: flipperzero-firmware#2260. I quickly received a reply, and it should be fixed for newer versions of the Flipper Zero firmware.

import numpy


def read_sub_ghz_file(filename):
    """
    Read a .sub file as produced by Flipper Zero, and prepare it for plotting.

    This method contains some fixes that might truncate some of the data.
    These should be fixed with a newer release of the Flipper Zero firmware.
    """
    with open(filename, 'r') as f:
        values, durations = [], []
        for line in f.readlines():
            if line.startswith("RAW_Data:"):
                data = [int(x) for x in line[10:].split(' ')]
                # The two fixes below are for Github issue flipperzero-firmware#2260
                if data[0] > 0 and data[1] > 0:
                    data = data[2:]
                if data[0] < 0:
                    data = data[1:]
                for i, point in enumerate(data):
                    if i % 2 == 0:
                        values.append(point)
                    else:
                        durations.append(point)
    durations, values = numpy.abs(numpy.array(durations)), numpy.cumsum(numpy.abs(numpy.array(values)))
    max_len = min([len(durations), len(values)])
    return values[:max_len], durations[:max_len]

from matplotlib import pyplot

remote1_lower = read_sub_ghz_file('Remote1_lower.sub')
remote0_lower = read_sub_ghz_file('Remote0_lower.sub')

# all the numbers below don't mean anything, and are just to align the plot a bit
pyplot.figure(figsize=(16, 8))
pyplot.ylim(-500, 2000)
pyplot.xlim(-2500, 15000)
pyplot.step(remote0_lower[0] - 941300,
            remote0_lower[1], where='pre')
pyplot.step(remote1_lower[0] - 761825,
            remote1_lower[1] - 400, where='pre')
pyplot.show()

Now that we have plotted the signals produced by two different remotes nicely, it is time to start speculating on the encoding. My best guess currently is that we’re looking for a 3-byte sequence: two bytes to identify the remote, and one byte that specifies the command to execute. These are the raw bits I think I can read from the plot:

signal_blue =   '1100001001000000110011000'
signal_orange = '1000000100001000010011111'

len(signal_blue) // 8

There are many different ways to encode a digital signal over analog radio. This video by Jacob Schrum explains some common ones quite well, and has helpful examples.

I might return to this project later, in an attempt to find the encoding. I’ll be familiarizing myself with some signal processing tools, or perhaps try to bruteforce all possible encodings with some custom scripting.

Replaying the signal is nice, but the end goal of course is to create a Flipper application that can ignite any DRU fireplace. Sources used:

1. https://github.com/cberetta/flipperzero_scripts/blob/master/docs/fzsubtk_example.md
2. https://www.bouwmansvuurtotaal.nl/wp-content/uploads/2016/04/Bouwmans_Vuur_Totaal_handleiding_compleet_Dru-Metrik-GV60.pdf
3. https://www.kachelsenhaardenwinkel.nl/winkel/onderdelen/merk/dru/afstandsbediening-mertik-g6r-h4t28-xd/
4. https://docs.flipperzero.one/sub-ghz/frequencies
5. https://www.youtube.com/watch?v=i_TLLACZuRk&ab_channel=JacobSchrum
6. https://github.com/flipperdevices/flipperzero-firmware/blob/27ee0f73f7b81e6791c07f036939015dec9f6a85/documentation/file_formats/SubGhzFileFormats.md
7. https://github.com/flipperdevices/flipperzero-firmware/issues/2260

Exploring how to create binary wheels for Pythonista

This is a follow-up on the previous post on how to get Pip working with Pythonista. We ended with a working pip but didn’t have a way of installing binary packages yet (like scipy and scikit-learn).

Using the Oracle Cloud, which offers (free!) aarch64 intances, I tried to build some Python wheels for my iPhone.

sudo apt install zlib1g-dev make libssl-dev curl
git clone https://github.com/deadsnakes/python3.6
cd python3.6
./configure
make
curl -L https://bootstrap.pypa.io/pip/3.6/get-pip.py > ./get-pip.py
./python get-pip.py
./python -m pip wheel scikit-learn

After uploading it to my PyPI repository we can try to install it using pip.

Except unfortunately, this wheel is still not in the expected format.

This is related to how the wheel file format is specified in PEP 0427. The short summary is that the platform tag can be seen in the filename: {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

To find out what platform tags Pythonista asks for on iPhones, I added a debug line in pip/_internal/index/package_finder.py, and ran pip again.

This resulted in the following list of platform tags:

The Python wheels will need to comply to this expected platform tag. Instead of cp36-cp36m-manylinux2014_aarch64 it looks like it needs to say cp36-cp36-darwin_21_5_0_iphone12,8.

Update: Right after finishing this post I noticed that the platform tag changed to cp36-cp36-macosx_15_0_iphone12,8 (instead of the darwin tag above). This might have been caused by an iOS update.

After running auditwheel fix on the created wheel all relevant system libraries are copied into the wheel.

Now I run the following script to modify the wheel to use the expected platform tag:

import zipfile


with zipfile.ZipFile('scipy-1.5.4-cp36-cp36m-linux_aarch64.whl', 'r') as input_wheel:
    with zipfile.ZipFile('scipy-1.5.4-cp36-cp36-macosx_15_0_iphone12,8.whl', 'w',
                         compression=zipfile.ZIP_DEFLATED) as output_wheel:
        for input_zipinfo in input_wheel.infolist():
            if input_zipinfo.filename.endswith('.dist-info/WHEEL'):
                output_wheel.writestr(
                    input_zipinfo.filename,
                    input_wheel.read(input_zipinfo.filename).replace(
                        b'cp36-cp36m-linux_aarch64',
                        b'cp36-cp36-macosx_15_0_iphone12,8')
                )
            elif input_zipinfo.filename.endswith('.dist-info/RECORD'):
                output_wheel.writestr(
                    input_zipinfo.filename,
                    input_wheel.read(input_zipinfo.filename).replace(
                        b'.cpython-36m-aarch64-linux-gnu',
                        b'')
                )
            else:
                output_wheel.writestr(
                    input_zipinfo.filename.replace('.cpython-36m-aarch64-linux-gnu', ''),
                    input_wheel.read(input_zipinfo.filename)
                )

Now it is recognized by pip as suitable for the platform and installs without issue.

When trying to use the newly installed scipy however, it still can’t find the correct shared objects.

If we try to directly import this shared object using ctypes, we can see better why it will not work:

DLLs need to be Mach-O, instead of the a.out format.

But how does Pythonista include the non-standard-libraries it ships with? To find out, I made a copy of the app itself. This was quite easy to do, since Pythonista ships with Python:

Using this dump, I could determine that the extra packages like numpy and matplotlib all live in Frameworks/Py3Kit.framework/pylib/site-packages. However, in this directory, all shared objects that normally also live there, are missing.

If we decompile the app’s Py3Kit.framework executable, we can see that it actually contains these binary Python modules that were missing in site-packages. They are all added to the built-in Python packages, using the _PyImport_AppendInittab method available in Python’s C API.

void PYK3Interpreter::registerBuiltinModules(ID param_1,SEL param_2)

{
  _PyImport_AppendInittab("speech",_PyInit_speech);
  _PyImport_AppendInittab("reminders",_PyInit_reminders);
  _PyImport_AppendInittab("contacts",_PyInit_contacts);
  _PyImport_AppendInittab("sound",_PyInit_sound);
  _PyImport_AppendInittab("linguistictagger",_PyInit_linguistictagger);
  _PyImport_AppendInittab("_ui",_PyInit__ui);
  _PyImport_AppendInittab("_notification",_PyInit__notification);
  _PyImport_AppendInittab("_pythonista",_PyInit__pythonista);
  _PyImport_AppendInittab("_keyboard",_PyInit__keyboard);
  _PyImport_AppendInittab("_dialogs",_PyInit__dialogs);
  _PyImport_AppendInittab("_appex",_PyInit__appex);
  _PyImport_AppendInittab("_font_cache",_PyInit__font_cache);
  _PyImport_AppendInittab("_scene2",_PyInit__scene2);
  _PyImport_AppendInittab("console",_PyInit_console);
  _PyImport_AppendInittab("_clipboard",_PyInit__clipboard);
  _PyImport_AppendInittab("_photos",_PyInit__photos);
  _PyImport_AppendInittab("_photos2",_PyInit__photos2);
  _PyImport_AppendInittab("_webbrowser",_PyInit__webbrowser);
  _PyImport_AppendInittab("_twitter",_PyInit__twitter);
  _PyImport_AppendInittab("location",_PyInit_location);
  _PyImport_AppendInittab("_motion",_PyInit__motion);
  _PyImport_AppendInittab("keychain",_PyInit_keychain);
  _PyImport_AppendInittab("_cb",_PyInit__cb);
  _PyImport_AppendInittab("_canvas",_PyInit__canvas);
  _PyImport_AppendInittab("_imaging",_PyInit__imaging);
  _PyImport_AppendInittab("_imagingft",_PyInit__imagingft);
  _PyImport_AppendInittab("_imagingmath",_PyInit__imagingmath);
  _PyImport_AppendInittab("_imagingmorph",_PyInit__imagingmorph);
  _PyImport_AppendInittab("_np_multiarray",_PyInit_multiarray);
  _PyImport_AppendInittab("_np_scalarmath",_PyInit_scalarmath);
  _PyImport_AppendInittab("_np_umath",_PyInit_umath);
  _PyImport_AppendInittab("_np_fftpack_lite",_PyInit_fftpack_lite);
  _PyImport_AppendInittab("_np__compiled_base",_PyInit__compiled_base);
  _PyImport_AppendInittab("_np__umath_linalg",_PyInit__umath_linalg);
  _PyImport_AppendInittab("_np_lapack_lite",_PyInit_lapack_lite);
  _PyImport_AppendInittab("_np_mtrand",&_PyInit_mtrand);
  _PyImport_AppendInittab("_np__capi",_PyInit__capi);
  _PyImport_AppendInittab("_mpl__backend_agg",_PyInit__backend_agg);
  _PyImport_AppendInittab("_mpl__image",_PyInit__image);
  _PyImport_AppendInittab("_mpl__path",_PyInit__path);
  _PyImport_AppendInittab("_mpl_ttconv",_PyInit_ttconv);
  _PyImport_AppendInittab("_mpl__cntr",_PyInit__cntr);
  _PyImport_AppendInittab("_mpl_ft2font",_PyInit_ft2font);
  _PyImport_AppendInittab("_mpl__png",_PyInit__png);
  _PyImport_AppendInittab("_mpl__delaunay",_PyInit__delaunay);
  _PyImport_AppendInittab("_mpl__qhull",_PyInit__qhull);
  _PyImport_AppendInittab("_mpl__tri",_PyInit__tri);
  _PyImport_AppendInittab("_counter",_PyInit__counter);
  _PyImport_AppendInittab("_AES",_PyInit__AES);
  _PyImport_AppendInittab("_ARC2",_PyInit__ARC2);
  _PyImport_AppendInittab("_ARC4",_PyInit__ARC4);
  _PyImport_AppendInittab("_Blowfish",_PyInit__Blowfish);
  _PyImport_AppendInittab("_CAST",_PyInit__CAST);
  _PyImport_AppendInittab("_DES3",_PyInit__DES3);
  _PyImport_AppendInittab("_DES",_PyInit__DES);
  _PyImport_AppendInittab("_MD2",_PyInit__MD2);
  _PyImport_AppendInittab("_MD4",_PyInit__MD4);
  _PyImport_AppendInittab("_RIPEMD160",_PyInit__RIPEMD160);
  _PyImport_AppendInittab("_SHA224",_PyInit__SHA224);
  _PyImport_AppendInittab("_SHA256",_PyInit__SHA256);
  _PyImport_AppendInittab("_SHA512",_PyInit__SHA512);
  _PyImport_AppendInittab("_XOR",_PyInit__XOR);
  _PyImport_AppendInittab("strxor",_PyInit_strxor);
  _PyImport_AppendInittab("pykit_io",_PyInit_pykit_io);
  return;
}

In a next post I’ll be looking into compiling the wheels with Mach-O shared libraries (or bundles as Apple calls them).

Adding pip to Pythonista for iOS

Pythonista is probably the most popular Python app for iOS. This post is a summary of the work I did to get pip working. Here’s how to do it:

import requests
import sys
from io import BytesIO
from zipfile import ZipFile

# Get the location of the Python 3 site-packages
site_packages = next(filter(
  lambda x: 'site-packages-3' in x,
  sys.path
))

# extract directly into site-packages
ZipFile(BytesIO(requests.get(
    'https://files.pythonhosted.org/packages/90/a9/1ea3a69a51dcc679724e3512fc2aa1668999eed59976f749134eb02229c8/pip-21.3-py3-none-any.whl'
).content)).extractall(site_packages)

print("Downloaded pip")

This downloads pip to the site-packages folder for Python 3. Pythonista calls this folder site-packages-3.

Now that we have pip set up, we can start downloading our first package:

import pip
import sys

site_packages = next(filter(
  lambda x: 'site-packages-3' in x,
  sys.path
))

print(
  pip.main(f'install ',
           f'--target {site_packages} '
           f'tqdm'.
  split(' '))
)

This works a bit differently from how you would typically use pip. Since we use it as a library, we call the pip.main function with a list of arguments (created by .split(' ')).

The default directory pip tries is not writable. It’s part of the Pythonista app. We therefore manually indicate it should write to our site-packages-3 folder using --target. Note that this probably will not yet work for dependencies with binary extensions (libraries like scipy etc.).

Of course, I also tried to use StaSh. This seemed quite suitable at first, but upon closer inspection, the pip it contained is not the common version. In fact it contains its own pip.py which approximates the canonical pip’s behaviour.

In a next post I’ll explore how to use pip to get binary wheels to install on your iDevice. This will involve building wheels specific for iOS and maybe even setting up a PyPI mirror.

Pyodide requests shim: Binary requests are working!

In the previous post I showed how shimming the Python module requests was done. In the meantime I have made processing binary responses possible, using a slightly weird browser feature that probably still exists for backward compatibility reasons.

Since the requests API is a simple blocking Python call, we can’t use asynchronous fetch calls. This means XMLHttpRequest is the only (built-in) option to perform our HTTP requests in JavaScript (from Python code). So the two challenges are that the requests need to be done with XMLHttpRequest, and they should be synchronous calls.

Normally, if you want to do something with the raw bytes of an XMLHttpRequest, you would simply do:

request = new XMLHttpRequest();
request.responseType = "arraybuffer";
// or  .responseType = "blob";

However, if this responseType is combined with the async parameter set to false in the open call, you get the following error (and deprecations):

request = new XMLHttpRequest(); 
request.responseType = 'arraybuffer'; 
request.open("GET", "https://httpbin.org/get", false); 
request.send();
// Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end 
//     user’s experience. For more help http://xhr.spec.whatwg.org/
// Use of XMLHttpRequest’s responseType attribute is no longer supported in the synchronous mode in window context.
// Uncaught DOMException: XMLHttpRequest.open: synchronous XMLHttpRequests do not support timeout and responseType

The Mozilla docs provide helpful tricks for handling binary responses, back from when the responseTypes arraybuffer and blob simply didn’t exist yet. The trick is to override the MIME type, say that it is text, but that the character set is something user-defined: text/plain; charset=x-user-defined.

request.overrideMimeType("text/plain; charset=x-user-defined");
request.responseIsBinary = true;  // as a custom flag for the code that needs to process this

The request.response we get contains two-byte “characters”, some of which are within Unicode’s Private Use Area. We will need to strip every other byte to get the original bytes back. Note that the following code block contains Python code made for Pyodide. The request object is still an XMLHttpRequest, but it’s accessed from the Python code:

def __init__(self, request):
    if request.responseIsBinary:
        # bring everything outside the range of a single byte within this range
        self.raw = BytesIO(bytes(ord(byte) & 0xff for byte in request.response))

Even though this works right now, some concessions have been made to achieve the goal of performing HTTP requests from Pyodide. The worst concession is running on the main thread, with the potential of freezing browser windows. The future of this project is to write asynchronous Python code using aiohttp, and shim aiohttp to use the Javascript fetch API.

To see all these things in action:

• Check the current state of shimming requests on Github: bartbroere/requests#1
• Try out my custom Pyodide build, from a Starboard Notebook, or configure this as your Pyodide CDN URL: https://python.rijksoverhemd.nl

Making the Python requests module work in Pyodide

The Pyodide project compiles the CPython interpreter and a collection of popular libraries to the browser. This is done using emscripten and results in Javascript and WebAssembly packages. This means you get an almost complete Python distribution, that can run completely in the browser.

Pure Python packages are also pip-installable in Pyodide, but these packages might not be usable if they (indirectly) depend on one of the unsupported Python standard libraries. This has the result that you can’t do a lot of network-related things. Anything that depends on importing socket or the http library will not work. Because of this, you can’t use the popular library requests yet. The project’s roadmap has plans for adding this networking support, but this might not be ready soon.

Therefore I created an alternative requests module specifically for Pyodide, which bridges the requests API and makes JavaScript XMLHttpRequests. I’m currently developing it in a fork at bartbroere/requests#1. Helping hands are always welcome!

Since most browsers have strong opinions on what a request should look like in terms of included headers and cookies, this new version of requests will not always do what the normal requests does. This can be a feature instead of a bug. For example, if the browser already has an authenticated session to an API, you could automatically send authenticated requests from your Python code.

This is the end result, combined with some slightly dirty hacks (in python3.9.js) to make the script MIME type text/x-python evaluate automatically:

<script src="https://pypi.bartbroe.re/python3.9.js"></script>
<script type="text/x-python">
    from pprint import pprint
    import requests
    pprint(requests.get('https://httpbin.org/get', params={'key': 'value'}).json())
    pprint(requests.get('https://httpbin.org/post', data={'key': 'value'}).json())
</script>

Hopefully, my requests module will not have a long life, because the Pyodide project has plans to make a more sustainable solution. Until then, it might be a cool hack to support the up to 34K libraries that depend on requests in the Pyodide interpreter.