A Machine Learning Workflow for the iPad Pro

20 minute read

Published:

The iPad is a revolutionary device. I take all my notes with it, read & annotate papers and do most of my conceptual brainstorming on it. But how about Machine Learning applications? In todays post we will review a set of useful tools & venture into the love story of the iPad Pro & the new Raspberry Pi (RPi). Afterwards, you will be empowered to establish your very own “mini”-ML workflow, know the ML-tailored application landscape & get started with setting up a RPi as an iPad-powered USB-C device. More specifically, we will discuss how I integrate a set of useful apps in my daily research routines, how I got started with the iPad & RPi and how one can use the iPad to run larger compute jobs on remote servers. So let’s get started with today’s agenda:

Apps which I use throughout my daily research! :mag:

The iPad Pro has been one my the best financial investments. It is primarily designed for multi-media creatives who need an intuitive and efficient workflow. This includes artists, creators and media editors. Not necessarily people who would like to program. Only recently, Apple has been pushing towards making the iPad Pro a stand-alone computer replacement. The biggest step towards this is the USB-C port which allows for simultaneous charging and data exchange. In combination with the new iPadOS we can start to dream & experiment. Furthermore, recent updates to apps and new launches introduce a plethora of possibilities for the code savy peepz. Here are some of my favourite ones:

Note Taking & Paper Annotation with Notability & Procreate :pencil2:

I use the majority of my “iPad time” reading papers, taking notes at talks or brainstorming sessions for projects. There are many options which allow you to do so. I personally really enjoy Notability. It is very flexible while being minimal in its design. A feature that I greatly appreciate is the ease of cropping text snippets & how you can smoothly include images from presentations/white boards/online search. After being done writing or if you want to share your notes, you can export them to any cloud storage or share them via AirDrop/email. This makes collaborating super easy! I also use Notability for all the visual-ml-notes (check out the repository). Finally, a feature which I am not using enough (but should) is the audio recording & playback of notes. Notability allows you to record while writing. Afterwards, you can simply press play to get a form of video of the creation process of your notes while listening to the audio again. You can “tap” into the note and the audio timeline will automatically jump to the audio that was recording at the time of writing. Pretty neat, right? If I want to be a little more artsy, I like to go with Procreate. Sometimes it is really relaxing to simply loose yourself in some fine details of a drawing (sometimes I wonder whether Sundays were made for “peocreat-inating”). My recent blog thumbnails are illustrated in Procreate. Procreate is a “serious” app for artists (and arts lovers :hugs:) and has tons of features to tune colours, drawing styles, canvases & so on. I got to admit - the shear amount of options was initially a little overwhelming. And that’s where the world wide web came to rescue me. There are soo many awesome YouTube tutorials out there. Even if you are not the most talented at drawing (which I am most definitely not) Procreate has a set of tools to support you (line snapping, different layers, etc.). Something I really enjoy is the mirroring option which makes every little doodle appear as a calligraphic master piece (maybe I am exaggerating a little here).

Connecting to remote machines with blink.sh :eyes:

Often times when I attend conferences, I don’t want to carry my laptop since I know that I won’t be getting serious amounts of focused time to work on projects. But I still might run some jobs or test some hyperparameter intuition. In that case I simply need some way to tunnel into my remote machine so that I can monitor what is going on and start new jobs on demand. And that’s where blink.sh comes in. Blink is a Mosh (mobile shell) application which lets you smoothly ssh into any remote machine of your choice. Blink allows you to setup multiple hosts & shortcuts for your ssh servers. I have setup a couple for a small AWS EC2 instance, my lab’s cluster and my Raspberry Pi (more on that later). Given an LTE connection I can log into the host machine from wherever without any hastle of setting up hotspots. Once you are logged in (and have mastered a terminal emulator like tmux/screen) you have ported all the remote capabilities of an regular Linux OS to the iPad! There is one more awesome feature of Blink I need to tell you about: Using an external monitor and multiple Blink windows. All you have to do is hook up your iPad to an external monitor via USB-C. While usually the external monitor is synced to your iPad, Blink Shell automatically projects a second window onto the external monitor so that you can get full use out of your setup. You can then easily switch between Blink windows using cmd + o and within Blink windows with remote tmux sessions & ctrl + b + arrows. To me this feature is really crucial in making me feel like I can seamlessly switch between doing smaller type tasks in the train & switching into more serious work in an office-type environment.

Blink is minimal in its design and focuses on the mosh/ssh service. A few things that I am looking forward to being integrated (or maybe haven’t discovered yet) are the following:

  • Splitting blink panes: Sometimes one would like to log into two separate machines at the same time. No problem for Blink. But one has to create a new Blink ‘window’ & maneuver between them. Instead I would like to split the current window into multiple panes.
  • A way to vim edit files directly on the iPad itself & integration with the iPadOS file system.
  • Some easy way to rsync files from iPad to a remote machine & the other way around.

Editing and version control with Kodex & WorkingCopy :floppy_disk:

If I want to code something locally on the iPad, this usually involves 4 main steps:

  1. Locally clone a repository from my GitHub using the version control app WorkingCopy.
  2. Edit the necessary files locally using the free & awesome Kodex editor.
  3. Test the code using either a local IDE (such as e.g. Pythonista3 or Juno for Jupyter - see below) or doing so remotely after copying over the files using scp.
  4. commit and push the edits back to your desired repository branch again with WorkingCopy.

WorkingCopy allows you to connect your iPad with your GitHub account and provides all the version control functionalities that one may need. Kodex, on the other hand, is a text editor for the iPad which supports linter utilities for almost all programming languages. You can directly edit files from your cloud storage (Dropbox, etc.) or locally on the iPad. Both, Kodex and WorkingCopy, are free of any charge!

Connecting to Jupyter Servers with Juno Connect :satellite:

So let’s review what we have covered so far: Paper work and artsy stuff, connecting to remote machines & editing/managing code-based projects. What are we missing? Exactly, the favourite application of every data scientist - Jupyter notebooks - and here is how to do Jupyter on the iPad: The Juno collection of apps provides us with two options to setup/connect to Jupyter servers & work with notebooks either locally on the iPad or on remotely:

  1. Juno for Jupyter: The local edition allows you to run notebooks directly on your iPad. You can install packages via the standard PyPi repository. From there on you are free to prototype. Note: Having a full-“mini” environment on your iPad can be helpful, but I personally prefer running things through the Raspberry Pi if I don’t have any internet connection (see below).

  2. Juno Connect for Jupyter: My true all-star is Juno Connect. The connect-version makes it super simple to tunnel into any port through which you have forwarded a remote jupyter server. This is done via launching and exposing the jupyter port via jupyter notebook --ip 0.0.0.0 --port 8899 --no-browser on your remote machine.

The user interface and notebook rendering then looks like this:

Something that I am missing from Juno and would love to see in the future are functionalities of Jupyter Lab which includes support for system shells as well as other nice extensions. One of my favourite features of Jupyter Lab (compared to plain Jupyter notebook) is the built-in terminal multiplexer which allows you to easily switch between command line and notebook editing.

iPad Pro + Raspberry Pi Model 4B = :heart:

The iPad Pro by itself does not have a full operating system and should also not be thought of as a compute machine :robot:. While Apple claims that the 64-bit A12X/A12Z bionic chips are faster than many Windows computers, their performance is optimized for graphics operations such as rendering/4K editing/gaming and not for running Python code. And this is where the Raspberry Pi comes in. The new quad-core RPi model 4 comes in a couple of different variants (1/2/4 GB) RAM - and most importantly with a USB-C connection which can power as well as exchange data simultaneously. It is not bigger than a pack of cigarettes 🚬 & can easily be hooked up to the iPad Pro. So what are the main benefits of this setup?

  1. You are able to prototype small applications - always and everywhere.
  2. You have an extended (and glorified) dongle with you. The RPi has a set of USB 2.0 and 3.0 ports as well as an audio jack.
  3. You have smoother git/version control support through Raspbian Buster/the Linux environment.
  4. You are no longer dependent on a stable internet connection to log into remote servers.
  5. You can mount external SSDs/storage to exchange larger files with the iPad and to have a backup with you at all times.

So how do we make this work?

Equipment, equipment :wrench: - What do we need?!

So what are the ingredients I am currently using in my setup? Here is a short list:

  1. An iPad Pro 3rd (or higher) generation with USB-C.
  2. A RaspberryPi Model 4 again with USB-C. There are many starter packs which give you everything you need to get started for a reasonable price.
  3. A set of basic RPi basics including a micro-SD-card, some heat sinks and a way to keep the Pi cool (for mini-overclocking of the CPU).
  4. A USB-C to USB-C cable that can charge the RPi which we will use to connect the iPad with the RPi. Note that the RPi does not support every USB-C cable. I am using a cable from Anker.
  5. A way to connect the RPi to a screen for an initial boot (e.g. micro-HDMI cable).
  6. A fast external SSD to swap around files & data. (I am using the Samsung T5 with 500 Gigs)
  7. A bluetooth compatible mouse. (I am using the Logitech MX Master 2.)
  8. Additional RPi gadgets & extensions such as sensors and a camera.

The last three are all optional can can smooth out the experience.

Steps to setup the Raspberry Pi :strawberry:

  1. Download a tool for flashing OS images (e.g. Etcher).
  2. Download a Raspbian Buster image and flash it onto the Pi’s Micro-SD. Afterwards, insert the Micro-SD slot of the Pi (on the back of the board).
  3. Put the Raspberry Pi in the case, attach heatsinks and the fan (using pins 4 and 6).
  4. Boot the Pi and set a new password. Afterwards you can connect your RPi to the internet.
  5. Set up Miniconda 3 (e.g. using these instructions).
  6. Set up the Pi as a USB-C device by following these instructions.

Note The correct final instruction should be add sh /root/usb.sh to /etc/rc.local.

The first steps are a lot easier if you have access to an external monitor to check that everything works out as desired.

Steps to setup the iPad Pro :iphone:

At this point you are ready to get everything rolling on the iPad:

  1. Download a shell client. For example my favourite Blink shell.
  2. Connect the Raspberry Pi to the iPad via USB-C. At this point the Pi shoot detect a power source and start booting up. This may take 30 seconds.
  3. Go to the settings, inspect if there is a new “Ethernet” flag and make sure the Pi is detected. If not please make sure that step 6 of setting up the RPi as a USB-C device was successful!
  4. Go to your iPad shell and SSH into the Pi via ssh pi@10.55.0.1 which is the static IP we set before. You should then be able to login using your RPi username and password.

You should have logged into the RPi for the first time and we are now able to dive deeper. The full RPi-iPad setup looks something like this:

Next things to do :information_desk_person:

You have made it & can now start customising all the features according to your needs! Some of my first steps included the following:

  • Setting up LTE usage via creating iPad Hotspot and automatically connecting the RPi to your WiFi. For that I used a shortcut (see final section on tricks of the trade).
  • Install Jupyter and use Juno packages to create a working environment.
  • Setup hosts in the Blink shell and start working on other remote machines.
  • Become comfortable with tmux or screen to work with multiple terminal panes.
  • Become comfortable using a command line-based editor like vim or nano for editing.

In my experience the RPi does not suck too much power from the iPad. Without any power source, a full iPad charge gives me around 5+ hours of RPi usage (at medium compute and with active fan). This is usually more than enough for me. Something that might come in handy can be a USB-C adapter with two USB-C ports. This way you can charge the iPad while at the same time powering the RPi.

Compute, COmPUtE, ComPUTE :chart_with_upwards_trend:

At this point we have discussed two main ways how to do scientific computing with the iPad:

  1. Use a remote server and ssh into it using Blink.sh.
  2. Use Juno Connect as a comfortable way to port your Jupyter server.

And now we have so many more options:

  1. RPi + Miniconda/Docker + Jupyter Notebook Prototyping + Intel’s Movidius Neural Compute Stick

As previously mentioned Juno also lets you connect to a notebook running on your RPi. In order to do so, you simply have to setup Miniconda (with Python >= 3.6) on the RPi and then again run a browser-less notebook process with port forwarding. And you don’t even need any internet connection!

Some Tricks of the Trade :hammer:

Managing files with external SSDs, scp & Netatalk

One of the advantages of the RPi is always having a supercharged adapter dongle with you :smile: This allows you to make better use of the single USB-C port of the iPad and opens up the possibility of exchanging files between the RPi and the iPad. One of the first things I did was to mount a SSD drive to the RPi and to automate the process when booting up. This allows you to explore larger datasets on the RPi & to keep a backup of my main laptop with me at all times. Furthermore, the iPad’s file system allows us to easily pull files from our RPi onto the iPad or any computer.

Using the new iPadOS shortcuts application to integrate things smoothly.

Shortcuts come in really handy when booting & setting up the RPi and more generally during everyday activities. Here are a few of my favourites:

  1. Powering off the RPi via shortcut by TechCraft
  2. Connecting the RPi with WiFi via shortcut by TechCraft
  3. Sharing file size reduced images by Jason Snell

Monitoring Remote Tensorboard Loopback addresses with ngrok

Another cool general tool to have under your belt is ngrok. ngrok allows you to access local servers from a public URL. As the word public may already tell - security is not necessarily guaranteed and may require some additional effort from your side. I mainly use ngrok to implicity forward a port which I use for simple monitoring of tensorboards. The reason for that is that I haven’t found a way to do so solely from within blink shell. After signing up & installing ngrok you will have to add an authentification id to your ngrok config and then tensorboard port forwarding is as simple as:

  • Start tensorboard on a port of your remote machine: tensorboard --logdir experiments/tboards/ --port 6006
  • Expose the port to a public URL (also from the remote machine): ./ngrok http 6006

ngrok will then forward your localhost instance to a web URL with .ngrok.io extension. And you can thirst your learning curve quench from wherever you are :smile:.

Some concluding Thoughts :thought_balloon:

Wow this was quite a bit of text & images :smile:. But there is still soo much more to talk about (e.g. markdown editing with Bear). In the future I look forward to setting up a small Kubernetes cluster on a set of RPis. Currently I don’t really have any use for it, but it would be a great (and not too costly) learning experience - given that I know little about distributed compute. Furthermore, I would love to get JAX to run on the RPi. As of right now there is an open issue to create ARM wheels similar to how TensorFlow can be installed on a RPi. The CPU speed-up could really facilitate some of my mobile Deep Learning projects.

A few final public service announcements:

:point_right: Inspirational props go out to Rob’s (nope another one) YouTube channel TechCraft. He has an awesome video on powering a general RPi setup with the iPad Pro. Check it out!

:point_right: Most of the covered apps and setups cost money. I have always believed in investing into education & tools that make me most productive. But you might think differently or be in a different position. And that is okay. For most of the things I cover in this blog, there are free alternatives & you don’t need an Apple tablet to create a similar workflow.

:point_right: I am not getting paid/gaining anything from the app companies, Apple or anyone for writing this blog (never thought I would write this :bowtie:). I simply like to share things that I love and make my life easier.

Happy tooling with your new setup,

Rob

P.S.: Let me (@RobertTLange) know if I missed out on your favourite tool!