Jupyterhub Maintenance Nightmare

I just finally finished what was a week-long nightmare involving Jupyterhub.

It started simply enough. I have always kept my own documentation on “how to do” various things. In this case I had recorded steps to update all the packages in Jupyterhub on an ongoing basis. Jupyterhub lives inside something called Anaconda, or conda for short. Upgrading is supposed to be as simple as “conda upgrade xxx” where xxx is the package to upgrade.

I have been using this process since first installing Jupyterhub on a server back in January 2019.

Last week I started working through the 8 or so packages that I normally update, and one of them failed the update with a blizzard of messages and warnings. But that wasn’t the worst part. The worst part was that it also killed conda dead. Even typing “conda -h”, which should print out a simple help file, instead resulted in a page of error messages warning of missing libraries before failing. Nothing I tried would work – conda was dead.

I grabbed my notes from January, and started to reinstall anaconda/conda. It’s really quite simple: delete the anaconda3 directory and reinstall.

It all worked until I came to the last package; xeus-cling, which provides the C++ support I require. That installation failed with (again) a set of rather bizarre “missing stuff” messages.

I left it at that… after all everything else worked, so I left a message on the xeus-cling github page and waited.

This week I got a reply: conda 4.7.9 was broken. Conda 4.7.10 worked with xeus-cling. I checked, and sure enough I was on version 4.7.9. I updated (conda upgrade –all) and tried xeus-cling. It did not work. Instead, it just hung trying to resolve the ‘environment’.

Eventually, after several trial-and-error sessions, I resolved the situation and once again have a perfectly good, working Jupyterhub. I’m very glad that reinstalling anaconda3 is so simple, as this was done several times until I got the order of things correct.

In a nutshell, you must install anaconda3 (from 2019.03 script) (v 4.6.11), then install xeus-cling before anything else. Only in this way can you avoid the timeout problem with xeus-cling and conda 4.7.10. This is because all installs via conda update conda to the latest version.

Once xeus-cling is installed, everything else installs just fine, and the system is back up and running. Still, it was a nightmare figuring out what was wrong, and I can’t say I’m very impressed with conda for breaking things which were working in pre-production environment.