The FNNDSC's Digital Infrastructure

[jennydaman]

A Rant: How it works and what it could be

This article, in the form of loosely connected thoughts, explains the digital infrastructure of the FNNDSC and how it enables (and in some cases, hinders) scientific computing.

Preface: Some Thoughts

The big picture is that we have a bunch of various components configured in various ways. Not everything is the product of intelligent design: some of it, good and bad, are just how things fell into place.

When it comes to information technology (IT), there's the option to "pay someone else to do it," either part of it or all of it. It saves effort and is situationally cheaper. Most importantly, you have someone else to blame when things go wrong, and since you're paying them, they better fix their s***. This is what "enterprise support" typically means.

I'm realizing now that our "stack" (choice of which technologies and protocols to use) reflects an outdated computing workflow. in some ways, it holds the entire field of neuroscience back (a hot take). Some people are "stuck in the past" whereas others "have their heads in the clouds."

Gmail, YouTube, Google Drive, all work under one harmonious system which is so easy that it's got everyone addicted. Why is it that to run FreeSurfer, you need to learn bash and write a script?

Many IT setups for computational science are UNIX-based, meaning they implement designs of a 50+ year old system. In the age of smartphones, tablets, and everything on the web, I wonder whether UNIX is still the best way to go.

Structure

For each section I will explain the need, solution, alternatives, and my opinions for each issue.

1. Identify Service

In isolation, accounts are local to a device. You might be able to log onto your computer, but that username and password won't work on any other computer.

FNNDSC uses OpenLDAP/slapd, an open-source LDAP server. Our slapd server is a single server which is a central network location where identities are stored. This way, you only need to memorize one username+password combo which works across all of our computers.

Our LDAP situation is, in Rudolph's own words, "a ticking time bomb." The slapd software is out of date, however it "works" (most of the time) so nobody bothers to upgrade it. It crashes from time to time, breaking logins and programs on all our computers for a period of time until someone runs systemctl restart slapd.

BCH runs its own identity service, which is not interoperable with ours. That's why logging into an FNNDSC account uses a different login than your BCH account. Literally everything else, such as Outlook, PeopleSoft, E2, ..., use the same username and password, thanks to BCH's identity service which provides "single sign-on" (SSO).

I think BCH uses ActiveDirectory (AD), which is proprietary (not open source).

Why do we run our own LDAP instead or relying on BCH's account? Well, actually it might be easier for everyone if we used BCH's identities instead of our own. Nonetheless, running our own services has advantages: we have greater freedom in what we do, and we do not have to go through the red tape of working together with BCH ISD (Information Services Department). Since ISD implements "enterprise" standards, their security expectations can slow down operations with audits and approval processes. With our LDAP, we can do whatever we want when we want. Create a new user? Done. Delete someone's data? Done.

Dana Farber Cancer Institute, Brigham and Women's Hospital, Beth Deaconess Israel Hospital, and Mass General Hospital are all unified under the Mass General Brigham (MGB, previously known as Partners Healthcare) IT infrastructure. Boston Children's Hospital is the only Harvard Medical School affiliated Longwood area institution which does not participate. The MGB system is efficient in some ways but not others. In general, the economies of scale pay off.

From the point of both computer security and UX (user experience), "identity management" is a nightmare. The less you have to think about it, the fewer passwords you need to memorize, the better. At Dana Farber and the Johns Hopkins Hospital they had* a (should be standard IMO) quality-of-life (QoL) feature: since your desktop login is the same as your email login, when you log into a desktop and then open Email in a web browser, you are automatically logged in.

*It only worked in Internet Explorer, which is now dead, so I don't know whether the feature is still supported.

At the FNNDSC, most non-technical users inaccurately refer to an "LDAP identity" as a "Linux login." For convenience, we'll keep using that term. FNNDSC Linux logins can also be used for SSO with http://cube-next.tch.harvard.edu. A few of the "experiments" I run, such as Granfana, support signing in using your BCH account via Gitlab OAuth. Honestly it's very easy to set up but TBH I suspect that ISD isn't aware of this loophole and they probably wouldn't approve it!

2. Storage

Not only can you log into any computer at the FNNDSC with the same username and password, but also your home directory (/neuro/users/$USER) is the same too! It's even the same on E2, even though E2 accounts use a different login than FNNDSC machines.

We rent storage from BCH Research Computing (RC): we pay them yearly per TB of space, they provide to us storage over the NFS version 3 protocol. RC provides reliability guarantees and they make backups nightly. I think it's a good deal. Storage is mission-critical: if it breaks, you lose everything. That is why we opt to pay for the "enterprise support" RC can provide to us.

NFS has incompatibilities and gotchas, for example we can't use Flatpak.

FNNDSC workstations are "thick clients" meaning they use local disks mounted on / and network-attached home directories. This makes it easy to install new software on individual computers, but managing the cluster as a whole can be more complicated. Moreover, making things easy is not always desirable. We often have to think about how to protect the user from themselves, because users make mistakes and uninformed decisions.

Even though you can log onto different FNNDSC workstations, doing so can break things, because software write configurations into your shared home directory. If the version of software are different (or missing) across different workstations, you get problems.

An alternative to the "thick client" architecture is a "thin client" architecture, e.g. LTSP. The idea is that the entire system is loaded from the network, so that the system administrator has greater control over everything and is able to provide more consistency and reliability at the cost of user freedom.

Another solution to "protecting users from themselves" is for us to use "stateless operating systems" such as Fedora Silverblue, CoreOS, or NixOS, however the learning curve for those systems is way to steep.

2.1. FNNDSC + E2

FNNDSC and E2 use different logins, however the user accounts "look" the same and are able to see the same home directory. This is because the "UID," the ID number of your user, is the same for both the FNNDSC LDAP identity and your E2 identity. Even though they are different identities, simply because the UIDs are the same, the systems consider the two identities to be the same user.

[jennydaman]

Is there a better way?

FNNDSC Slack and Matrix (and in some cases, ChRIS) use different logins. That's annoying!

Scientific computing has its roots in UNIX systems and will always be Linux-based, however so many things these days are moving to the web. I wonder if it would be feasible to have our entire digital infrastructure be "web-first" instead of "terminal-first", using web native technologies such as Oauth instead of LDAP.

[jennydaman]

Other Labs with similar setups

I once worked at the MCIN at McGill, Montreal, CA: https://acelab.ca/

In some ways, FNNDSC's IT is a descendant of the Matrinos Center's IT.