Rendered at 07:01:00 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
fxwin 20 hours ago [-]
OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol
I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily.
Basically every issue they seem to have is solved by using VSCode + Remote development (Transferring files, Customizing hotkeys, looking at outputs)
>And as for compute, I also realised that there are many small tasks that can actually be run on my machine
Yes, if you have a $3000 Computer, you can do many things that other people would need to use remote compute for. Depending on the "Cluster" they are talking about, the mac might even have better specs lol
jltsiren 16 hours ago [-]
The default VSCode configuration can cause problems in HPC clusters. VSCode likes to crawl the files it can see repeatedly. If you have a few users, it doesn't matter. But with many concurrent users and many files, it can destroy file system performance.
azalemeth 8 hours ago [-]
Most clusters I use are architected assuming you use vim and look at graphics locally. I do. I also use img-cat. I've never liked VSCode anything.
linksnapzz 15 hours ago [-]
Every user should get their own vscode endpoint, on a severely underprovisioned VM with 10mb/sec access to the cluster filesystem.
_notreallyme_ 20 hours ago [-]
Most issues can be solved by RTFM. They don't seem to have ever considered that...
16 hours ago [-]
icedchai 13 hours ago [-]
Don't forget ZScaler and other corporate spyware.
linksnapzz 15 hours ago [-]
>OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol
Sad that it's necessary, but it's still a shitty environment. That employers foist it on their employees doesn't make it better.
>I also wonder how they didn't come across X Forwarding at all?
Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
gregdaniels421 15 hours ago [-]
> Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
During COVID I had the wonderful opportunity to remote into a work computer that had a Matlab license and watch the screen slowly paint in. Each keystroke took multiple seconds to display for me.
linksnapzz 15 hours ago [-]
X was never, ever, designed for high-latency networks. I don't understand what people are thinking when they recommend that use-case for X apps.
tjoff 14 hours ago [-]
x2go can be a bit janky but when setup correctly it can deliver the promise of x-forwarding with good performance even over internet.
Love the ability to just forward one or several applications directly instead of the whole desktop environment. Ability to resume connections are also a godsend.
Seems like that usecase isn't given a lot of effort nowadays.
fxwin 15 hours ago [-]
Yea it's a shitty environment, it's just a "oh sweet summer child" kind of thing to say :)
I agree that X forwarding is a terrible way for GUI apps, but the OP read like they were just looking at logs or plots/visualizations without much interactivity. X forwarding is good enough for that (At least good enough to give it a try over scping files back and forth)
NullifyNAN 20 hours ago [-]
It seems like you’re so close but yet so far. It’s interesting that you concluded that Claude makes it easier to run tasks locally but did not conclude that claude makes it easier to use the cluster. Feels we’re in need of a part 2 where you use CC to automate all of the mundane cluster work.
Also just a hot tip, if you’re able to keep any always-only PC on a university subnet (your PI should have some form of a server) you can use tailscale as a jump proxy and bypass the whole VPN dance.
linksnapzz 14 hours ago [-]
> if you’re able to keep any always-only PC on a university subnet (your PI should have some form of a server) you can use tailscale as a jump proxy and bypass the whole VPN dance.
This is an excellent way to have yr. campus infosec ppl shut the network port on said server, and then take the hardware to get its OS install scrutinized, and get your accounts locked in the process.
Don't do this.
whinvik 19 hours ago [-]
Yeah I came to the same conclusion.
Universities use all kinds of hard to setup stuff which are often custom built. Slurm configs, custom configured HPC servers etc. All of this pays off though if you need to do heavy computations which a local machine won't do.
IMO it would be better to make Claude build out something that makes interacting with the HPC much easier. A web UI, Tailscale whatever.
frantathefranta 17 hours ago [-]
I mean Open OnDemand has been built for that purpose and is deployed in like 2000 locations at this point.
> You know those really annoying shower controls in fancy hotels when you need to adjust two separate knobs for the pressure and temperature separately? That's clunky.
I like being able to control those two variables separately.
eluusive 16 hours ago [-]
I'm so glad I left my previous job. As an SRE, I had to use about 10 tools regularly, and they all had 2 minute timeouts with "SSO" + 2fa. I literally spent about 30 minutes out of every hour just logging in to shit.
The security team wouldn't budge things up to 1 day even.
When you have 2 or 3 factors on every tool -- even things like prometheus. It gets excessive.
linksnapzz 15 hours ago [-]
Security has no responsibility towards production, just for preventing breakins. Ergo, their process will always ratchet towards not letting anyone do anything.
Engine of a lawnmower, brakes of a Rolls-Royce, etc.
n4r9 18 hours ago [-]
Recently Claude Code said my authentication credentials had become invalid and I needed to log in again. I recorded the ridiculous number of steps this took:
- Copy-paste link to browser
- Wait several minutes while page load hangs
- Click refresh a couple of times, page finally loads
- Type in work email
- Redirect to work portal, type in work email again
- Asked for work password: open KeepassXC and type in Keepass database password
- KeepassXC additionally demands that I enter my laptop PIN
- Copy work password into work portal
- Asked for Authenticator code: go look for phone
- Find phone, load up authenticator app, type into work portal
- Wait several minutes while page load hangs
- Click refresh a couple of times, eventually get code to paste back into Claude session.
gandreani 15 hours ago [-]
You can disable the laptop PIN requirement in keepassXC and store the OTP code in the app as well.
Not the best security practice but thought you should know!
fragmede 11 hours ago [-]
Or just use a code and enter a couple of digits to log in and not do that dance.
Qem 18 hours ago [-]
> There is also a layer of friction between doing the computations and then analysing the logs that comp chemistry programs spit out. If you want to visualise orbitals or look at the outputs of your simulations, it's really awkward to run Python whilst being in the cluster, and the cluster has no GUI display.
By the description it appears computational chemistry is stuck in batch processing era. They are yet to discover their equivalent to REPL tight feedback loop workflow.
sseagull 18 hours ago [-]
I’m a computational chemist who has been working on this kind of stuff for a while, and I will say it’s just a weird field. The overall issue is that it’s still very academic and very very fragmented. People have tried to make businesses out of this stuff but it has rarely worked, and there isn’t enough money/funding to sustain the level of work needed to really advance the field in that way. Especially now that government funding is drying up and business are reluctant to spend on fundamental research like this.
StableAlkyne 16 hours ago [-]
> People have tried to make businesses out of this stuff but it has rarely worked
The only two startups I'm aware of who have managed to make compchem-as-a-service a viable business are Mat3ra and Materials Square. Both are a little rough in their own ways (to be expected from startups), but it's still interesting to see some of this move beyond university clusters.
StableAlkyne 16 hours ago [-]
> it appears computational chemistry is stuck in batch processing era
I wouldn't call it "stuck," since it's one of the few fields where batch processing makes sense.
You generally run your jobs on a supercomputer that's being shared by a few hundred other people. You need a scheduling system to allocate resources (e.g., 5 jobs requesting 10 nodes for 8 hours, 1000 jobs requesting 0.5 node for 5 minutes, etc).
It's hard to tighten the workflow when the average calculation takes 2-3 hours of time.
theideaofcoffee 14 hours ago [-]
Like another comment has mentioned, the field doesn't really lend itself to quick iterating jobs and loops and favors the long batch processes, for several reasons. Oftentimes, a structure calculation or optimization, if run using a "cheap" (computationally-speaking, say a semi-empirical or molecular mechanical) model, those can be done in seconds or minutes, sure. But for calculations that wish to approach or approximate characteristics of a structure seen in the real world, those can takes days or sometimes weeks, depending on the level of theory. There's also the interaction between different programs, or "codes" as they're called, feeding the output of one into another program to look at different phenomena, which is ideal for a batch system, especially if you are exploring a set of related structures or functionalities.
You will see a lot of similar procedures in, say, AI model training and refinement. Those depend on batch jobs as well, pretty heavily. So the batch process isn't really restricted to the weirdness of academia. My source there is actually working on the systems doing training for frontier models...
(Please forgive some of my hand-waviness or vagueness, it's been nearly 20 years since I last touched a lot of this).
selimthegrim 18 hours ago [-]
You can fly by wire SLURM via Claude Code and bash fairly easily or run MATLAB batch over ssh tunnel etc
angry_octet 7 hours ago [-]
I agree the local computation can be a win, a MacBook has a crazy amount of compute. Data visualisation from a local PCIe SSD is way faster.
With a bit of automation you can work locally, submit big jobs to the remote cluster, and transfer the data back automatically. It can help to have an always-on computer that you control. For me this is:
[macbook]*tailscale*[my server]---ssh---[cluster]
This solves the problems of pushing data back to a laptop that might be turned off or moving location, on 4G etc. While a job is running, output files start streaming back with rsync.
lubitelpospat 16 hours ago [-]
At some point I made a tool called Orbit (here https://github.com/vertices-solutions/orbit) that works as a local interface to a Slurm.
It reached a stage when it does everything I need, but I don't know if it's a good fit to others. OP, if you want to run Slurm jobs with them feeling more local - come check it out!
dtj1123 17 hours ago [-]
A modern terminal emulator like Ghostty or Kitty can render images, even over ssh. And a tui editor like Neovim or Helix works over ssh just fine. If you need something more interactive, open a Jupyter or Marimo notebook.
Working on a cluster can be a pleasure if you use the right tools.
iyihz 20 hours ago [-]
i think instead of trying to do more of the work locally, it might be worth considering trying to work more frictionlessly within the cluster environment instead? e.g. using claude to script visualizing the results and transferring them to your local environment for convenient inspection, or even designing a local GUI for running the jobs and inspecting the results?
it doesn't handle the VPN/2FA friction I guess, but I'd think unless most work can be done locally it would be better to invest in making cluster work more convenient
smitty1e 20 hours ago [-]
Living this dream with the CI/CD pipeline deploying to a cloud system.
It's really, really hard to shrink the problem and troubleshoot with large, opaque, containerized chunks of functionality and services.
Can I just connect to a DB in a terminal and run some SQL to figure out the problem?
Why, no: no, you cannot.
DarmokJalad1701 14 hours ago [-]
How is OP able to access university computing resources after they have left the university?
ordersofmag 16 hours ago [-]
In my previous life I did similar computational physics work with similar environments. I ran into a similar set of problems and found that with a little work one can cobble together tools to automate all the steps that are a hassle. And then the hassle goes away. And this was pre-Claude Code. And pre-Google. And actually pre-ssh. Good old telnet for me. Some shell script or a little AppleScript. Granted no 2FA in those day. But the same learned helplessness with people sitting in front of a device who's entire purpose is to automate things and when presented with a task that would be better automated they fail to use the tools at hand to solve the problem. Kids these days (and back in those days too for that matter).
krautburglar 18 hours ago [-]
man 5 ssh_config
/ControlMaster
tjoff 18 hours ago [-]
how would multiplexing ssh connections help for this?
angry_octet 18 hours ago [-]
Not a help for the VPN part, but I have to 2FA for SSH, with an app or YubiKey, and that would be tedious to set up with multiple shells. With a control channel it's instant login for subsequent shells, sshfs mounts, etc.
tjoff 17 hours ago [-]
Sure, but that is not a problem listed in the post.
gosub100 19 hours ago [-]
My work has the vpn simplified to just typing a single code from an app, just an 8 digit number. It's not that bad, but I despise the additional latency the VPN adds.
I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily.
Basically every issue they seem to have is solved by using VSCode + Remote development (Transferring files, Customizing hotkeys, looking at outputs)
>And as for compute, I also realised that there are many small tasks that can actually be run on my machine
Yes, if you have a $3000 Computer, you can do many things that other people would need to use remote compute for. Depending on the "Cluster" they are talking about, the mac might even have better specs lol
Sad that it's necessary, but it's still a shitty environment. That employers foist it on their employees doesn't make it better.
>I also wonder how they didn't come across X Forwarding at all?
Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
During COVID I had the wonderful opportunity to remote into a work computer that had a Matlab license and watch the screen slowly paint in. Each keystroke took multiple seconds to display for me.
Love the ability to just forward one or several applications directly instead of the whole desktop environment. Ability to resume connections are also a godsend.
Seems like that usecase isn't given a lot of effort nowadays.
I agree that X forwarding is a terrible way for GUI apps, but the OP read like they were just looking at logs or plots/visualizations without much interactivity. X forwarding is good enough for that (At least good enough to give it a try over scping files back and forth)
Also just a hot tip, if you’re able to keep any always-only PC on a university subnet (your PI should have some form of a server) you can use tailscale as a jump proxy and bypass the whole VPN dance.
This is an excellent way to have yr. campus infosec ppl shut the network port on said server, and then take the hardware to get its OS install scrutinized, and get your accounts locked in the process.
Don't do this.
Universities use all kinds of hard to setup stuff which are often custom built. Slurm configs, custom configured HPC servers etc. All of this pays off though if you need to do heavy computations which a local machine won't do.
IMO it would be better to make Claude build out something that makes interacting with the HPC much easier. A web UI, Tailscale whatever.
https://www.openondemand.org/
I like being able to control those two variables separately.
The security team wouldn't budge things up to 1 day even.
When you have 2 or 3 factors on every tool -- even things like prometheus. It gets excessive.
Engine of a lawnmower, brakes of a Rolls-Royce, etc.
- Copy-paste link to browser
- Wait several minutes while page load hangs
- Click refresh a couple of times, page finally loads
- Type in work email
- Redirect to work portal, type in work email again
- Asked for work password: open KeepassXC and type in Keepass database password
- KeepassXC additionally demands that I enter my laptop PIN
- Copy work password into work portal
- Asked for Authenticator code: go look for phone
- Find phone, load up authenticator app, type into work portal
- Wait several minutes while page load hangs
- Click refresh a couple of times, eventually get code to paste back into Claude session.
Not the best security practice but thought you should know!
By the description it appears computational chemistry is stuck in batch processing era. They are yet to discover their equivalent to REPL tight feedback loop workflow.
The only two startups I'm aware of who have managed to make compchem-as-a-service a viable business are Mat3ra and Materials Square. Both are a little rough in their own ways (to be expected from startups), but it's still interesting to see some of this move beyond university clusters.
I wouldn't call it "stuck," since it's one of the few fields where batch processing makes sense.
You generally run your jobs on a supercomputer that's being shared by a few hundred other people. You need a scheduling system to allocate resources (e.g., 5 jobs requesting 10 nodes for 8 hours, 1000 jobs requesting 0.5 node for 5 minutes, etc).
It's hard to tighten the workflow when the average calculation takes 2-3 hours of time.
You will see a lot of similar procedures in, say, AI model training and refinement. Those depend on batch jobs as well, pretty heavily. So the batch process isn't really restricted to the weirdness of academia. My source there is actually working on the systems doing training for frontier models...
(Please forgive some of my hand-waviness or vagueness, it's been nearly 20 years since I last touched a lot of this).
With a bit of automation you can work locally, submit big jobs to the remote cluster, and transfer the data back automatically. It can help to have an always-on computer that you control. For me this is:
[macbook]*tailscale*[my server]---ssh---[cluster]
This solves the problems of pushing data back to a laptop that might be turned off or moving location, on 4G etc. While a job is running, output files start streaming back with rsync.
Working on a cluster can be a pleasure if you use the right tools.
it doesn't handle the VPN/2FA friction I guess, but I'd think unless most work can be done locally it would be better to invest in making cluster work more convenient
It's really, really hard to shrink the problem and troubleshoot with large, opaque, containerized chunks of functionality and services.
Can I just connect to a DB in a terminal and run some SQL to figure out the problem?
Why, no: no, you cannot.