2026-04-09 20:58 UTC
@rysiek @StumpyTheMutt @neurovagrant I work in data-intensive academic HPC, and a small summary of how it works is that:
1. Researcher uploads datasets to facility.
2. Researcher runs software on these inputs to produce outputs.
3. Researcher downloads results to their home institute.
The total and relative size of inputs and outputs varies, but tens of TB is common, hundreds of TB happens. Times hundreds or thousands of researchers. All on different schedules.
Replies (1)
-
@maswan@mastodon.acc.sunet.se 2026-04-09 21:01
@rysiek @StumpyTheMutt @neurovagrant The baseline graphs for bandwidth use are spiky and blocky graphs without any "normal" that you could apply as a baseline. Some run a single rsync slowly chugging along for a week or two at fairly steady rate, some have systems that push hard and can saturate at least 100Gbit/s links. Some have a mix of data types that gives jagged lines. In a large facility, there's dozens of these activities at all times..