logo

drewdevault.com

[mirror] blog and personal website of Drew DeVault git clone https://hacktivis.me/git/mirror/drewdevault.com.git

2025-03-17-Stop-externalizing-your-costs-on-me.md (4728B)

    

  1. ---
  2. title: Please stop externalizing your costs directly into my face
  3. date: 2025-03-17
  4. ---
  6. *This blog post is expressing personal experiences and opinions and doesn't
  7. reflect any official policies of SourceHut.*
  9. Over the past few months, instead of working on our priorities at SourceHut, I
  10. have spent anywhere from 20-100% of my time in any given week mitigating
  11. hyper-aggressive LLM crawlers at scale. This isn't the first time SourceHut has
  12. been at the wrong end of some malicious bullshit or paid someone else's
  13. externalized costs -- every couple of years someone invents a new way of ruining
  14. my day.
  16. Four years ago, we decided to [require payment to use our CI services][0]
  17. because it was being abused to mine cryptocurrency. We alternated between
  18. periods of designing and deploying tools to curb this abuse and periods of
  19. near-complete outage when they adapted to our mitigations and saturated all of
  20. our compute with miners seeking a profit. It was bad enough having to beg my
  21. friends and family to avoid "investing" in the scam without having the scam
  22. break into my business and trash the place every day.
  24. [0]: https://man.sr.ht/ops/builds.sr.ht-migration.md
  26. Two years ago, we threatened to [blacklist the Go module mirror][1] because for
  27. some reason the Go team thinks that running terabytes of git clones all day,
  28. every day for every Go project on git.sr.ht is cheaper than maintaining any
  29. state or using webhooks or coordinating the work between instances or even just
  30. designing a module system that doesn't require Google to DoS git forges whose
  31. entire annual budgets are considerably smaller than a single Google engineer's
  32. salary.
  34. [1]: https://sourcehut.org/blog/2023-01-09-gomodulemirror/
  35. [jj]: https://github.com/jj-vcs/jj/discussions/4849
  37. Now it's LLMs. If you think these crawlers respect robots.txt then you are
  38. several assumptions of good faith removed from reality. These bots crawl
  39. everything they can find, robots.txt be damned, including expensive endpoints
  40. like git blame, every page of every git log, and every commit in every repo, and
  41. they do so using random User-Agents that overlap with end-users and come from
  42. tens of thousands of IP addresses -- mostly residential, in unrelated subnets,
  43. each one making no more than one HTTP request over any time period we tried to
  44. measure -- actively and maliciously adapting and blending in with end-user
  45. traffic and avoiding attempts to characterize their behavior or block their
  46. traffic.
  48. We are experiencing dozens of brief outages per week, and I have to review our
  49. mitigations several times per day to keep that number from getting any higher.
  50. When I do have time to work on something else, often I have to drop it when all
  51. of our alarms go off because our current set of mitigations stopped working.
  52. Several high-priority tasks at SourceHut have been delayed weeks or even months
  53. because we keep being interrupted to deal with these bots, and many users have
  54. been negatively affected because our mitigations can't always reliably
  55. distinguish users from bots.
  57. All of my sysadmin friends are dealing with the same problems. I was asking one
  58. of them for feedback on a draft of this article and our discussion was
  59. interrupted to go deal with a new wave of LLM bots on their own server. Every
  60. time I sit down for beers or dinner or to socialize with my sysadmin friends
  61. it's not long before we're complaining about the bots and asking if the other
  62. has cracked the code to getting rid of them once and for all. The desperation in
  63. these conversations is palpable.
  65. Whether it's cryptocurrency scammers mining with FOSS compute resources or
  66. Google engineers too lazy to design their software properly or Silicon Valley
  67. ripping off all the data they can get their hands on at everyone else's expense…
  68. I am sick and tired of having all of these costs externalized directly into my
  69. fucking face. Do something productive for society or get the hell away from my
  70. servers. Put all of those billions and billions of dollars towards the common
  71. good before sysadmins collectively start a revolution to do it for you.
  73. Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of
  74. this garbage. I am begging you to stop using them, stop talking about them, stop
  75. making new ones, just *stop*. If blasting CO<sub>2</sub> into the air and
  76. ruining all of our freshwater and traumatizing cheap laborers and making every
  77. sysadmin you know miserable and ripping off code and books and art at scale and
  78. ruining our fucking democracy isn't enough for you to leave this shit alone,
  79. what is?
  81. If you personally work on developing LLMs et al, know this: I will never work
  82. with you again, and I will remember which side you picked when the bubble
  83. bursts.