So Reddit has decided that plain HTML is unsafe

Reddit-The-Company

If you don’t know Reddit, it basically is the host of many popular forums. And like any company which encourages you to “come for the cats [and] stay for the empathy,” Reddit seems to be in the business of extracting as much value as it can from said forums without completely destroying them.

After all, simply fostering community is not a noble enough goal for the New Tech, and fortunately for Reddit, genuinely human-generated data is now gold in the LLM Age. You are welcome to read about the last time they decided to pluck the metaphorical liver from their communities.

Reddit-The-Search-Results

While I no longer wish to engage with Reddit, I still visit it occasionally, especially in the LLM Age. This is because appending site: reddit.com to a search query is basically a surefire way to find results written by genuine humans. Which, just to be extremely clear, I still find desirable.

Now behind a login… sort of

After doing such a query yesterday, to my absolute delight I was greeted with

I guess this makes me old, but I use the original frontend for Reddit, old.reddit.com. I’m going to try really hard not to preach about why it’s a better frontend, but that’s all it is! A design for Reddit.

Look, I know I’m a fringe user. I use Firefox. I noscript! I am no stranger to being forced off a product that worked just fine because someone decides to no longer support the two people who still use it.

• It happened to my phone of 8 years, which works fine by the way, but is on too outdated of an OS. May it rest in peace in its tiny glory.

• It happened to my tablet of 10 years, which works even better than my phone and holds a charge like champ, for the same reason.

• It happened to the API for Stack Exchange used by my copy of their outdated app long removed from the app store. This one hurt me the most.1

Where I feel like things get personal here is that Reddit is saying that this is all

To keep Reddit safe

Keep Reddit safe from whom exactly? Me? My desire for knowledge??

Safety is what exactly?

So let’s see what they have to say on this matter by going to the announcement, which of course I didn’t see because I don’t read Reddit anymore: https://old.reddit.com/r/modnews/comments/1ujtebf/logging_in_to_use_old_reddit/. Hope you’re logged in.

Old Reddit’s logged-out experience is a significant source of abusive scraping and automated traffic on the platform.

Hmmm, OK. But then why is New Reddit still accessible logged out? Oh, someone asked that.

[Question]: What’s so different about new reddit that people don’t try to scrape that? Seems to me like it would be better to just implement that on old reddit too. Besides, won’t this just cause people to try and scrape new reddit?

[Admin reply]: I was about to type an answer but just saw u/Nestramutat- gave a really eloquent answer in another comment!

[The comment (snipped)]: … To your first question, the shape of malicious traffic is always changing. It’s going to be a constant cat and mouse game as you ban one method, a new one gets developed. It’s easy to see abusive traffic in hindsight, but it’s harder to pre-emptively block it. Given that they’re claiming Old Reddit doesn’t have the modern security stack, this is likely proving to be an even greater challenge…

So it doesn’t have “the modern security stack.” Now I may not have really earned my Full Stack stripes, but I can right click and select Inspect Element so I’d say I’m qualified enough to see why.

Old versus New

Old Reddit

Let’s start by — begrudgingly — logging in to see what is so insecure about old.reddit.com. I’ll use their announcement thread to test. Ahhh, so much nicer.

Let’s check what Old Reddit is doing that makes it so insecure. The best I can guess is that their precious, precious user-created content is available in plain HTML, since that’s basically all Old Reddit does: you don’t even need JS unless you want to load more comments (ask me how I know).

It DLs about 1 megabyte and sends about half a megabyte. It’s not shown there, but the page’s HTML itself comprises most of the response. I’ve certainly seen worse, but what’s with the load time? GitHub loads its massive payload about 4x as fast (relative to size).

Oh… 2 whole seconds of waiting for a reply. Smells of rate-limiting. Or was Reddit always this slow?

Let’s load some more comments. (Reddit never loads all of the comments initially)

Well that was nice and lean, and pretty snappy. I don’t see my secrets being sniffed. All I got here is that Old Reddit is a pretty normal webpage, which I guess makes it insecure in comparison to…

New Reddit

Let’s see why New Reddit is so much better. In case you have unrealistic expectations, let me right them: New Reddit will not load anything more than the post itself without Javascript (JS). That’s probably what makes it more secure.

There’s a lot loading here (about 5x Old Reddit), and this is why my analysis gets rather unscientific. Rather than try to get around the “security things” (whatever that means), I instead tried to do the bare minimum necessary to fetch the content. In browser — I did not feel like writing a scraper.

This led to me basically blocking all requests to domains (including reddit.com) except for

• www.redditstatic.com/js/concat
• www.reddit.com/svc/shreddit/more-comments/
• www.reddit.com/svc/shreddit/comment/

When you do this, the page loads a lot less, but it does load. When you click to load more comments, it spins forever, but I inspected the request fired off and it did get a response with comment text. So as far as I can gather, simply running the JS on the page is sufficient to get enough information to get comments. So I guess that’s what’s stopping the scrapers? Executing Javascript?

Just for the heck of it, let’s load some more comments without the request filter.

Well, that’s certainly less lean than Old Reddit.

In my (again, unscientific) experimenting, I reloaded the page several times and tried to load comments and replies and didn’t get any failures. One time, when I had the request filter off, I got redirected and saw a captcha field sent as a query param, but I couldn’t reproduce that. I don’t know whether you get a captcha if you just gun it directly for the comments.

But I did notice this helpful heartbeat sent back to Reddit every time I scrolled or moved my cursor.

I guess that makes me feel safer?

So what’s safer?

Let’s dispel any notion that there are safety issues arising from a frontend, because that’s pure PR crap. Instead, if we read between the lines, Reddit doesn’t want people scraping (because it’s their gold, dammit!) and they think that shafting a few Old Reddit users will disrupt the scraping enough.

Does this actually stop scraping? I don’t know!

I won’t claim to have proven you can still scrape New Reddit: surely it must be harder, but it seems to just be that scrapers — surprise, surprise — prefer using a leaner form of Reddit. Which, by the way, loads 8x more comments by default (200 vs 25; yes, New Reddit really only loads 25 comments initially, going up to like 35 automatically if you scroll some).

Cynically, it seems like Reddit discovered they can make scraping harder by making your browser load more and work more, which they had already done by rewriting a nice piece of HTML into a 5x more bloated mess of web components or whatever. (I gave up trying not to editorialize, sorry)

The thing is that I wouldn’t have beef with Reddit if they had just quietly issued a 40X/30X error forold.reddit.com. Or if they had said “no one uses this, we’re removing it.” This is upsetting but predictable. Claiming that they need to rug-pull me because of vague assertions about security or scraping that are really not my problem? That makes me mad. I enable JS for Anubis, dammit!

Anyway

I’ll have to ponder whom this blog endangers, seeing as its core functionality — like Old Reddit’s — is serving text in plain HTML. Although maybe it’s fine for me because my blog is my content, and as such is less valuable because it was already mine to begin with. Unlike the stolen hoard of user-created content which Reddit is trying to keep secure from Big Scraping. Finders keepers!

---

But why not…

Log in?

Sure. I might do that. Just let me be angry that I have to, please?

Use “New” Reddit?

I might have to anyway, who knows if they’ll keep supporting Old Reddit. But mark my words, they will close the gates on logged-out New Reddit users if they think they can get away with it.

Use an LL…

Am I not allowed to bemoan the loss of the internet that once was and could still have been? Why must I consult the world’s smartest and most expensive computer just to read what people have to say about my random question? Is it so strange to want to read text written by humans?