There's a significant mental leap here. "I block these IP to conserve my resources, therefore they belong to clueless or malicious organisations". It's wrong in both directions:
* I don't think Google, Bing and other crawlers are inherently malicious, and certainly not clueless. Search engines serve a very important role in the internet. Ditto archive.org, and probably dozens of other bots.
* IP based blocklists work well for honest bots (not malicious, or at least not illegal). Malicious bot operators just buy SIM cards and use regular mobile internet for the crawling (basically unblockable, because the IP may be renewed every day or every hour). And the really malicious actors use residential proxies, i.e. botnets that proxy traffic through normal users' computers. Anyway I wonder how many of those 56MM IP addresses are regular dynamically allocated consumer grade ISPs.
>1-5-2024
For the love of all that is holy, what is this date format.
What a coincidence! Right now from my phone on my carrier's network while traveling in the UK I am unable to reach https://cheapskatesguide.org My phone's IP address is likely on this guy's blacklist of 56M addresses. So I am forever going to remember whatever service this website provides may be arbitrarily unavailable unless I'm on a know good IP address. Overly aggressive blacklists like this are lame, make you lose customers, and go against the Internet's fundamental principle which is about the ability to exchange information with anyone in the world.
"Oh because your dynamic IP address was assigned to and used by shady guys 2 years ago, now you cannot access my website, or all your emails will be flagged as spam."
Don't do that people.
For-profit corporations aren't inherently malicious or benevolent. They are legal structures that maximize return on capital without moral judgement or care about what it does to others. A bit like Cthulhu.
I come from a dd/mm/yyyy country, but you can’t say mm/dd/yyyy is completely illogical.
dd/mm/yyyy isn't objectively better IMO. It's what you're used to, making it easier for you to read. It's not what I'm used to, making it harder for me to read.
yyyy-mm-dd is the best format because it's sortable and unambiguous and standard: https://en.wikipedia.org/wiki/ISO_8601
yyyy/mm/dd isn't standard or commonly used AFAIK, so best to avoid it.
mm-dd-yyyy also isn't standard or commonly used AFAIK, so best to avoid it.
The block doesn't need to be permanent. There are people out there publishing list of IPs known to belong to botnets and they're regularly updated. You can ban an IP for, say, 72 hours, and update your ipset regularly.
But anyway I've got a philosophical question...
If a customer has its computer owned by a botnet operator and that computer connects to a banking website, is the customer legit?
>For the love of all that is holy, what is this date format.
Most commonly used in the US. 'Cause we gotta show it like we say it. I think.
See https://eforms.state.gov/Forms/ds5507.PDF for example, page two. Though again, it's not as common as slashes.
>> -A INPUT -j DROP
Yep, it's all is needed to know about this guy.
Edit: FWIW I think you’re fighting the good fight and I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN.
I like to think world class nits are picked here but you're right - this is one of the more rational neighborhoods.
> I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN.
I kludged some bash together to grep the logs and give me IP hit-counts and network owners. I next struggled to make decisions about the IPs - due to 1) the prevalence of residential proxies and 2) learning one of our widgets refreshes 3x/min.
> In case of a lot of rules, the use of the ngx_http_geo_module module variables is preferable.
Just do your own thing, learn from it, and repeat. That's all we can really want in our projects, on our limited lease on planet Earth. Kudos that you found something to work on.
But then there are residential IP blocks, specially some with dynamic enough IPs or NATed ISPs. Some people of those blocks may have hostile or clueless behaviour, some may be used as proxy because malware or because they intentionally installed some of the residential proxy servers agents. There you may be blocking legitimate visitors, if a few clients of some ISP are very active you may end blocking a lot of innocent people. And, in this case too, you can target the IP blocks of its autonomous system if you feel that from there you only get bad traffic.
But in the end, is your site. you are free to decide to block what you understand that are bad neighbourhoods.
It seems to be a static site. Bots should cause only a neglible amount of traffic per month. My guess would be less than $1.
And aren't there free CDNs for static sites these days? I guess you can just push the whole frontent data (html+assets) into a public git repo, put it behind a github page with custom domain and call it a day?
Apparently, he self-hosts on a Raspberry Pi 3B+. I guess you need to block half the internet if your server has about the same performance as a decade-old smartphone.
It's charming in a way, like someone who daily-drives a classic car.
https://cheapskatesguide.org/articles/self-hosters-nightmare...
There's a certain charm to Rube Goldberg machines as well.
"To worldofmathew on Hacker News,
While I do sincerely appreciate the fact that you have taken the time to post several cheapskatesguide articles to Hacker News, this may have caused them to designate you as a blog spammer and block your posts. In other words, they may think that you own this site and are posting your own articles. May I suggest that you contact a Hacker News moderator and set him straight before you post another cheapskatesguide article?"
For instance, it bothers me that certain networks quite literally do nothing about malicious actors attempting intrusion from those networks. Abuse complaints are ignored. Some people might say, "just run blocklistd", "use a non-standard port", et cetera, but the real issue is that people shouldn't be allowed to attempt intrusion, and when I send an email to a netblock's contacts with 1,000 login attempts, they should remove the accounts linked to that attempted intrusion.
They don't, so I block whole netblocks based on the lack of response and action. Should I also run blocklistd? Sure, and I do, but that's orthogonal to blocking.
So while the person running this site could do things differently and/or better, for certain definitions of better, they're doing what works for them.
Is reporting spam going to make a change in the world? No, but it FEELS better to do it, and it does affect my mailbox. Likewise, is blocking IPs making a change in the world? No, but it probably makes Cheapskate feel more in control of their server.
when I send an email to a netblock's contacts with 1,000
login attempts, they should remove the accounts linked
to that attempted intrusion.
> "I added hyphens to the opening and closing PHP statements to prevent my web server from interpreting them as code."
It could be a static site, and one could also automate the job that this person spends their precious time doing every day, copying and pasting text (IP addresses) that matches a pattern from one place to another. But they seem content, so...
Anyhow, it does not matter how they produce the frontend html+assets. They can still take it all and push it somewhere where it is hosted for cheap or free. "wget -r 127.0.0.1" and a few more lines of code should be all that is needed to autmate it.
As long as you understand the limitations, ramifications, and futility in doing this, I have no problems with it as one of the many tactics to defend your footprint and de-noise your logs, etc.
It's a never ending endeavor and you will see the shifting attack sources from the baddies along with the games on stub blocks, prefix broker IP block swaps, and more.
Just know there is automation and horsepower into that entire attack infrastructure you can't possibly compete with but maybe you can mitigate with the limited time and resources you have and that will be enough to get you through.
Is there a reason why they don't use a firewall?
More a testament to the years of optimization nginx has undergone.
>Is there a reason why they don't use a firewall?
Some other comment said they have a custom error page for blocked requests although I think you could still have the firewall rewrite the destination port and have a listener that serves the custom access denied page there.
And if you want to do the same? For the love of god get a firewall and subscribe to some RBLs like a sane person.
Maybe they will call me naive or compare me to Don Quixote fighting windmills.
Maybe they will call me stupid or paranoid for not using some centralized block list.
Maybe they will object to how I characterize those who employ web-crawling robots.
Ha. He is so wrong. We're going to criticize his nginx config and his use of PHP.
I mean yeah. We'll probably get after that other stuff too but still.
¯\_(ツ)_/¯
Oh, tell me, how much? The whopping $5/month? Oh, maybe this is a high load WordPress/like CMS running on LAMP stack... so $8/month?
> I wrote the following small PHP script to search though my Nginx configuration file and tally up the number of IP addresses that I am blocking.
Holy shit. Blocking bots through nginx configuration, more so, blocking 56M addresses through nginx configuration...
Okay, for those of you who never did the thing or have no idea:
Just use the firewall (most of the time it is built-in in your OS), use some way to tell the firewall about the 'offenders' (eg fail2ban though there are options) and don't ever block something indefinitely, it's totally meaningless, just use timeouts.
If some Bob got his computer infected in 2015 and that computer tried to access /wp-admin.php then there is absolutely no reason to assume what in 2024 the IP address Bob's computer had in 2015 is still 'malicious'.
Automatic activity like the scans, bruteforcing and whatever is all about opportunity. They are searching for an easy opportunities to exploit and scanning a server what actively blocks you even for 30m at time is just pointless, there is way, way more opportunities in other places than wasting ~4 weeks trying to scan this server.
> I have custom 403 and 404 error pages that explain to those who may care why they are being blocked and how to regain access to the website
You should timeout IP for few days instead