Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation.
We ping an IP address from multiple servers across the world and identify the location of the IP address through a process called multilateration. Pinging an IP address from one server gives us one dimension of location information meaning that based on certain parameters the IP address could be in any place within a certain radius on the globe. Then as we ping that IP from our other servers, the location information becomes more precise. After enough pings, we have a very precise IP location information that almost reaches zip code level precision with a high degree of accuracy. Currently, we have more than 600 probe servers across the world and it is expanding.
The publicly available information that you are referring to is sometimes not very reliable in providing IP location data as:
- They are often stale and not frequently updated.
- They are not precise enough to be generally useful.
- They provide location context at an large IP range level or even at organization level scale.
And last but not least, there is no verification process with these public datasets. With IPv4 trade and VPN services being more and more popular we have seen evidence that in some instances inaccurate information is being injected in these datasets. We are happy and grateful to anyone who submits IP location corrections to us but we do verify these correction submissions for that reason.
From my experience with our probe network, I can definitely say that it is far easier and cheaper to buy a server in New York than in any country in the middle of Africa. Location of an IP address greatly influences the value it can provide.
We have a free IP to Country ASN database that you can use in your project if you like.
Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.
But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!
Some questions:
- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?
- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?
- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?
This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik...
I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.
For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.
By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.
But at a previous company I worked at that ran a very large chunk of the internet, we did indexing of nearly the entire internet (even large portions of the dark web) approximately every two weeks. There were about 500 servers doing that non-stop. So, I think it is relatively reasonable if you have 600 servers to do that.
The challenge of being a data provider is that you can use our data in a million ways, and we don't have coverage of all. So, when you come up with questions or ideas, we can help you better.
As you mentioned, audit logs. I highly recommend you look into the ASN field.
The ASN identifies an organization that owns a block of IP addresses. In my experience, I have found that the combination of ASN+Country is the most valuable information you can use in spam and fraud detection. You can fake the IP geolocation information with a VPN. However, it is not as easy to fake the ASN information of the IP address. So, when you use a combination of country + ASN, you can have a robust cybersecurity system.
I know it can be done with CSV but it's not as smooth.
We usually just send users the documentation of ingesting the data in CSV or NDJSON format (Newline Delimited JSON). We don't actually get many requests for data downloads in Parquet format. I think we have a few customers where we deliver the data in parquet format directly to their cloud storage bucket.
But keep an eye out for our emails if we announce the parquet data downloads. I will talk with the folks about this.
BUT, there are some good news.
At least for the free database, we deliver the data directly to data warehouse platforms. Not even storage buckets. And we supply a good amount of documentation.
We have the free database in Snowflake, GCP, Kaggle, and Splitgraph, and we are working on a few more deals. For the free database, atleast, we are working on better things than parquet. Like literally one-click solution to bring the IP data to your data warehouse.
Kaggle: https://www.kaggle.com/code/ipinfo/ipinfo-ip-to-country-asn-...
Snowflake: https://app.snowflake.com/marketplace/listing/GZSTZSHKQ4QY/i...
If you want to use our free IP database on Google Cloud or BigQuery, please send us an email (support@ipinfo.io) and mention that the DevRel sent you from HN. I can easily set you up with the free IP database in GCP/BQ.
We have a simplified explanation of our probe network here: https://ipinfo.io/blog/probe-network-how-we-make-sure-our-da...
The only update is the number of servers is like 600+ now. The probe network is growing extremely rapidly.
Our IP geolocation process is quite complicated, and we have a team of data engineers, infrastructure engineers, and data scientists working on various aspects of it. Therefore, our approach is users can ask us questions, and we will try our best to answer them.
I don't know if that provider terminates long running calls, but the calls would stay up too regardless of tower.
It feels like it couldn't be abused by 'freeloaders', because i'd guess their use-case is viewing other peoples.
You can enter IP addresses on the right side to look up information here: https://ipinfo.io/what-is-my-ip
Additionally, we offer some enjoyable tools that you can use here: https://ipinfo.io/tools
The CLI tool is particularly entertaining.
You can also use our API service without signing up, with a limit of 1000 requests per day.
If you do choose to sign up for a free account, you will receive 50,000 requests per month, free IP databases, a bulk lookup feature, and more.
IP geolocation is mainly used in cybersecurity and marketing analytics. There are many ways to geolocate someone. I once came across a project that could estimate the country a user is from based on their writing style and grammar mistakes. For example, American people sometimes use "should of" instead of "should have". Knowing the geolocation of an IP address isn't super creepy. It's just how things work on the internet.
The data is not derived from the IP address itself, but rather from the process itself. And it's just a ping. Moreover, the majority of the IP addresses are not pingable. So, we rely on other in house statistical and scientific models to estimate the location. The probe infrastructure is extremely complicated and there are billions and billions of IP addresses, which is why we do not have a robust range filter mechanism.
You can implement a dynamic ping blocking mechanism or use our data to find hosting ASNs and block ranges of those ASNs. You can download the database for free: https://ipinfo.io/developers/ip-to-country-asn-database
But I encountered 2 things using ipinfo: Hetzner Server that are in Germany in a fixed location that never moved are sometimes located in another country, for me it was once s Server placed into Moscow and once in South America.
How does this happen?
I guess it is because of IPv4 trading or IP address shuffling.
As far as I know, Hertzer, like many hosting companies, is buying IPv4 addresses around the world. Here is an article on the IPv4 trades:
https://tech.marksblogg.com/ipinfo-free-ip-address-location-...
When a company buys an IP address block or relocates an IP block from one of its data centers to another, the location of those IP addresses changes.
If your IP address is static, but we have made an error in geolocation, I would love to take a closer look. You can email our support (support@ipinfo.io) and send a link to the comment. We can discuss it further from there.
A time series IP database requires a substantial amount of storage and computational cost to query, as I imagine. The city level geolocation data we have is ~1.5 gb in size. IP range data is complicated to query efficiently as you need to understand data platform settings and good amount of computer network math and computer science stuff. Adding a layer of time series complexities on top of that, makes this process quite difficult.
To give you some context of how IP metadata lookups work, you can check out this article
https://ipinfo.io/blog/ip-address-data-in-snowflake/
Even if you keep all your database in a binary format, the computational cost is still non-negligible.
A behavior pattern could be that your IP address is being shuffled around random locations that go beyond the normal location shuffling of an ISP connection.
Also, if your IP range is listed in some public datasets that belong to a VPN service, we could recognize your IP as a VPN.
Please reach out to our support and let us know about this. Thanks
You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation
Use it for small applications, I don't mind. Just don't start sending me 10M requests per day ;-)
That said, for any analytics use cases of this data, be aware that MaxMind will group a lot of what should be unknowns in the middle of a country. Or, in the case the US now, I think they all end up in the middle of some lake, since some farm owners in Butler County, Kansas got tired of cops showing up and sued MaxMind. It can cause odd artifacts unless you filter the addresses out somehow.
1 https://developers.cloudflare.com/support/network/configurin...
2 https://www.maxmind.com/en/geoip-demo
3 https://www.maxmind.com/en/geoip2-city-accuracy-comparison
- Download a few free IP databases - Generate a random list of IP addresses - Do the IP address lookups across all those databases - Identify the IP address that can be pinged - Visit a site that can ping an IP address from multiple server - Sort the results by lowest avg ping time
Then check where the geolocation provider is locating the IP address and what is the nearest server from there.
Would you mind open sourcing the code for that?
export function onRequest(context) {
return new Response(JSON.stringify([parseFloat(context.request.cf.longitude), parseFloat(context.request.cf.latitude)]), {headers: {"Content-Type": "application/json;charset=UTF-8"}})
}
They call it "web experience personalization" in the industry, and it is annoying. I have never recommended anyone to do that. The best way to do website personalization through IP geolocation:
- Taxes and stuff (if applicable)
- Delivery costs (if applicable)
- Putting the user's country first in those country selection drop-down menu
And that's about it from the top of my head. In my experience, these translations never work and only create distractions. Regardless of the positive intention the website has, using Google Translate to create a native language version of the website is just not a good idea.
The example I often use to illustrate this problem is that there are roughly 4 million Norwegian speakers in the world, but 14 million speakers of Catalan. Visit an international website in Spain and you rarely get given the option to have it in Catalan.
Good example is Amazon.es https://www.amazon.es/customer-preferences/edit?from=mobile&...
Few technologies manage to make my day-to-day internet experience than these sorts of databases.
I wish they would just go away.
Websites could just ask me my zipcode on first load instead of guessing it wrong every single time and then burying the flow to fix it behind multiple links and page loads.
Also: There is no way to fix the database to produce the “correct” or “better” answer. I rarely want a website to use my current location.
Instead, I check inventory for stores in places where I will be. This whole space is trying to solve an ill-posed problem.
The best way to build a geolocation service is to have a billion devices that report their location to you at the same time they report their IP to you. That's basically Apple and Google. They have by far the best geolocation databases in the world, because they get constant updates of IP and location.
The trick is basically to make an app where people willingly give you their location, and then get a lot of people to use it. That's the best way to build an accurate geo-location database, and why every app in the world now asks for your location.
4-square had the right idea, they were just ahead of their time.
Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’t allowed to use or record results from Apple Maps’ reverse geo service outside of the context of a live user request (finding nearby restaurants, etc).
> but not ASN
Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.
To clarify, the scenario I described is as follows: 1. Initially, when I open Google Maps in a clean browser it defaults to my real location. 2. I repeatedly browse some other location. 3. When I open Google Maps in a clean browser, it defaults to that other location. The only reason for Google Maps to pick that other location is my map browsing.
Because Cloudflare and Maxmind geolocate me to the exact same longitude/latitude.
The last time I checked (maybe a decade ago [grin]) it worked pretty much perfectly for a country, imperfectly for a region, and better-than-a-coin-toss for city resolution. All the data is free.
I don't think they have it on the site any more, but I used to have a rotating 3D-cube thing (x,y,z were the first 3 octets of the address) for things like known-addresses, recent lookups, etc. I used different colours for different groups (country, continent,...) It was so old it was written as a Java applet. Yeah. I guess if I were to do it again, it'd be WebGL.
--
*: I sold it a long time ago, with the proviso that the data must always remain free. I actually didn't believe the offer at first (it came as an email, and looked like a scam) but it went through escrow.com just fine, and I think we both walked away happy. That was almost 2 decades ago now though.
I can infer certain details from airport codes in node hostnames, for example.
It would also be possible - I guess - to infer locations based on average RTT times, presuming a given node's not having a bad day.
Anyone have any other ideas?
Edit: A couple of troublesome example IPs are 193.142.125.129, 129.250.6.113, and 129.250.3.250. They come up in a UK traceroute - and I believe they're in London - but geolocate all over the world.
Traceroute to those IPs certainly looks like the networking goes to London.
The google IP doesn't respond to ping, but the NTT/Verio ones do. I'd bet if you ping from London based hosting, you'll get single digit ms ping responses, which sets an upper bound on the distance from London. Ping from other hosting in the country and across the channel, and you can confirm the lowest ping you can get is from London hosting, and there you go. It could also be that its connectivity is through London, but it's elsewhere --- you can't really tell.
Check from other vantage points, just to make sure it's not anycast; if you ping 8.8.8.8 from most networks around the world, you'll get something nearby; but these IPs give traceroutes to london from the Seattle area, so probably not anycast (at least at the moment, things can change).
If you don't have hosting around the world, search for public looking glasses at well connected network that you can use for pings like this from time to time.
"TULIP's purpose is to geolocate a specified target host (identified by IP name or address) using ping RTT delay measurements to the target from reference landmark hosts whose positions are well known (see map or table)."
https://tulip.slac.stanford.edu/
But the endpoint it posts to seems dead.
Using RIPE atlas probes to get RTT to the IPs from known locations is close to your idea and probably the best anyway.
If I'm running a popular app/web service, I would have my own AS number and I will have purchased a few blocks of IP addresses under this AS and then I would advertize these addresses from multiple owned/rented datacenters around the world.
These BGP advertisements would be to my different upstream Internet service providers (ISPs) in different locations.
For a given advertisement from a particular location, if you see a regional ISP as upstream, you can make an educated guess that this particular datacenter is in that region. If these are Tier 1 ISPs who provide direct connectivity around the world, then even that guess is not possible.
You can see the BGP relationships in a looking glass tool like bgp.tools – https://bgp.tools/prefix/193.142.125.0/24#connectivity
If you have ability to do traceroute from multiple probes sprinkled across the globe with known locations, then you could triangulate by looking at the fixed IPs of the intermediate router interfaces.
Even this is is defeated if I were to use a CDN like Cloudflare to advertise my IP blocks to their 200+ PoPs and ride their private networks across the globe to my datacenters.
Everyone who's aware of RIPE Atlas has that ability.
I have almost a billion RIPE Atlas credits. A single traceroute costs 60. I have enough credits to run several traceroutes on the entire IPv4 internet. (the smallest possible BGP announcement is /24, so max of 2^24 traceroutes, but in reality it's even less).
They are always multiple states off, and checking multiple different services pretty much never even seem to agree.
You know you can just run a whois query per ip you want to analyze, no point in scraping the whole ipvN space.
Also I only need to scrape as many WHOIS records as there are different networks out there. So for example for the IPv4 address space, there are much less networks as there are IPv4 addresses (2^32).
Also, most RIR's provide their WHOIS databases for download.
Therefore, "scraping" is not really the correct word, it's an hybrid approach, but mostly based on publicly available data from the five RIR's.
Unless you think CSV is a database?
Not the definition of "from scratch" in my book
How'd that happen?
> On one hand, I love that there’s some good alternatives in the geolocation space, but misleading geolocation precision can lead to very undesirable side effects[0].
[0] https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
I built a page to compare IP geolocation providers: https://resolve.rs/ip/geolocation.html
I'll work on adding ipapi.is shortly!
However, when I saw that a few API didn't return any response, I thought maybe the site was not maintained.
[0] https://ipinfo.io/products/free-ip-database
----
Tangent
I find the geographic coordinate values returning up to 15 decimal places is absurd for an IP geolocation response. IP geolocation is never that precise and, this level of "precision" is not warranted and frankly distracting. Like at best it should be 4 decimal places.
relevant xkcd: https://xkcd.com/2170/
If I have logs from 10 years ago, can I look up information about that IP as it was at the time?
[1]: https://git.ipfire.org/?p=location/location-database.git;a=s...
I have heard there is much effort to use BGP data to build GeoIP database.