The point is the query plan will most probably look totally different. A fulltext search or GIS search is calculated differently than a btree lookup and PostgreSQL choose a complete different joining strategy. This can also happen when combining such a condition with another btree index lookup. It doesn‘t have to be a fault of PostgreSQL, sometimes its not even possible to make it faster. E.g. all fulltext search results need to be calculated and then joined.