I haven't actually experienced this, someone correct me if I am wrong: you can choose from different policies (LRU, TTL, random) to expire/remove keys, otherwise if you disable maxmemory it will start using the OS virtual memory and slow down.

Redis had a virtual-memory option like you describe at some point, but it was scraped due to poor performance and other issues.