Harry Potter should not be "free", but as a common cultural artifact it should be available according to the contract between the society and the author. For example, author is entitled to a share of profits from any distribution or derivative work, but the author is NOT allowed to decide who to allow and who to deny to distribute and derive it.

These days it is clear that copyright should only be granted by society in exchange to obligations to make the work available to the society at reasonable terms; as time passes the terms should be more and more permissive.

The copyright owners should not dictate their will to the society which empowered them in the first place when it's clearly hurts the society.

For example, I believe that every song recorded before 1997 should be available for purchase in any music web store (some regulated part of the revenue should still go to the musicians if they can be located, which is what recording industry association can do) And all of those songs should be free to derive and cover (some small regulated part of the revenue on derived song should go to musicians via the same mechanism).

Or else old people would tend to become unavailable at all because you can't locate original authors. And music becoming available is very bad. Much worse than "scary criminal pirate" bad.