Book rec:
https://www.amazon.com/Concepts-Techniques-Models-Computer-P...
Language & library recs:
Java is actually a pretty shitty language to learn concurrency on, because the concurrency primitives built into the language & stdlib are stuck in the 1970s. There've been some more recent attempts to bolt more modern concurrency patterns on as libraries (java.concurrent is one; Akka is another; Quasar is a third), but you're still very limited by the language definition. Some other languages to study:
Erlang, for message-passing & distributed system design patterns. Go has a similar concurrency model, but not as pure.
Haskell for STM.
Python3.5/ES2017/C#, for async/await & promises. Actually, for a more pure implementation of promises, check out E or the Cap'n Proto RPC framework.
Rust, for mutable borrowing. Rust's concurrency story is fairly unique; they try to prove that data races can't exist by ensuring that only one reference is mutable at once.
JoCaml for the join calculus. Indeed, learning formal models like CSP, the pi-calculus, or the join-calculus can really help improve your intuitions about concurrency.
Hadoop for MapReduce-style concurrency. In particular, learning how you might represent, say, graph algorithms on a MapReduce system is a great teacher. Also look at real-time generalizations of MapReduce paradigms like Storm or Spark.
Paxos & Raft for the thorny problems in distributed consensus.
Vector clocks, operational transforms and CRDTs. One approach to concurrency is to make it not matter by designing your algorithms so that each stage can be applied in arbitrary order (or can compensate for other operations that have occurred in the meantime). That's the idea behind this, and it perhaps has the most long-term promise.
Project & job recs:
The best way to really learn concurrency is to take a job at a company that has to operate at significant scale. Google or Facebook are the most prominent, but any of the recent fast-growers (AirBnB, Uber, Dropbox, probably even Instacart or Zenefits) will have a lot of problems of this type.
Failing that, I've found that implementing a crawler is one giant rabbithole in learning new concurrency techniques. The interesting thing about crawlers is that you can implement a really simple, sequential one in about 15 minutes using a language's standard library, but then each step brings a new problem that you need to solve with a new concurrency technique. For example:
You don't want to wait on the network I/O, so you create multiple threads to crawl multiple sites at once.
You quickly end up exhausting your memory, because the number of URLs found on pages grows exponentially, and so you transition to a bounded thread pool.
You add support for robots.txt and sitemaps. Now you have immutable data that must be shared across threads.
You discover some URLs are duplicates; now you need shared mutable state between your fetch threads.
You start getting 429 and 403 request codes from the sites, telling you to back off and stop crawling them so quickly. Now you need a feedback mechanism from the crawl threads to the crawl scheduler, probably best implemented by message queues.
You want to process the results of the crawl. Now you need to associate the results of multiple fetches together to run analyses on it; this is what MapReduce is for.
You need to write out the results to disk. This is another source of I/O, but with different latency & locking characteristics. You either need another thread pool, or you want to start looking into promises.
You want to run this continuously and update a data store. Now you need to think about transactions.
