Definitely a real issue, I hit it myself.
Assuming you commit when you stop, why not make the commit message detailed enough to summarize a) where you are, b) what problems you are currently experiencing, and c) where you think you need to go next. Explicitly think what your future self needs.
I certainly value of design notes (even for small projects), but committing regularly and often to a git repo for the project (it doesn't have to be complex, a simple local repo will do) seems like the ideal amount of tracking. It generally works for me.