We weren't given many constraints related to the chess piece recognition itself. The course instead asks us to implement a CV research paper, and we chose an existing research project which focused on chess piece recognition.
That lack of constraints led us into running face first into issues of generalisation and variability within datasets. As in, exactly what you allude to with limiting the piece sets.
I think in my undergraduate naivety my aspirations were too high with what could reasonably be accomplished. I've spent a lot of time trying to improve an aspect of the project that really didn't need to be improved, which prevented meaningful progress.
Now finals are coming up and I feel terribly stressed. Having trouble functioning. Brain fog, etc. I feel so sad right now.
EDIT: I keep forgetting my password so apparently I have multiple throwaway now. Sorry.