One thing on my mind is the notion of a "Star Trek Economy". The idea being that in Star Trek, specifically TNG, the basic needs aren't an issue. Food is provided when needed, entertainment and physical fitness are available, etc. I use the Star Trek metaphor because people around HN are very likely to understand what I mean.
I think that things like living arrangements, mental health and necessities (food,etc) must be provided in a way that allows focus on learning. The larger problem seems to be a cultural one, which isn't an easy fix, but the aforementioned items are a decent start.
From my thoughts, I believe that the approach necessitates starting at a University-type level, with the self-directed learners. From the self-directed learners, we can then extrapolate flexible structure for those that want/need it and expand further to the concept of "classes" or "programs" down the line. I am interested in the fact that graduate level programs in the current model of education tend to go from the rigorous structure of K-12 to very unstructured, research driven.
My personal experience doesn't include extreme poverty, but I will make sure I consider the impact of deficits on the educational process.