I actually think the linked page makes it very easy to understand my main critique. The main problem here is that downscaling is a destructive process. It destroys information. Zoom in on that sign, can you read it?[0] No! But can you in the high res?[1] Of course!
We can of course train the model on those signs alone and then get it to recognize what the sign should say, the same way you might do this (not by reading words, but by reading the symbol), but we may run into problems when downsampling images, especially with subtle biases that those algorithms can create, which even includes tiling[3].
If the main thesis is "training on larger resolution results in better performance on high resolution images" then this seems to be a conclusion we already knew from a pure mathematical understanding of entropy, and is something many researchers have been discussing for decades.
There are a lot of evaluations here but it is not explicitly clear to me that the architecture is playing the main role. There is very little in the ablation study and a larger focus on dataset coverage. Table 1 is difficult interpret. While I commend the fine tuning of ViT it would not distinguish the entropy problem as (IIRC) VIT was pretrained on 224x224 resolution images and then fine-tuned to a higher resolution. More fine tuning isn't going to make that problem go away. Table 2 helps us understand pooling but does more in terms of dataset coverage than the coverage of solution space.
I don't think it is bad in the way of "this is not a useful thing that was built" but "the way this is communicated makes it difficult for me as a researcher to interpret the reason for these results." In a way, my criticism here is much more general than just this paper. I am frustrated with the recent trends in AI research that there is more focus being put into coverage of datasets over interpretation. Interpretation such as more in depth ablations (e.g. holding variables constant, changing specific parameters for a test[4]). There isn't infinite compute, so I'm not expecting the world. But in the trade-off between dataset coverage and more thorough ablations, I'd significantly prefer the latter. It is entirely possible that the architectural changes here are critical to the model's ability to properly encode the information. There are hints at it in the paper but it is difficult to distinguish form training procedures and simply the entropy. There's many moving parts and the information provided is not enough to distinguish (or distinguish to an acceptable threshold). I don't entirely blame researchers for making their choice in trade-offs, we can't encourage more in depth ablations until reviewers stop using "what about x dataset" as a excuse[5]. This paradigm of dataset coverage really feels like a lot of wasted compute. And honestly, I suspect we'd make far more improvements were we to change paradigms, as well as many of those improvements would come from much smaller labs without these large compute resources.
[0] Small Res: http://0x0.st/8nU3.png
[1] High Res: https://0x0.st/8nUE.png
[2] https://www.cs.cmu.edu/~clean-fid/
[3] https://arxiv.org/abs/2104.05704
[4] It would be nice to change one parameter at a time but sometimes things are coupled.
[5] "I'm curious about performance on x dataset because x dataset has y quality that I think is important" is a perfectly fine critique. But I rarely see that type of criticism in reviews. They include the demand but not the motivation for the demand. Just leads to noisy reviewing as an author can't infer if reviewer is asking because they're lazy or because they think lack of inclusion undermines the author's claims.
I think you missed the part where the word performance is doing double duty here. Performance as in accuracy of the result and performance as in the time it takes to achieve said result.
The expectation is that training on a larger resolution will worsen performance in the second sense. You also mentioned that downsampling images will destroy information, hence FastVLM should also perform worse in the first sense, since it is clearly running its transformer layers on downsampled images through the patch embedding halving the image resolution with each layer.
To be fair, the presented network architecture does not really look like anything special. Three CNN layers with two transformer layers is just good product engineering. The real insight to be had here is that writing your own custom downsampling algorithm is a waste of time. You should make the downsampling learnable and part of the model.
> The expectation is that training on a larger resolution will worsen performance in the second sense.
> downsampling images will destroy information, hence FastVLM should also perform worse in the first sense
Note that their model is smaller. That actually can give us good analysis opportunities, as this suggests what I'm implying: more efficient embedding.
> Three CNN layers with two transformer layers is just good product engineering. The real insight to be had here is that writing your own custom downsampling algorithm is a waste of time. You should make the downsampling learnable and part of the model.
I think we're pretty much in agreement. I just really want to see more ablations