undefined | Better HN

0 pointsaaron-santos4y ago0 comments

Thank you for calling this out. It's critically important that people understand the difference between model, code, and paper and what they mean.

It's also important that people understand that even if code is provided, it's commercially useless. From the NVAE license as an example[1]

> The Work and any derivative works thereof only may be used or intended for use non-commercially.

It's a great example of the difference between open source (which it is) and free software which it is not. So we're back to square one where it is probably best to clean-room the implementation from the paper, which is nearly useless to reproduce the model.

[1] https://github.com/NVlabs/NVAE/blob/master/LICENSE

0 comments

sillysaurusx4y ago

Unfortunately, I must call you out too, my friend. With love.

Because it’s crucially important that we protect the scientific method here.

The sole goal is to help people like me reproduce the model. If I can’t reproduce the model, I can’t verify the paper.

When I saw “commercial” and then “open source” in your comment, I said “oh no…”

My duty is to the scientific method, so I don’t care if it’s the most restrictive code on the planet as long as I can use it to reproduce the model in the paper.

Because at that point, I have a baseline for evaluating the paper’s claims.

The reason I assume the paper is false until proven otherwise, is because the paper often doesn’t have enough detail to reproduce the model shown in the videos on this tech demos. Meaning, if they’re the it to help me, the ML researcher, then they’re failing to tell me how to evaluate their claims rigorously.

(That said, it’s breaking my heart that I can’t agree with you here, because I want to so badly. I’ve felt similarly for years that scientific contributions need to be “free as in beer” commercially. But I recognize signs of zealotry when I see them, and I can’t let my personal views creep in, because people like me would stop listening if I was here e.g. arguing vehemently that nVidia needed to be delivering us something commercially viable along with a high quality codebase. The price for entry to the scientific method isn’t so high.)

aaron-santosOP4y ago

That's fine, at least you're open about the knowledge for knowledge's sake position. There's more than one way to judge something.

xwolfi4y ago

It's more than that. Here there's a fake demo we can only consider an advertisement. If they wanted it to be a scientific paper it should be reproducible and counter checked. What is the point of making a paper full of screenshot ?

It's not just a knowledge for knowledge's sake issue here, it's that it's not even knowledge they're publishing. They're publishing nothing.

They would make a license that says the code can only be provided for peer review and counter validation, then that'd be knowledge. Then, the sake of it is another secondary problem.

j / k navigate · click thread line to collapse