His comment immediately after describes exactly what happened:
> Even before it has ceased to exists, the MPEG engine had run out of steam – technology- and business wise. The same obscure forces that have hijacked MPEG had kept it hostage to their interests impeding its technical development and keeping it locked to outmoded Intellectual Property licensing models delaying market adoption of MPEG standards. Industry has been strangled and consumers have been deprived of the benefits of new technologies. From facilitators of new opportunities and experiences, MPEG standards have morphed from into roadblocks.
Big companies abused the setup that he was responsible for. Gentlemen's agreements to work together for the benefit of all got gamed into patent landmines and it happened under his watch.
Even many of the big corps involved called out the bullshit, notably Steve Jobs refusing to release a new Quicktime till they fixed some of the most egregious parts of AAC licencing way back in 2002.
https://www.zdnet.com/article/apple-shuns-mpeg-4-licensing-t...
It was sweet to see “over the Net”…
https://theonion.com/new-5-000-multimedia-computer-system-do...
I also remember when they went through and re-encoded all of the videos so they could play on the original model iPhone.
More context for this: Chiariglione has been extremely vocal that FRAND patent royalties are entirely necessary for the development of video compression tools, and believes royalty-free standards outpacing the ones that cost money represents the end of innovation in video codecs.
To be clear, Chiariglione isn't opposed to royalty-free standards at all, he just wants them to be deliberately worse so that people who need better compression will pay independent researchers for it. His MPEG actually wound up trying to make such a standard: IVC. You've never heard of MPEG IVC because Samsung immediately claimed ownership over it and ISO patent policy does not allow MPEG to require notice of what specific patents to remove so long as the owner agrees to negotiate a license with a patent pool.
You might think at this point that Chiariglione is on the side of the patent owners, but he's actually not. In fact, it's specifically those patent owners that pushed him out of MPEG.
In the 90s, patent owners were making bank off MPEG-2 royalties, but having trouble monetizing anything newer. A patent pool never actually formed for H.263, and the one for MPEG-4 couldn't agree on a royalty free rate for Internet streaming[0]. H.264 practically is royalty free for online video, but that only happened because Google bought On2[1] and threatened to make YouTube exclusively serve VP8. The patent owners very much resent this state of affairs and successfully sabotaged efforts at MPEG to make dedicated royalty-free codecs.
The second and more pressing issue (to industry, not to us) is the fact that H.265 failed to form a single patent pool. There's actually three of them, thanks to skulduggery by Access Advance to force people to pay for the same patent license twice by promising a sweetheart licensing deal[2] to Samsung. I'm told H.266 is even more insane, mostly because Access Advance is forcing people to buy licenses in a package deal to cover up the fact that they own very little of H.266.
Chiariglione is only pro-patent-owner in the narrow sense that he believes research needs to be 'paid for'. His attempt to keep patent owners honest got him sidelined and marginalized in ISO, which is why he left. He's actually made his own standards organization, with blackjack and hookers^Wartificial intelligence. MPAI's patent policy actually requires companies agree to 'framework licenses' - i.e. promise to actually negotiate with MPAI's own patent pool specifically. No clue if they've actually shipped anything useful.
Meanwhile, the rest of the Internet video industry coalesced around Google and Xiph's AV1 proposal. They somehow manage to do without direct royalty payments for AV1, which to me indicates that this research didn't need to be 'paid for' after all. Though, the way Chiariglione talks about AV1, you'd think it's some kind of existential threat to video encoding...
[0] Practically speaking, this meant MPEG-4 ASP was predominantly used by pirates, as legit online video sites that worked in browsers were using Flash based players, and Flash only supported H.263 and VP6.
[1] The company that made VP3 (Theora) and VP6
[2] The idea is that Samsung and other firms are "net implementer" companies. They own some of H.265, but they need to license the rest of it from MPEG-LA. So Access Advance promised those companies a super-low rate on the patents they need if they all pulled out of MPEG-LA, and they make it up by overcharging everyone else, including making them pay extra if they'd already gotten licenses from MPEG-LA before the Access companies pulled out of it.
Codec development is slow and expensive becuase you can't just release a new codec, you have to dance around patents.
In the early days of MPEG codec development was difficult, because most computers weren't capable of encoding video, and the field was in its infancy.
However, by the end of '00s computers were fast enough for anybody to do video encoding R&D, and there was a ton of research to build upon. At that point MPEG's role changed from being a pioneer in the field to being an incumbent with a patent minefield, stopping others from moving the field forward.
As long as IP law continues in the same form, the alternative to that is completely closed agreements among major companies that will push their own proprietary formats and aggressively enforce their patents.
The fair world where everyone is free to create a new thing, improve upon the frontier codecs, and get a fair reward for their efforts, is simply a fantasy without patent law reform. In the current geopolitical climate, it's very very unlikely for nations where these developments traditionally happened, such as US and western Europe, to weaken their IP laws.
The point is, if there had been no incentives to develop codecs, there would have been no MPEG. Other people would have stepped into the void and sometimes did, e.g. RealVideo, but without legal IP protection the codecs would just have been entirely undocumented and heavily obfuscated, relying on tamper-proofed ASICs much faster.
I disagree. Video is such a large percentage of internet traffic and licensing fees are so high that it becomes possible for any number of companies to subsidize the development cost of a new codec on their own and still net a profit. Google certainly spends the most money, but they were hardly the only ones involved in AV1. At Mozilla we developed Daala from scratch and had reached performance competitive with H.265 when we stopped to contribute the technology to the AV1 process, and our team's entire budget was a fraction of what the annual licensing fees for H.264 would have been. Cisco developed Thor on their own with just a handful of people and contributed that, as well. Many other companies contributed technology on a royalty-free basis. Outside of AV1, you regularly see things like Samsung's EVC (or LC-EVC, or APV, or...), or the AVS series from the Chinese.... If the patent situation were more tenable, you would see a lot more of these.
The cost of developing the technology is not the limitation. I would argue the cost to get all parties to agree on a common standard and the cost to deploy it widely enough for people to rely on it is much higher, but people manage that on a royalty-free basis for many other standards.
Daala was never meant to be widely adopted in its original form — its complexity alone made that unlikely. There’s a reason why all widely deployed codecs end up using similar coding tools and partitioning schemes: they’re proven, practical, and compatible with real-world hardware.
As for H.265, it’s the result of countless engineering trade-offs. I’m sure if you cherry-picked all the most experimental ideas proposed during its development, you could create a codec that far outperforms H.265 on paper. But that kind of design would never be viable in a real-world product — it wouldn’t meet the constraints of hardware, licensing, or industry adoption.
Now the following is a more general comment, not directed at you.
There’s often a dismissive attitude toward the work done in the H.26x space. You can sometimes see this even in technical meetings when someone proposes a novel but impractical idea and gets frustrated when others don’t immediately embrace it. But there’s a good reason for the conservative approach: codecs aren’t just judged by their theoretical performance; they have to be implementable, efficient, and compatible with real-world constraints. They also have to somehow make financial sense and cannot be given a way without some form of compensation.
H.264 was something like >90% of all video a few years ago and wasn't it free for streaming if the end user wasn't paying? IIRC someone also paid the fees for an open source version. There were pretty good licensing terms available and all the big players have used it extensively.
Anyway, my point was only that expecting Google to develop every piece of tech in the world and give it all away for free isn't a general model for tech development, whereas IP rights and patent pools are. The free ride ends the moment Google decide they need more profit, feel threatened in some way or get broken up by the government.
Maybe you don’t remember the way that the gif format (there was no jpeg, png, or webp initially) had problems with licensing, and then years later having scares about it potentially becoming illegal to use gifs. Here’s a mention of some of the problems with Unisys, though I didn’t find info about these scares on Wikipedia’s GIF or Compuserve pages:
https://www.quora.com/Is-it-true-that-in-1994-the-company-wh...
Similarly, the awful history of digital content restriction technology in-general (DRM, etc.). I’m not against companies trying to protect assets, but data assets historically over all time are inherently prone to “use”, whether that use is intentional or unintentional by the one that provided the data. The problem has always been about the means of dissemination, not that the data itself needed to be encoded with a lock that anyone with the key or means to get/make one could unlock nor that it should need to call home, basically preventing the user from actually legitimately being able to use the data.
The GIF page on wikipedia has an entire section for the patent troubles https://en.wikipedia.org/wiki/GIF#Unisys_and_LZW_patent_enfo...
I don't know about video codecs but MP3 (also part of MPEG) came out of Fraunhofer and was paid by German tax money. It should not have been patented in the first place (and wasn't in Germany).
The release of VP3 as open source predates Google's later acquisition of On2 (2010) by nearly a decade.
(I know nothing about the legal side of all this, just remembering the time period of Ubuntu circa 2005-2008).
Audio and video codecs, document formats like PDF, are all foundational to computing and modern life from government to business, so there is a great incentive to make it all open, and free.
Basically MBA drool material.
My expectation from experience when implementing something from a DSP paper is that the result will be unreproducable without contacting the authors for some undisclosed table of magic constants. After obtaining it, the results may match but only for the test images they reported results on. Results on anything else will be much worse.
Also it's normal for techniques from the literature to have computational/memory bandwidth costs two orders of magnitude greater than justified for even their (usually exaggerated) stated levels of performance.
And then their comparison points are almost always inevitably implemented so naively as to make the comparison useless.
It's always difficult because improvements in this domain (like many other engineering domains) are significantly about tradeoffs ... and tradeoffs are difficult to weigh in a pure research environment without the context of concrete applications. They're also difficult to weigh with implementation cleverness having such a big impact particularly since industry heavily drains academia of naturally skilled software engineers.
And as other comments have pointed out, academia is in some sense among the worst of the patent abusers. They'll often develop technology just far enough to lay patent mines around the field, but not far enough to produce something useful out of it. The risk that you spend the significant effort to turn a concept into something usable only to have some patent holder show up with a decade old patent to shake you down is a big incentive against investment.
Until the new codec comes to together all those small optimizations aren’t really worth much, so it’s a long term research project with potentially zero return on investement.
And yes, most of the small optimizations are patented, something that I’ve come to understand isnt’t viewed very favorably by most.
If that stuff worked better, linux would have failed entirely, instead near everyone interfaces with a linux machine probably hundreds if not thousands of times a day in some form. Maybe millions if we consider how complex just accessing internet services is and the many servers, routers, mirrors, proxies, etc one encounters in just a trivial app refresh. If not linux, then the open mach/bsd derivatives ios uses.
Then looking even previous to the ascent of linux, we had all manner of free/open stuff informally in the 70s and 80s. Shareware, open culture, etc that led to today where this entire medium only exists because of open standards and open source and volunteering.
Software patents are net loss for society. For profit systems are less efficient than open non-profit systems. No 'middle-man' system is better than a system that goes out of its way to eliminate the middle-man rent-seeker.
No, just no. We've had free community codec packs for years before Google even existed. Anyone remember CCCP?
We started CCCP because at the time, anime fansubs were predominantly traded on P2P filesharing services like Kazaa, Gnutella, eDonkey, Direct Connect, and later Bittorrent. The most popular codec pack at the time was K-Lite / Kazaa Codec Pack which was a complete and utter mess, and specifically for fansubbing, it was hard to get subtitles to work properly unless they were hard embedded. Soft-subbing allowed for improvements, and there were a lot of improvements to subtitling in the fansubbing community over the years, one of the biggest came when the Matroska (MKV) container format came about, that allowed arbitrarily different formats/encodings to share a single media container, and the community shifted almost entirely to ASS formatted subtitles, but because an MKV could contain many different encodings, any given MKV file may play correctly or not on any given system. CCCP was intended to provide an authoritative, canonical, single-source way to play fansubbed anime correctly on Windows, and we achieved that objective.
But let's be clear, nobody involved was under any illusions that the MPEG-LA or any other license holders of for instance h264 were fans of our community or what we're doing. Anime fansubbing at all came out of piracy of foreign-language media into the English market via the Internet and P2P filesharing. None of us gave a shit, and the use of Soviet imagery in the CCCP was exactly a nod to the somewhat communist ideal that knowledge and access to media should be free, and that patent encumbering codecs and patenting software isn't just stupid, it's morally wrong. I still strongly feel software patents are evil.
Nonetheless, at no point was CCCP through it's life fully legal/licenses appropriately for usage, and effectively nobody cared, not even the licensing authorities, because the existence of these things made their licenses for encoders more valuable for companies producing media, as it was easier for actual people to consume.
And regarding ”royalty-free” codecs please read this https://ipeurope.org/blog/royalty-free-standards-are-not-fre...
Such a huge catch that the companies that offer you a royalty-free license, only do so on the condition that you're not gonna turn around and abuse your own patents against them!
How exactly is that a bad thing?
How is it different from the (unwritten) social contracts of all humans and even of animals? How is it different from the primal instincts?
Unsurprisingly companies that are losing money because their rent-seeking on media codecs is now over will spread FUD [0] about royalty free codecs.
[0] https://en.wikipedia.org/wiki/Fear%2C_uncertainty_and_doubt
Not to mention the computer clusters to run all the coding sims, thousands and thousands of CPUs are needed per research team.
People who are outside the video coding industry do not understand that it is an industry. It’s run by big companies with large R&D budgets. It’s like saying ”where would we be with AI if Google, OpenAI and Nvidia didn’t have an iron grip”.
MPEG and especially JVET are doing just fine. The same companies and engineers who worked on AVC, HEVC and VVC are still there with many new ones especially from Asia.
MPEG was reorganized because this Leonardo guy became an obstacle, and he’s been angry about ever since. Other than that I’d say business as usual in the video coding realm.
(The answer is that most of the work would be done by companies who have an interest in video distribution - eg. Google - but don't profit directly by selling codecs. And universities for the more research side of things. Plus volunteers gluing it all together into the final system.)
Our industry has come to take Google's enormous corporate generosity for granted, but there was zero need for it to be as helpful to open computing as it has been. It would have been just as successful with YouTube if Chrome was entirely closed source and they paid for video codec licensing, or if they developed entirely closed codecs just for their own use. In fact nearly all Google's codebase is closed source and it hasn't held them back at all.
Google did give a lot away though, and for that we should be very grateful. They not only released a ton of useful code and algorithms for free, they also inspired a culture where other companies also do that sometimes (e.g. Llama). But we should also recognize that relying on the benevolence of 2-3 idealistic billionaires with a browser fetish is a very time and place specific one-off, it's not a thing that can be demanded or generalized.
In general, R&D is costly and requires incentives. Patent pools aren't perfect, but they do work well enough to always be defining the state-of-the-art and establish global standards too (digital TV, DVDs, streaming.... all patent pool based mechanisms).
The question is more, "who would write the HTTP spec?" except instead of sending text back and forth you need experts in compression, visual perception, video formats, etc
You might be misunderstanding that almost all of Linux development is funded by the same kind of companies that fund MPEG development.
It's not "engineers in their basement", and never was
https://www.linuxfoundation.org/about/members
e.g. Red Hat, Intel, Oracle, Google, and now MICROSOFT itself (the competitive landscape changed)
This has LONG been the case, e.g. an article from 2008:
https://www.informationweek.com/it-sectors/linux-contributor...
2017 Linux Foundation Report: https://www.linuxfoundation.org/press/press-release/linux-fo...
Roughly 15,600 developers from more than 1,400 companies have contributed to the Linux kernel since the adoption of Git made detailed tracking possible
The Top 10 organizations sponsoring Linux kernel development since the last report include Intel, Red Hat, Linaro, IBM, Samsung, SUSE, Google, AMD, Renesas and Mellanox
---
curl does seem to be an outlier, but you still need to answer the question: "Who would develop video codecs?" You can't just say "Linux appeared out of thin air", because that's not what happened.
Linux has funding because it serves the interests of a large group of companies that themselves have a source of revenue.
(And to be clear, I do not think that is a bad thing! I prefer it when companies write open source software. But it does skew the design of what open source software is available.)
People don’t develop video codecs for fun like they do with software. And the reason is that it’s almost impossible to do without support from the industry.
We'd be where we are. All the codec-equivalent aspects of their work are unencumbered by patents and there are very high quality free models available in the market that are just given away. If the multimedia world had followed the Google example it'd be quite hard to complain about the codecs.
The top AI companies use very restrictive licenses.
I think it’s actually the other way around and AI industry will actually end up following the video coding industry when it comes to patents, royalties, licenses etc.
How about governments? Radar, Laser, Microwaves - all offshoots of US military R&D.
There's nothing stopping either the US or European governments from stepping up and funding academic progress again.
If we did that we would probably be stuck with low-bitrate 720p videos on YouTube.
THIS PRODUCT IS LICENSED UNDER THE AVC PATENT PORTFOLIO LICENSE FOR THE PERSONAL AND NON-COMMERCIAL USE OF A CONSUMER TO (I) ENCODE VIDEO IN COMPLIANCE WITH THE AVC STANDARD ("AVC VIDEO") AND/OR (II) DECODE AVC VIDEO THAT WAS ENCODED BY A CONSUMER ENGAGED IN A PERSONAL AND NON-COMMERCIAL ACTIVITY AND/OR WAS OBTAINED FROM A VIDEO PROVIDER LICENSED TO PROVIDE AVC VIDEO. NO LICENSE IS GRANTED OR SHALL BE IMPLIED FOR ANY OTHER USE. ADDITIONAL INFORMATION MAY BE OBTAINED FROM MPEG LA, L.L.C. SEE HTTP://WWW.MPEGLA.COM
It's unclear whether this license covers videoconferencing for work purposes (where you are paid, but not specifically to be on that call). It seems to rule out remote tutoring.
MPEG LA probably did not have much choice here because this language requirement (or language close to it) for outgoing patent licenses is likely part of their incoming patent license agreements. It's probably impossible at this point to renegotiate and align the terms with how people actually use video codecs commercially today.
But it means that you can't get a pool license from MPEG LA that covers commercial videoconferencing, you'd have to negotiate separately with the individual patent holders.
[0] https://mpeg.chiariglione.org/standards/mpeg-7/reference-sof...
EDIT: Here is the Wikipedia page of BiM which evidently made it even into an ISO Standard [1]
Has AV1 solved this, to some extent? Although there are patent claims against it (patents for technologies that are fundamental to all the modern video codecs), it still seems better than the patent & licensing situation for h264 / h265.
Just check pirated releases of TV shows and movies.
Av1 for 7
The problem is every platform wants to force their own codec, and get earn royalties from the rest of the world.
They literally sabotaging it. Jxl support even got removed from chrome.
Investment in adopting in software is next to 0.
In hardware it’s a different story, and I’m not sure to what extent which codec can be properly accelerated
I remember this same guy complaining investments in the MPEG extortionist group would disappear because they couldn't fight against AV1.
He was part of a patent Mafia is is only lamenting he lost power.
Hypocrisy in its finest form.
https://blog.chiariglione.org/a-crisis-the-causes-and-a-solu...
He is not a coder, not a researcher, he is only part of the worst game there is in this industry: a money maker from patents and "standards" you need to pay for to use, implement or claim compatibility.
> At long last everybody realises that the old MPEG business model is now broke
And the entire post is about how dysfunctional MPEG is and how AOM rose to deal with it. It is tragic to waste so much time and money only to produce nothing. He's criticizing the MPEG group and their infighting. He's literally criticizing MPEG's licensing model and the leadership of the companies in MPEG. He's an MPEG member saying MPEG's business model is broken yet no one has a desire to fix it, so it will be beaten by a competitor. Would you not want to see your own organization reform rather than die?
Reminder AOM is a bunch of megacorps with profit motive too, which is why he thinks this ultimately leads to stalled innovation:
> My concerns are at a different level and have to do with the way industry at large will be able to access innovation. AOM will certainly give much needed stability to the video codec market but this will come at the cost of reduced if not entirely halted technical progress. There will simply be no incentive for companies to develop new video compression technologies, at very significant cost because of the sophistication of the field, knowing that their assets will be thankfully – and nothing more – accepted and used by AOM in their video codecs.
> Companies will slash their video compression technology investments, thousands of jobs will go and millions of USD of funding to universities will be cut. A successful “access technology at no cost” model will spread to other fields.
Money is the motivator. Figuring out how to reward investment in pushing the technology forward is his concern. It sounds like he is open to suggestions.
He first points out that a royalty-free format was actually better than the patent-pending alternative that he was responsible for pushing.
In the end, he concludes that the that the progress of video compression would stop if developers can't make money from patents, providing a comparison table on codec improvements that conveniently omits the aforementioned royalty-free code being better than the commercial alternatives pushed by his group.
Besides the above fallacy, the article is simply full of boasting about his own self-importance and religious connotations.
MPEG was also joint with the video conferencing standards group within the CCITT (now International Telecommunications Union), which generally required FRAND declarations from patent holders.
My recollection is that MPEG-LA was set up as a clearing house so that implementers could go to one licensing organization, rather than negotiating with each patent owner individually.
All the patents for MPEG 1 and MPEG 2 must be expired by now.
Besides patent gridlock, there is a fundamental economic problem with developing new video coding algorithms. It's very difficult to develop an algorithm that will halve the bit rate for the same quality, to get it implemented in hardware products an software, and to introduce it broadly in the existing video services infrastructure. Plus, doubling the compression is likely to more than double the processing required. On the other hand, within a couple of years the network engineers will double the bit rate for the same cost, and the storage engineers will double the storage for the same cost. They, like processing, follow their own Moore's Law.
So reducing the cost by improving codecs is more expensive and takes more effort and time than just waiting for the processor, storage and networking cost reductions. At least that's been true over the 3 decades since MPEG 2.
If you're interested in this, it's a good idea reading about the Hutter prize (https://en.wikipedia.org/wiki/Hutter_Prize) and going from there.
In general, lossless compression works by predicting the next (letter/token/frame) and then encoding the difference from the prediction in the data stream succinctly. The better you predict, the less you need to encode, the better you compress.
The flip side of this is that all fields of compression have a lot to gain from progress in AI.
Fabrice Bellard's nncp (mentioned in a different comment) leads.
If you have a predictor/compressor LLM which was trained on all the movies in the world, would that not also be infringement?
DCVC-RT (https://github.com/microsoft/DCVC) - A deep learning based video codec claims to deliver 21% more compression than h266.
One of the compelling edge AI usecases is to create deep learning based audio/video codecs on consumer hardwares.
One of the large/enterprise AI usecases is to create a coding model that generates deep learning based audio/video codecs for consumer hardwares.
This makes zero sense, right? Even if this was applicable, why would it need a standard? There is no interoperability between game servers of different games
Copyright is cancer. The faster AI industry is going to run it into the ground, the better.
Or is it MPEG LA? https://wiki.endsoftwarepatents.org/wiki/MPEG_LA
And, boy howdy, they did.
Maybe these sorts of handshake agreements and industry collaboration were necessary to get things rolling in 198x. If so, then I thank the MPEG group for starting that work. But by 2005 or so when DivX and XviD and h264 were heating up, it was time to move beyond that model towards open interoperability.
