undefined | Better HN

0 pointsarthurcolle7y ago0 comments

That's with no images, and iirc it also didn't include any of the math markup last time I looked at one of the xml dumps in one of the loaders (think it was kiwix)

0 comments

leowoo917y ago

Totally forgot about images. I just dug little further and found this:

"The size of the media files in Wikimedia Commons, which includes the images, videos and other media used across all the language-specific Wikipedias was described as well over 23 TB near the end of 2014"

Considering 30 TB hdd costs around 1500$, still interesting.

arthurcolleOP7y ago

I wonder what the size is as of this moment, maybe double? I read somewhere recently that 90% of the content on the Web was created in the last 2 years. Not sure if thats "special media" focused, i.e. video & pictures, as opposed to text. But I'm sure the size of the dump has increased substantially in the last 4 years.

leowoo917y ago

Very likely, there at least need of a filter at some level to size down (maybe getting single language or converting videos to low-res)

crtasm7y ago

If you grab those WD Easystore 8tb drives when they're on sale you can have 32tb for around $560

1 more reply

j / k navigate · click thread line to collapse

0 comments

leowoo917y ago

Totally forgot about images. I just dug little further and found this:

Considering 30 TB hdd costs around 1500$, still interesting.

arthurcolleOP7y ago

leowoo917y ago

Very likely, there at least need of a filter at some level to size down (maybe getting single language or converting videos to low-res)

crtasm7y ago

If you grab those WD Easystore 8tb drives when they're on sale you can have 32tb for around $560

1 more reply

j / k navigate · click thread line to collapse