undefined | Better HN

0 pointsrobryan13y ago0 comments

Has anything that has built on top of wikipedia as the primary product got much traction? I think it is something that would be hard to really get widespread without it.

Can a 3rd party even maintain a fully up to date set of wikipedia content? Not sure what their policy on scraping is?

0 comments

thebigshane13y ago

Wikipedia also has an API (although this does little to address tptacek's concerns of database meltdown, unless you cache frequent requests):

http://en.wikipedia.org/w/api.php

http://www.mediawiki.org/wiki/API:Main_page

http://www.mediawiki.org/wiki/API:Query

For example:

http://en.wikipedia.org/w/api.php?action=query&prop=revi...

brings back:

   { 
     "query": { 
       "pages": {
         "1666533": {
           "pageid": 1666533, 
           "ns": 0, 
           "title": "Y Combinator (company)", 
           "revisions": [
             {
               "*": "{{Infobox company\n| company_name = Y Combinator\n 
                     [...redacted full wikitext content of article...]"
             }
           ]
         }
       }
     }
   }

jarek13y ago

No need to scrape, Wikipedia has full dumps for the articles available for download: http://en.wikipedia.org/wiki/Wikipedia:Database_download#Eng...

robryanOP13y ago

Yeah I am aware, but wikipedia changes daily. There would have to be a pretty impressive feature set built on top before many would consider using an out of date wikipedia when they can easily see the latest version.

gromgull13y ago

There is a feed of changes though - both a "realtime" IRC feed that is restricted access and some sort of batch feed (daily? Hourly?)

jarek13y ago

Hm... for most pages it shouldn't really matter, they don't change drastically within days. For developing or current subjects I guess it would...

apgwoz13y ago

This dump is updated once a month, though the API is live.

One problem I've found with the dump is that you still need to render it (unless someone knows of another dump) which has caused us a few problems with the current thing I'm working on.

j / k navigate · click thread line to collapse

0 comments

thebigshane13y ago

Wikipedia also has an API (although this does little to address tptacek's concerns of database meltdown, unless you cache frequent requests):

http://en.wikipedia.org/w/api.php

http://www.mediawiki.org/wiki/API:Main_page

http://www.mediawiki.org/wiki/API:Query

For example:

http://en.wikipedia.org/w/api.php?action=query&prop=revi...

brings back:

   { 
     "query": { 
       "pages": {
         "1666533": {
           "pageid": 1666533, 
           "ns": 0, 
           "title": "Y Combinator (company)", 
           "revisions": [
             {
               "*": "{{Infobox company\n| company_name = Y Combinator\n 
                     [...redacted full wikitext content of article...]"
             }
           ]
         }
       }
     }
   }

jarek13y ago

No need to scrape, Wikipedia has full dumps for the articles available for download: http://en.wikipedia.org/wiki/Wikipedia:Database_download#Eng...

robryanOP13y ago

gromgull13y ago

There is a feed of changes though - both a "realtime" IRC feed that is restricted access and some sort of batch feed (daily? Hourly?)

jarek13y ago

Hm... for most pages it shouldn't really matter, they don't change drastically within days. For developing or current subjects I guess it would...

apgwoz13y ago

This dump is updated once a month, though the API is live.

One problem I've found with the dump is that you still need to render it (unless someone knows of another dump) which has caused us a few problems with the current thing I'm working on.

j / k navigate · click thread line to collapse