undefined | Better HN

0 pointsBeefWellington3y ago0 comments

WGET can be pretty trivially told to send custom headers.

0 comments

Try to do that to a site with CF bot protection cranked up... Not happening without a custom build/custom ssl proxy that mimics the SSL fingerprint of Chrome.

harry83y ago

CF blocks you hard simply for enabling "do not track" in settings. The discussion about how awful they are needs to be had.

mananaysiempre3y ago

I haven’t seen a custom build of Wget, but for Curl there is curl-impersonate[1].

[1] https://github.com/lwthiker/curl-impersonate

bigfudge3y ago

It would be a lot of work to make it mimic a common profile though.

justinclift3y ago

That work was probably done once, years ago. Might need a few string tweaks every few years, which could be automated.

bigfudge3y ago

No, because any “RMS” set of headers would only be shared by the small number of nerds who care, fingerprinting us more accurately again.

1 more reply

j / k navigate · click thread line to collapse

0 comments

no_time3y ago

Try to do that to a site with CF bot protection cranked up... Not happening without a custom build/custom ssl proxy that mimics the SSL fingerprint of Chrome.

harry83y ago

CF blocks you hard simply for enabling "do not track" in settings. The discussion about how awful they are needs to be had.

mananaysiempre3y ago

I haven’t seen a custom build of Wget, but for Curl there is curl-impersonate[1].

[1] https://github.com/lwthiker/curl-impersonate

bigfudge3y ago

It would be a lot of work to make it mimic a common profile though.

justinclift3y ago

That work was probably done once, years ago. Might need a few string tweaks every few years, which could be automated.

bigfudge3y ago

No, because any “RMS” set of headers would only be shared by the small number of nerds who care, fingerprinting us more accurately again.

1 more reply

j / k navigate · click thread line to collapse