undefined | Better HN

0 pointsnorthwest654y ago0 comments

Back when we had to scrape airline websites to get the deals they withheld for themselves, residential IP was indeed the way. Once the cottoned on to it and blocked id, you'd simply cycle the ADSL model, get a new IP, and off you'd go again.

Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad allocated to web scraping). The airline threw ridiculous dollars at trying to stop it, and we just used a caffeine fueled nerd to keep it running. It wasn't all fun though, they'd often release their new anti scraping stuff on a Friday afternoon. They were less than impressed when they learnt who the 'enemy' was. Good times!

0 comments

Aperocky4y ago

Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Jugurtha4y ago

>Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?

Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?

xnyan4y ago

Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser.

Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.

1 more reply

northwest65OP4y ago

Yeah, but this was before Ajax when you had to hand roll XML RPC calls (i.e. long long ago).

1cvmask4y ago

What do you mean by deals withheld for themselves?

northwest65OP4y ago

Most flights are available through the airline booking systems such as Sabre. However, airlines might have flights available only on their own website at (sometimes massively) reduced cost, which needs to be booked through that site. So the web scraping became two parts, one to provide the data to our search engine to present to our customer (travel agent) customers. The second part was then we would book via the airlines website with the details provided by our customer's customer.

j / k navigate · click thread line to collapse

0 pointsnorthwest654y ago0 comments

0 comments

Aperocky4y ago

Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Jugurtha4y ago

>Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?

Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?

xnyan4y ago

Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.

1 more reply

northwest65OP4y ago

Yeah, but this was before Ajax when you had to hand roll XML RPC calls (i.e. long long ago).

1cvmask4y ago

What do you mean by deals withheld for themselves?

northwest65OP4y ago

j / k navigate · click thread line to collapse