Why scraping and ecommerce are a perfect fit (opens in new tab)

Oldnavy and a few other retailers are not active yet. We've pre-built these integrations despite not having requests to sell their inventory just yet.

I'd mention more on the BD side but can't at this point for competitive reasons. The fact that we currently support sending orders through to 450 retailers does not mean we have deals in place with all of them, but that the infrastructure is built to allow this to happen -- if affiliates or publishers get an approval from retailers or the affiliate networks that govern this. Perhaps we should make this clearer on the supported stores page.

All in due time. The industry as a whole is being pushed to decide which models they will embrace -- and as always some will be slower to adapt than others. The pressure comes from lost revenue on mobile which makes retailers a LOT more flexible now compared to even 6 months ago when talking about this.

Considering multiple format screens and devices fragmenting retailers distribution channels over the next years this is set to become an even bigger chapter down the line.

Probably they don't, because so much of the web has robots files like

User-Agent: established_company

Allow: /some-stuff

User-Agent: *

Disallow: /

# keeps out filthy peasants

And you're either stuck following them, and not having data that would be offered up for free if you were someone else, or being a bad person and ignoring it. You don't really see the services that follow the rules.

Also, good paper on how much being on robots.txt preferred helps, which makes you a better product, which makes you more preferred...

https://etda.libraries.psu.edu/paper/9230/4516

Tom, that most likely wasn't us.

We don't spider retailer websites. That means we don't follow links or go hardcore on building a database of products.

We hit your website:

* if someone has asked us information about a product url

* when we place an order

* weekly for regression tests

Ping us on contact@ and we're more than happy to jump on a call and describe exactly what we're doing. Most of the time we're completely un-noticeable except for the fact that you're getting more orders.

We know for sure nobody is spidering through us.

AFAIK, the main legal issue is a trespass to chattels tort. The data collected is generally uncopyrightable if not reproduced in it's entirety without modifications. The relevant case is Feist v. Rural [0].

IANAL, but I think the best bet for staying technically legal is to use jurisdictional arbitrage and tit-for-tat to liberate the data. If someone scrapes a US server and are in the US and they generate enough load to deprive the owner of use, then they are technically liable for damages under trespass to chattels. If they instead trade scraping labor with people in other jurisdictions, then that other entity would be liable. There might be some other legal defense/attack that might be usable by the entity who has the data being liberated, but I reckon it would be tenuous at best.

[0] http://en.wikipedia.org/wiki/Feist_v._Rural

The only thing illegal about it is you're potentially violating local copyrights. Sites generally state their stance on scraping in the terms of use.

Good question. That's because this method doesn't scale and fails as a solution to the industry's challenges.

There's companies that are trying to get retailers to implement APIs but this leads to a fragmented ecosystem. Year's past payment processors that sold "pay/checkout with ..." buttons and wallets have failed to achieve significant merchant adoption despite being fuelled with marketing spend in the billions.

The solution everyone embraces seems to rely in building an independent and neutral piece of infrastructure (an API) that any publisher can integrate and that plugs into every checkout out there. It's the missing pipes in ecommerce, anyone can use it and nothing really changes (we don't process payments, it's all automated etc) -- and conversions go UP.

I'm repeating some ideas in the post but on the publisher side it's worth noting NONE would entertain the idea of integrating multiple APIs -- one for each merchant. Did I also bring up the required combined efforts of all merchants to keep those APIs up & running? :)

So pro-scraping because it's the only way to build adoption in ecommerce.

Completely agree! Returns and not breaking the retailer's CRM is key in the space. Retailers are happy especially because we're not breaking their relationship with consumers nor obscuring payment/shipping data. It's very difficult to find and build a model that's accepted and actively supported by all stakeholders in the ecommerce space and we're very excited by current efforts in the industry. Long story short, we've had no complaints or confusion from consumers so far on returns, customer support etc. By now most consumers are aware of in-stream or remote buying and following basic guidelines helps a lot too (clearly displaying the retailer logo, user messaging etc)

And yes, PCI DSS compliance is also crucial to storing and handling credentials. We're going through the process again this year at Two Tap, but the effort is worth it.

Are you sure that is true for all retailers of those products?

I deal with a high return rate industry [specialty products many customers can't size correctly] and I only see return rates of 3-7% depending on the product. 40% seems very high.

At least one of them might have been using our technology in the backend, especially if they're one of the top 5 shopping search engines.

The downside to feeds is that they become obsolete very quickly, especially if the product is popular. Products sell out very quickly, retailers lose money on traffic they can't onboard and shoppers get frustrated.

Thanks for your thoughts!

I'd be interested in seeing this.

Nice :) We currently focus on the US market with both retailer as well as publisher integrations, we should make that clearer perhaps.

Stay tuned though, we'll have news on this.

Oops, it's because it's the french version of the website. We currently support only US retailers.

If you try this (same product, US version) it would work perfectly: http://shop.lego.com/en-US/King-s-Castle-70404?_requestid=25...

We simulate what a shopper would do. We first go through your affiliate link (which drops a cookie) and then go on the retailer website to place the order.

All the commissioning, connecting/talking to retailers, receiving the money, is directly between you and the affiliate network. We're plug and play :)

We do have a stats backend where you can see all the purchases that went through Two Tap. And you can also use CJs dashboard just like you are probably doing right now.

Not really, we're not building a product catalog.

We're fetching the live data only for the products requested via an URL. Two Tap mimics a consumer visiting the retailer and getting that info for themselves which also allows retailers to retain their analytics layer with no negative impact.

Our current supported stores span the top 500 as well as a number of specific integration requests.

That isn't quite right. Many vendors provided them with XML feeds of their products just as they do with Amazon.

Paraphrasing newrelic "it takes a village to count our proxy nodes".

We have our placing orders infrastructure on AWS, and whole in-house cloud dedicated to product crawling built on top of Digital Ocean.

Yes, you could say that. We're laying down pipes in ecommerce so you can send an order to a merchant from anywhere on the web in a standardised API.

Retailers can extend their reach and make their inventory shoppable from anywhere with an internet connection and publishers can build ecommerce in their apps.