prasenjit_pro

Background

Proxies API is a rotating proxies API that developers use to fetch hard to scrape data at scale and consistently. We auto-rotate millions of proxy servers, and also handle auto retries, rotate user agent strings, handle cookies, CAPTCHAs behind the scenes.

The Constraints

The service has to scale to millions of URL fetches a day without lagging on speed and should work out of a single API like this.

curl “http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

The Architecture

The architecture looks somewhat like this.

A more in-depth look at this architecture is provided in this article.

Load balancing & Serving

The whole infrastructure is on Amazon EC2. The load balancer plays a significant role in calling the right service based on just the variables passed and the API KEY that holds some secret information like what pool of servers are assigned for them etc.

Our plans have higher concurrency as customers pay more, so we need to pass them to more reliable, dedicated servers as they have higher concurrency needs.

Luckily amazon elastic load balancer can make this using rules you can set, which look at the URL patterns and direct them to different pools of servers intelligently.

Without this exact combination of facilities, we would not have been able to create it all using a single API call without compromising speed and performance somewhere.

We use Apache if there is a need for HTTP servers. Many of these are super optimized to be able to crawl data concurrently using multiple cores. We will soon write an article on how we can get the most amount of crawling ability out of Apache instances.

Proxy Components

For CGI, we use a combination of Python and even PHP where required.

Memcache is used to monitor and throttle concurrencies gracefully, so some rouge code by one of our clients doesn’t bring down the entire setup.

As I described earlier, most of this throttling happens in the way we route out requests from the load balancer.

We use the Node Js/Puppeteer library extensively as an interface to connect to Browserless docker instances located on multiple servers. Browserless is optimized to run as many ready-to-go instances of Chromium as possible, so there is no lag in loading any of them for our clients.

Database

We use good old MySQL in a Master/Slave configuration to hold user info, cache info, millions of proxy info, quality metrics for each, etc.

Maintenance tools

We have a pool pruner that’s written in Python running as a Cronjob. Uses Scrapy and Scrapyd extensively in making calls, monitoring, benchmarking, and pruning proxies from our database.

We use the Beanstalk library for Queues. We use Datadog for monitoring and PagerDuty for alerts.

The author is the founder of Proxies API, a proxy rotation API service.

This article originally appeared here:https://www.proxiesapi.com/blog/The-Stack-Behind-A-Production-Level-Rotating-Proxy-Service.php

While building a bootstrapped startup, I realized early on that the whole heuristic of measuring a startup comes down to the habits that I have constructed that can repeat on a day to day basis consistently for months and years.

I have already written that in the case of Proxies API, my SAAS startup, the two main habits that have made it successful were.

a) Make small, consistent improvements to the product every single day.

b) Write (blog, make videos) every single day.

Now, this is a big ask. I have been able to maintain a streak for months now. So in this marathon, you don’t always feel it. Here are some hacks I have used to get my brain back into perspective that lead to the success of habit-building, which in turn leads to the success of the startup.

1. Have an impersonal, clear goal. I/we need to get x to y in x days/months under these constraints/conditions. The only rule here is that the goal should not be personal. Personal goals are ones that have a feeling of vendetta like a Gollum would create. They are created in mind, and you don’t get a sense that they are natural. Read more here

Personal vs. Natural Goals

2. Have a work backward excel sheet: Life is long. We eat three times, need to go to the toilet ten times, clean up, meet friends and family, power off for 6 hours, and then there are social media, there is Trump, and North Korea is attacking us. In all this, it is difficult to remember what it takes to get to that long term goal. What do we have to do this again? What should be the minimum calls I need to do to get my target number of clients? How many words should I write to keep on track for my goal of having a decent content base to attract SEO and other inbound traffic again? We can’t recalculate it every day. Best to have it in a ‘working backward’ sheet that can be tweaked based on what we learn from reality.

3. The brain needs reminding: Again, in a marathon with a hundred distractions, it is easy to forget the daily targets, the exact strategies we had researched and decided on early in the planning phase. I find that reading my goals and my strategies every day reminds my brain before it becomes vague.

4. Remind yourself first thing in the morning. Ideally, before checking the phone, definitely before any social media or emails. Your goals don’t get overshadowed by concern for how the world is about to end. Someone tweeted something because your neighbor posted a picture from their vacation.

5. Use the 2-minute rule to get a ton of small tasks done very quickly and promptly.

6. Do a brain dump. My brain will get overwhelmed because of all the choices of actions, tactics, growth hacks, and strategies out there. I use the brain dump every two weeks at least to overcome this and get moving again.

7. Make it inevitable. I approach everything with a sense of wanting to make the result inevitable, so instead of doing the bare minimum, I tend to do the overwhelming maximum, and also I make it almost a part of destiny by setting up my environment. More about this technique that I learned from Eben Pagan.

The author is the founder of Proxies API (https://www.proxiesapi.com/), a proxy rotation api service.

This article originally appeared here: https://www.proxiesapi.com/blog/Seven-Ways-To-Trick-Your-Brain-Into-Working.php

Having built Proxies API and made it to 100 paid users for the first time in my startup life, I tried to compile a list of tips for other developers like me who might be thinking of starting something on their own.

Developers are a different beast, and the advice here may not apply to anyone who is not a developer and is a normal human being.

Pick an idea you have used. When you start a startup one of the persistent problems you face is that you don't relate to the problem. If you are not a parent, its almost impossible to relate to any conversations that parents have about their troubles. You just don’t. This lack of empathy in a startup setting adds up to a huge disadvantage and will never allow you to be fully confident in what you are doing. 2. Pick an idea you have already paid for: If you have ever paid for a service, think very carefully about that. Can you code it yourself if you give yourself a few months? It is because developers don’t pay for shit. If an app forced you to do that, then there must a real need for that app.

3. Don’t pick something that needs a pretty UI: My product Proxies API is an API. I can get away with almost no UI. It is such a relief that I don’t have to work with a designer. I find that I am 4 to 5 times faster when I don’t have to deal with UI stuff.

4. Don’t pick anything that you need in-person sales for. Marketing is a developer’s friend. I realized this only later in my life. In-person sales are very weird for the developer personality. My advice. Don't do it. Learn how to market instead.

5. Pick something that is a self-serve model: People signup for a trial and decide to pay or not pay based on their trial experience. This is a beast that you can conquer. You don’t want to be going around talking to actual humans. It is not for us.

6. Try writing: If you can code and write, you will win the world. The whole success of Proxies API is based on constant improvements to code and a little bit of consistent writing over time.

7. Keep it real: Be clear on your motivations. I know that developers are not motivated by the prospect of making millions. But they are excited by the promise of financial freedom. So I was clear on my motivations when starting Proxies API — I didn’t want it to make me millions. I wanted it to give me financial freedom by earning what I earned in my day job. I didn’t need a penny more. By being clear about it, you are not living someone else’s life but your own, and it will give you the necessary fuel to keep going.

Have fun.

The author is the founder of Proxies API the rotating proxies service.

This article originally appeared here: https://www.proxiesapi.com/blog/Tips-For-Developers-Who-Want-To-Build-A-SAAS-Startup.php

For years, I have dismissed what I knew to be extremely potent, if only I implemented it in my life, as something that I could never do;

that’s not me; that is something I don’t enjoy

was always my refrain.

And yet here I am.

Hi, I am Mohan, and I am a chronic developer who couldn’t be anything else. But I have recovered.

How?

Because it needs to be done.

As developers, we are there to solve problems. We are not there to be good at C or JAVA or Python or some such tool. We have to address the issues. If you could use pixie dust and your pixie dust could solve all problems, your clients and I could care less about your fondness for Python or your identity as a ‘Linux guy’. We are here to solve problems.

In a broader sense, the business is there not only to solve a problem (represented by an excellent working product) but also to tell people that it exists and can solve the said problem.

So business has 2 problems. And it needs to solve these two problems at the meta-level.

1) Solve the problem customers have.

2) Reach them and let them know you have something that solves their problem.

The 2nd one is called marketing.

But who cares?

It is just problem-solving.

Does it matter that you don’t like it?

Because it solves a problem.

Ok. So we agree on that point at least. Let us look at this further.

“Marketers” will tell you there are many different types of marketing.

Digital marketing, SEO inbound, outbound, PR.

I can see that you are about to have seizures. YOU can NEVER do all that. Don't worry.

Instead of all this jargon, let’s return to the original problem of reaching and telling people you have something that solves their problems.

Let me now, gently propose the idea of Writing. Because, in my mind, Marketing is nothing but Writing.

The internet is a prominent place. It has only two functions when you reduce it to the basics.

“Someone writes something into it” Or “Someone reads something from it.”

Like, right now. I am writing this, and you are reading this. That’s it. This action, repeated a million times, is the internet. Everything else is arbitrage.

We don’t want anything to do with these other shady forms of marketing.

But we surely can’t have any problem giving something to the place that offers something to us?

Like, I am writing this (giving), and I will get some readers and maybe some clients (getting), right?

Fair transaction. Nothing shady. It is the most innocent form of the web. It is the most natural, honest, unbiased, and evergreen form of marketing.

So here is what I urge you to consider now.

Solve the 2nd problem I listed by simply writing into the web.

Just write. And it will give back to you.

I know you won’t like it.

But those are only in the beginning and remember, so what? So what if you don’t like it. It solves the problem.

Consider it. Let this not happen today.

Let this idea seep into your mind slowly over time.

Start slow and ramp up, accumulate writing over time. Like I did. After years of resisting, fighting, throwing tantrums, I think today I get it.

It has allowed me to build something that has given me financial freedom. Writing has.

All I did was solve both those problems I listed earlier. I have built my SAAS product Proxies API as well as the habit of writing.

Coding plus Writing makes me dangerous.

The author is the founder of Proxies API the rotating proxies service.

This article originally appeared here:https://www.proxiesapi.com/blog/Developers-If-You-Can-Master-Writing-You-Will-Be-Dangerous.php

After having failed with more than seven startups and finally achieved some success with my latest one Proxies API, here are a bunch of criteria I follow when picking a startup idea to bootstrap from the website

1. I am a bootstrapper, so I am interested in making recurring revenue and build an asset. So it has to be a SAAS idea. No consumer internet.

2. It has to have a competition. Again, I want to make money, not be a pioneer.

3. It need not be a multi-million dollar potential startup. I prefer it if it’s possible never exceeds a million dollars at its peak. It is because I don’t want to offer a platform, but a small subset of features that people find useful and the big guys find too small a market niche to address.

4. I prefer it if it does not need any design skills. With Proxies API, it is all about technology and how it works. I am in my element when I can run with just code and not have to pause for ‘prettifying stuff.’

5. I want a self-serve model. It is because I want to acquire customers by marketing rather than by sales. I have nothing against sales. I prefer this as I like to code, and I am a bit shy and introverted. I can write, in any case.

6. I want a field where I can write a lot about it. Just by looking at the area, I should be able to imagine what my first ten blog posts will be. If I can’t write, I can’t market. So I would slightly not touch it as I am not good at the rest of them.

7. The product should not be commoditized. It’s happening in almost any market after about five years. Social media tools are an example. They were selling at $100 pm a few years back now people are offering it free.

8. The technology should provide a moat against every amateur trying to start a company. That’s why there are a million to-do list apps. That’s practically the first thing they teach in any course.

9. I want to identify the primary keyword that drives traffic and search for that keyword on Google Trends. Google Trends should show the keyword gaining in popularity ideally, or at least it should be a flat line for the last five years. I will stay away from anything that is on its way out. 10. I want the users to get the AHA moment in one line or one image and not in multiple steps or after they “add 18 friends and two monkeys” rubbish. Just one look and the value prop is right there. With Proxies API, it was always this image and line. In essence, it immediately drove home this visceral power of using a single API call to solve an entire gamut of problem-related to scaling web crawling. The author is the founder of Proxies API the rotating proxies service. This article originally appeared here: https://www.proxiesapi.com/blog/My-Criteria-For-Picking-Startup-Ideas.php