tajddin

For a personal project, I’m designing a system that pulls information from various sources, including IMAP, twitter and facebook (and others). Ideally, I’d like the user to be notified when the system receives data from any of these sources (thus timeliness is important), but I’d like some input into how best to accomplish this when there are potentially hundreds of concurrent users.

To solve part of the problem, I will be creating a separate service on a different AWS instance that connects to each data source and interacts with the DB independently from IIS.

Should this be done on an interval (say, every five minutes), looping through each of the users’ accounts and pulling data, or alternatively, should I use a multithreading approach, where each account has an associated polling thread?

Keep in mind, the technologies used are: ASP.NET MVC 4, SQL Server.

Your insight is much appreciated.