Website Hosting

Check out serveraday.com from OLM - while todays deal isn't something to jump on, they often have commodity servers for $30 a month, and occasionally for $25 a month. These are decent servers, and give you full access to a dedicated machine for a very reasonable price. Additionally, their servers have been reliable and their network has been more than satisfactory.

Linode.com (I'm a customer).

Look into the cheapest Rackspace cloud servers.

Well if you're just starting out, you probably don't want to go all out. Start simple and cheap. There are plenty of affordable and reliable hosts out there. I use a dedicated server with JustHost... I upgraded from a shared server for pretty cheap. Their shared server lagged every now and then but was overall very reliable... and I've had 0 problems with their dedicated server. As far as control goes, I have quite a bit of control over everything - as much as I really need. You can add SSH/Shell access if you really need it. I'm sure other hosting companies out there offer a bit more control than JustHost (or similar) but if you want ABSOLUTE control, you'd need to host the site yourself. The best thing for you to do when starting out though is save your money... there's no point in going all out until you expect a lot of traffic.