Below is a simple, _lightweight_ ngrep solution. RE means a regular expression. This only saves packets with the RE you specify and does not save full packets, only the HTTP headers. 1024 is an arbitrary size to get all HTTP headers; adjust to taste. tcpdump is there only because ngrep does not work well with PPPoE. If you don't use PPPoE you don't need to include tcpdump.
case $# in
1)
# capture HTTP headers to pcap file
tcpdump -Ulvvvns1024 -w- tcp 2>/dev/null \
|ngrep -O$1 -qtWbyline 'GET|POST|HEAD' >/dev/null
;;
2)
# search HTTP headers in pcap file
ngrep -Wbyline -qtI$1 $2
;;
*)
echo usage: $0 pcap-file \[RE\]
esac
To dump your results, try $0 pcap-file . |less
And here's a little script to make URL's from your pcap file.
unvis just decodes URL's from the specs in RFC's 1808 and 1866. It assumes http:// URL's (no ftp://). The awk script ensures all URL's (not just consecutive ones) are unique. case $# in
[12])
above-script $1 ${2-.} \
|sed -n '
/GET/p;
/Host: /p;
' \
|tr '\012' '\040' \
|sed 's/GET/\
&/g' \
|awk '
!($0 in a){a[$0];print "http://"$5$2}
' \
|sed '
s/%25/%/g;
s/\.\//\//;
' \
|unvis -hH \
|sed '/^http:[/][/]./!d;
s/ /%20/g' \
;;
*)
echo usage: $0 pcap-file \[RE\] >&2
esac
It's trivial to dump HTTP. You can feed this to netcat (using sed to modify the HTML to your liking), then open the result in your browser. Whatever you are aiming to do (I'm still not exactly sure - can you give an example?), I reckon it can be automated without Python and heaps of libraries.We've been using http://www.charlesproxy.com/ for years, great tool (cheap albeit not free)
1) lot of library support, in multiple languages
2) without having to fake UAs, etc, the requests look more like a regular user (all media assets downloaded, normal browser UA, etc)
3) simple clustering: setting up a Selenium grid is very easy, and switching from local instance of Selenium to using the grid requires very little code change (1 line in most cases)
A few years ago, I wanted to analyze retail store customer feedback data collected by a third party company. The stores were franchises, and the third party was anointed by the franchising company. The data was presented to the user (franchisee store management) via a fancy web site with its own opinion about how the data should be analyzed. My opinion differed. I wanted the data in low-level, RDBMS-friendly form, so that I could recast it every which way (and come back and do it again a new way I thought of). However, such was not forthcoming (big company, little franchisee).
The solution was to make a robot that put the third party company's portal through its paces at the finest granularity, scraping the numbers into a DB as they tediously appeared. The robot was in JRuby††, allowing access to HtmlUnit's functionality without the tedium of Java coding. It was slow, but I didn't care — run it overnight once a month, then run reports off the DB generated.
The coding approach was simple: Pretend you are a user. Access each page, starting with the login page, and do what the user would do. Scrape the interesting numbers as they appear. Append appropriate rows to the DB.
1. Documented API. Failing that...
2. HTTP client fetching structured data (XHR calls). Failing that...
3. HTTP client fetching and scraping HTML documents. Failing that...
4. Headless browser
I recently found myself pushed to #4 to handle sites with over-complex JS or anti-automation techniques.