August 3rd, 2010
Quick HTTP Header grabbing script
Just a quick script I came up with when I had to grab multiple url http headers. The script prints output to the terminal as well as writes an output file in the CWD you run the script from. The list of urls needs to have full http:// syntax and one url per line. Click here to view and download the script.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | #!/usr/bin/python # Simple program which opens a file of urls, retrives their headers and prints them to tty and writes to a file import urllib2 import sys usage = ''' Port 80 Headers - Multiple site list Author: bostonlink Usage: ./80headers.py url_list' Notes: Use a custom list of urls, each url should be on a new line. eg: http://google.com http:yahoo.com if there is an empty new line at the end of the file, the script will terminate when the '\n' newline is passed to it. ''' if (len(sys.argv)!=2): print(usage) sys.exit(0) usrfile = open(sys.argv[1], 'r') outfile = open('output.txt', 'w') outfile.close() urls = usrfile.readlines() for url in urls: if url == '\n': break else: url.rstrip() header = urllib2.urlopen(url).info() print('=' * 60) print(url) print('-' * 60) print(header) print('=' * 60) print('') f = open('output.txt', 'a') f.write(('=' * 60) + '\n' ) f.write(url) f.write(('-' * 60) + '\n') f.write(str(header)) f.close() usrfile.close() |