Dekker1/Fourmi
Archived
1
0
Fork 0
Code

TODO on sppofing user agent

Browse Source
This commit is contained in:
Jip J. Dekker 2014-06-17 00:33:08 +02:00
parent 56e1d3cfb6
commit 6e16e9f23e
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
FourmiCrawler/settings.py
View File

@ -18,10 +18,10 @@ ITEM_PIPELINES = {
FEED_URI = 'results.json'
FEED_FORMAT = 'jsonlines'
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36'
# Crawl responsibly by identifying yourself (and your website) on the
# user-agent
# [todo] - Check for repercussions on spoofing the user agent
# USER_AGENT = 'FourmiCrawler (+http://www.yourdomain.com)'
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36'