Dekker1/Fourmi
Archived
1
0
Fork 0
Code

Added custom user-agent because otherwise it would block, because not amused by scraper

Browse Source
This commit is contained in:
Nout van Deijck 2014-05-21 16:11:02 +02:00
parent 4b377bb9a9
commit fb41d772f2
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
FourmiCrawler/settings.py
View File

@ -16,6 +16,8 @@ ITEM_PIPELINES = {
FEED_URI = 'results.json' FEED_URI = 'results.json'
FEED_FORMAT = 'jsonlines' FEED_FORMAT = 'jsonlines'
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36'
# Crawl responsibly by identifying yourself (and your website) on the # Crawl responsibly by identifying yourself (and your website) on the
# user-agent # user-agent