dmoonfire/dosage
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Activity

Also check robots.txt on image downloads.

Browse source 
We DO want to honour if images are blocked by robots.txt
This commit is contained in:
Tobias Gruetzmacher 2015-07-15 23:50:57 +02:00
parent 5affd8af68
commit 7c15ea50d8
1 changed files with 1 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
dosagelib/util.py
View file

@ -205,6 +205,7 @@ def getPageContent(url, session, max_content_bytes=MaxContentBytes):
def getImageObject(url, referrer, session, max_content_bytes=MaxImageBytes): def getImageObject(url, referrer, session, max_content_bytes=MaxImageBytes):
"""Get response object for given image URL.""" """Get response object for given image URL."""
check_robotstxt(url, session)
return urlopen(url, session, referrer=referrer, max_content_bytes=max_content_bytes, stream=True) return urlopen(url, session, referrer=referrer, max_content_bytes=max_content_bytes, stream=True)