Tobias Gruetzmacher 68d4dd463a Revert robots.txt handling. ... This brings us back to only honouring robots.txt on page downloads, not on image downloads. Rationale: Dosage is not a "robot" in the classical sense. It's not designed to spider huge amounts of web sites in search for some content to index, it's only intended to help users keep a personal archive of comics he is interested in. We try very hard to never download any image twice. This fixes #24. (Precedent for this rationale: Google Feedfetcher: https://support.google.com/webmasters/answer/178852?hl=en#robots)		2015-07-17 20:46:56 +02:00
..
__init__.py
a.py
arcamax.py
b.py
c.py
clonemanga.py
comicfury.py
comicgenesis.py
creators.py
d.py
drunkduck.py
e.py
f.py
footloosecomic.py
g.py
gocomics.py
h.py
i.py
j.py
k.py
keenspot.py
kindofnormal.py	fixed Wulffmorgenthaler (now Wumo), added TruthFacts and MeAndDanielle	2015-06-01 12:14:52 +12:00
l.py
m.py
n.py
nitrocosm.py
nuklearpower.py
num.py
o.py
p.py
pensandtales.py
petitesymphony.py
q.py
r.py
s.py
sandraandwoo.py
smackjeeves.py
snafu.py
t.py
u.py
v.py
w.py
webcomiceu.py
webcomicnation.py
wlpcomics.py
wordpress.py
x.py
y.py
z.py