dmoonfire/dosage
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Activity
1287 commits 1 branch 0 tags 31 MiB
Commit graph

1287 commits

This branch
This branch
All branches
Author SHA1 Message Date
Tobias Gruetzmacher d05316e3ac Seems ComicFury is deleting comics regularly... 
Well, there's nothing we can do: Remove them.
 2016-05-04 08:26:53 +02:00
Tobias Gruetzmacher 0c1aa9e8bd Move libxml < 2.9.3 workaround to base class. 2016-05-02 23:22:06 +02:00
Tobias Gruetzmacher b93a8fde65 Move PensAndTales comics and fix them. 2016-05-02 22:32:14 +02:00
Tobias Gruetzmacher 4006ced43d Move all HijinksEnsue comics into alphabetic files. 2016-05-02 01:25:34 +02:00
Tobias Gruetzmacher d5f91ecfd2 Fix some modules in m.py. 2016-04-30 01:59:28 +02:00
Tobias Gruetzmacher 1d52d33311 Remove missing SmackJeeves comics. 2016-04-30 00:56:20 +02:00
Tobias Gruetzmacher d796f3476c Fix some modules in d.py. 2016-04-30 00:44:18 +02:00
Tobias Gruetzmacher cc16fea880 Fix some modules in c.py 2016-04-29 00:35:02 +02:00
Tobias Gruetzmacher 1d94439715 Fix some more comic modules. 2016-04-27 00:31:27 +02:00
Tobias Gruetzmacher 8b1ac4eb35 Fix "tagsoup" on SmackJeeves 
Unfortunatly, browsers render < outside of HTML tags differently then
libXML until recently (libXML 2.9.3), so we need to preprocess pages
before parsing them...

(This was fixed in libXML commit 140c25)
 2016-04-26 08:05:38 +02:00
Tobias Gruetzmacher 00fb51341a "Better" selection of default test comics. 2016-04-26 08:03:34 +02:00
Tobias Gruetzmacher 035d6e94e4 Allow output level for warnings and errors. 2016-04-26 07:53:53 +02:00
Tobias Gruetzmacher 8ddf553eb4 Fix some more SmackJeeves modules. 2016-04-22 01:04:47 +02:00
Tobias Gruetzmacher fd85c8583a Unify similar code in fetchUrl and fetchText 2016-04-22 00:42:46 +02:00
Tobias Gruetzmacher 6574997e01 Refactor: All the other class methods. 
Turns out, it would have been better if all methods had been instance
methods and not class methods. This finished a big chunk of the rework
needed for #42.
 2016-04-21 23:52:31 +02:00
Tobias Gruetzmacher 0d436b8ca9 Refactor: url modifiers to normal methods. 
As before, to implement #42 these might want to access information from
the instance, so they should be normal methods.
 2016-04-21 21:39:25 +02:00
Tobias Gruetzmacher c3f32dfef7 Refactor: Make namer a method. 
When #42 is realized, the naming of files might differ between comic
modules, so the namer's logical location is the instance, not the class.
 2016-04-21 08:20:49 +02:00
Tobias Gruetzmacher 5bd2a49f48 Add debug output on matched XPath/CSS expression. 2016-04-20 23:51:54 +02:00
Tobias Gruetzmacher 1d2e1f2dd1 Update SmackJeeves update helper. 
Don't use it right now, it adds a HUGE amount of comics.
 2016-04-20 23:48:29 +02:00
Tobias Gruetzmacher fe51a449df Update SmackJeeves 
- Now uses _ParserScraper, which makes the pattern quite a bit more
  generic and IMHO more readable
- remove make_scraper magic
- No new comics, only fixed existing ones and removed some dead ones.
 2016-04-20 23:36:45 +02:00
Tobias Gruetzmacher 190cd3b063 Convert language & getDisabledReasons to methods. 
Both are more properties of a webcomic (this is part of the design
changes for #42)
 2016-04-19 23:53:46 +02:00
Tobias Gruetzmacher df46907f39 Register EXSLT extensions by default. 
This allows comic module authors to use the full power of regular
expressions in XPath expression, see http://exslt.org/regexp/regexp.html
for usage. Please be aware that these use the prefix re: instead of
regexp: here.
 2016-04-19 23:48:14 +02:00
Tobias Gruetzmacher 4204f5f1e4 Send "If-Modified-Since" header for images. 2016-04-19 00:36:50 +02:00
Tobias Gruetzmacher 13a3409854 Remove some comics that are gone or block us. 2016-04-17 19:42:43 +02:00
Tobias Gruetzmacher 1fbc844077 Update GoComics. 2016-04-17 18:40:09 +02:00
Tobias Gruetzmacher 820471b49b Move exclusion into 2nd stage. 2016-04-17 18:15:30 +02:00
Tobias Gruetzmacher 73e958670d Update ComicFury (again). 2016-04-17 16:19:44 +02:00
Tobias Gruetzmacher 27e1861001 Tox: Check __future__ imports in flake8 run. 2016-04-16 23:24:07 +02:00
Tobias Gruetzmacher a3a7c7f688 Tox: Pass {posargs} to py.test. 2016-04-16 22:57:45 +02:00
Tobias Gruetzmacher 35b9800da7 Tox: Always generate JUnit XML, run flake8. 2016-04-16 16:59:04 +02:00
Tobias Gruetzmacher b0481a01f7 Update languages. 2016-04-16 13:14:12 +02:00
Tobias Gruetzmacher 3329027e4b Update ComicFury. 2016-04-16 13:13:47 +02:00
Tobias Gruetzmacher 7708c678a6 Move test parametrization into its own file. 2016-04-16 12:01:43 +02:00
Tobias Gruetzmacher 7c3a87970b Fix test naming. 2016-04-16 11:54:29 +02:00
Tobias Gruetzmacher ee99c087d7 Remove prevUrlMatchesStripUrl. 
It was only used for one test.
 2016-04-16 01:14:26 +02:00
Tobias Gruetzmacher 92a688457a Remove useless indirection. 2016-04-15 23:42:24 +02:00
Tobias Gruetzmacher 52515b5fc5 Update GoComics. 2016-04-15 00:26:14 +02:00
Tobias Gruetzmacher 031a523846 Fix SnafuComics. 2016-04-14 23:52:35 +02:00
Tobias Gruetzmacher 7626b1e100 Webcomics Nation is gone. 2016-04-14 22:46:52 +02:00
Tobias Gruetzmacher dab5aef094 Refactor update helpers: Remove duplicate code. 2016-04-14 22:22:37 +02:00
Tobias Gruetzmacher 497653c448 Remove make_scraper magic from Arcamax. 2016-04-14 00:17:59 +02:00
Tobias Gruetzmacher db87ed95e7 Use new features to make modules simpler. 2016-04-13 23:28:43 +02:00
Tobias Gruetzmacher b266e28ae1 Remove debugging prints 😭 2016-04-13 22:59:06 +02:00
Tobias Gruetzmacher ff3b824311 Fix variable shadowing... 2016-04-13 22:43:34 +02:00
Tobias Gruetzmacher 060281e5ff Use concrete scraper objects everywhere. 
This is a first step for #42. Since most access to the scraper classes
is through instances, modules can now dynamically override url and name
(name is now a property).
 2016-04-13 22:17:30 +02:00
Tobias Gruetzmacher 0468f2f31a Refactor: Convert starter to simple method. 2016-04-13 20:01:51 +02:00
Tobias Gruetzmacher 1aebdce5d2 Fix test :) 2016-04-13 01:32:15 +02:00
Tobias Gruetzmacher 16004e43e4 Use default bounceStarter for site modules. 2016-04-13 01:24:13 +02:00
Tobias Gruetzmacher 9028724a74 Clean up update helper scripts. 2016-04-13 00:52:16 +02:00
Tobias Gruetzmacher 42e43fa4e6 Read starter parameters from class. 
This allows to specify starters in a more declarative and dynamic way.
 2016-04-12 23:11:39 +02:00