dmoonfire/dosage
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Activity
2129 commits 1 branch 0 tags 31 MiB
Commit graph

1429 commits

Author SHA1 Message Date
Techwolf
6a5823b3fe Add StarfireAgency 2019-12-17 01:28:02 +01:00
Techwolf
e62bcb6267 Add Nightshift 2019-12-17 01:28:02 +01:00
Techwolf
8e8d5e5723 Add TracesOfThePast (SFW/NSFW) 2019-12-17 01:28:02 +01:00
Techwolf
57046e23ca Add A&HClub 2019-12-17 01:28:02 +01:00
Techwolf
ae0b702107 Add Bethellium 2019-12-17 01:28:02 +01:00
Techwolf
085e1642fb Add Moonlace 2019-12-17 01:28:02 +01:00
Techwolf
280f4fdbb8 Add CutLoose 2019-12-17 01:28:02 +01:00
Techwolf
81e4942e7c Add Replay 2019-12-17 01:28:02 +01:00
Techwolf
7e6d33ddf5 Fix Lackadaisy 2019-12-17 01:28:02 +01:00
Techwolf
c9e3923442 Fix AlphaLuna 2019-12-17 01:28:02 +01:00
Techwolf
6cfde34663 Fix Altermeta 2019-12-17 01:28:02 +01:00
Techwolf
2916b39c98 Add ApartmentForTwo 2019-12-17 01:28:02 +01:00
Techwolf
ae82096e96 Add QuentynQuinnSpaceRanger 2019-12-17 01:28:02 +01:00
Techwolf
5fb39c5faf Add TalesOfTheQuestor 2019-12-17 01:28:02 +01:00
Techwolf
ff4fa7892d Add QuantumVibe 2019-12-17 01:28:02 +01:00
Techwolf
7d03d3369b Add LastResort 2019-12-17 01:28:02 +01:00
Techwolf
cb360bc983 Add Bloodline 2019-12-17 01:28:02 +01:00
Techwolf
9e52bea334 Add SixPackOfOtters 2019-12-17 01:28:02 +01:00
Techwolf
0648bbadc9 Add Kaspall 2019-12-17 01:28:02 +01:00
Techwolf
379f963442 Add TheClassMenagerie 2019-12-17 01:28:02 +01:00
Techwolf
a6aacf0cc7 Add DMFA side stories 
(this also moves all DMFA modules into their own file)
 2019-12-17 01:28:02 +01:00
Techwolf
b79f22fb65 Mark KeenSpot/GeneCatlow completed :( 2019-12-17 00:05:47 +01:00
Techwolf
f79d3c9309 Fix GunnerkriggCourt 2019-12-17 00:04:20 +01:00
Tobias Gruetzmacher
835f484673 Simplify ComicsKingdom extraction a bit 2019-12-16 23:47:14 +01:00
littauer
4d369376c0 Add ComicsKingdom.com (aka King Features) (#134) 
Note that going back more than about seven days is a paid feature and is untested.
 2019-12-16 23:18:04 +01:00
Tobias Gruetzmacher
adb3db3d22 Add some tests for the update check 2019-12-05 22:23:48 +01:00
Tobias Gruetzmacher
f5253c7f45 Return a fallback "version" if dosage is not installed 
Additionally, inform the user on how to fix the problem. Thanks to twb
for noticing this.
 2019-12-05 21:41:15 +01:00
Tobias Gruetzmacher
7615ee60d1 Switch from pkg_resources to importlib 
One dependency less (at least für Python 3.8+) and it should be faster
(not tested).
 2019-12-05 21:40:45 +01:00
Tobias Gruetzmacher
4511bab996 Fix xkcd pages where comic is linked 2019-12-05 07:10:49 +01:00
Tobias Gruetzmacher
66f154f074 Add throttling for GoComics (fixes #90) 
Since this was the goal of the whole throttling implementation ;)
 2019-12-04 00:28:27 +01:00
Tobias Gruetzmacher
a347bebfe3 Add simple host-based throttling 2019-12-04 00:28:27 +01:00
Tobias Gruetzmacher
e5e7dfacd6 Move basic HTTP setup into a new module 
We now subclass requests' Session to make further extensions of the HTTP
flow possible.
 2019-12-03 23:58:20 +01:00
Tobias Gruetzmacher
d2861d029f Fix "Extra Fabulous Comics" (fixes #129) 2019-12-03 19:50:41 +01:00
Tobias Gruetzmacher
ddba32391b Add BlondeSunrise (fixes #142) 2019-12-03 00:14:57 +01:00
Tobias Gruetzmacher
f5a5106381 Allow combining -n with -c or -a (related to #90) 
This allows fetching "all" comics (or catch up until the last existing
one) while setting an upper bound on how many pages to fetch at the same
time.
 2019-12-01 23:19:06 +01:00
Jakob Kogler
6fd3282047 Add comic "turnoff" (closes #139) 2019-12-01 21:46:00 +01:00
hkocharyan
31309a26d2 fixed oglaf next and previous tags (#141) 2019-11-19 20:56:54 +01:00
Tobias Gruetzmacher
00d0201c5f Fix a bunch of flake8 issues 2019-11-04 00:16:25 +01:00
Techwolf
255fbfa1b4 Add Evon 2019-11-03 23:51:58 +01:00
Techwolf
b230ef31d8 Add Guardia 2019-11-03 23:51:27 +01:00
Techwolf
70223bd38f Add YouSayItFirst 2019-11-03 23:50:47 +01:00
Techwolf
a5a868c2e7 Add NicoleAndDerek 2019-11-03 23:50:47 +01:00
Techwolf
be6a6722b5 Add UnlikeMinerva 2019-11-03 23:50:47 +01:00
Techwolf
fcdbd563a2 Add HavocInc 2019-11-03 23:44:49 +01:00
Techwolf
a575e5e0f0 Add Dissonance 2019-11-03 23:44:24 +01:00
Techwolf
14a01c3e47 Add OutOfPlacers 2019-11-03 23:43:23 +01:00
Techwolf
edc59a86e3 Add PlushAndBlood 2019-11-03 23:42:46 +01:00
Techwolf
4c7a654dcc Add WhiteNoiseLee 2019-11-03 23:42:25 +01:00
Techwolf
12b048d449 Add Savestate 2019-11-03 23:41:48 +01:00
Techwolf
4a783c11ec Add NotAVillain 2019-11-03 23:41:28 +01:00
Techwolf
013d10a1f2 Add SmackJeeves/FurryExperience 2019-11-03 23:40:58 +01:00
Techwolf
2a7d63b7eb Add CrimsonFlag 2019-11-03 23:39:53 +01:00
Techwolf
e06f31784e Add Supercell 2019-11-03 23:38:12 +01:00
Techwolf
f4e3c09717 Add CollegeCatastrophe 2019-11-03 23:37:46 +01:00
Techwolf
59c68bc447 Add NineToNine 2019-11-03 23:37:33 +01:00
Techwolf
a444798460 Add SwordsAndSausages 2019-11-03 23:37:13 +01:00
Techwolf
99ee4147f7 Add SuburbanJungleRoughHousing 2019-11-03 23:36:50 +01:00
Techwolf
f564989e36 Add SuburbanJungle 2019-11-03 23:36:31 +01:00
Techwolf
8a987d3d97 Add ButImACatPerson 2019-11-03 23:36:02 +01:00
Techwolf
44b65f9aac Add OrderOfTheBlackDog 2019-11-03 23:35:40 +01:00
Techwolf
15a5953120 Add ATaleOfTails 2019-11-03 23:35:06 +01:00
Techwolf
323bfc3a6a Add Dreamkeepers 2019-11-03 23:34:38 +01:00
Techwolf
70e78a87de Add CarryOn 2019-11-03 23:34:04 +01:00
Techwolf
e565b083be Add CavesAndCritters 2019-11-03 23:31:17 +01:00
Techwolf
6d76193a9f Add IslaAukate and IslaAukateColor 2019-11-03 23:29:02 +01:00
Techwolf
dd6e536a55 Add Kaerwyn and BlackTapestries 2019-11-03 23:26:03 +01:00
Techwolf
0eccdf737a Add Housepets 2019-11-03 21:53:13 +01:00
Techwolf
48ebffc756 Add HowToBeAWerewolf 2019-11-03 21:39:52 +01:00
Techwolf
f5b7b067b7 Switch AGirlAndHerFed to parser scraper 2019-11-03 21:37:05 +01:00
Techwolf
ed3acd2d2f Fix TheWhiteboard 2019-11-03 21:35:58 +01:00
Techwolf
b055a8574f Fix DominicDeegan 2019-11-03 21:34:53 +01:00
Techwolf
1b87afad7e Fix GrrlPower 2019-11-03 21:34:18 +01:00
Techwolf
9796f994e3 Add SSDD 2019-11-03 21:33:03 +01:00
Techwolf
6b319fdda8 Fix SabrinaOnline 2019-11-03 21:31:50 +01:00
Techwolf
b2db51c361 Add OriginalLife 2019-11-03 21:22:56 +01:00
Techwolf
016516e984 Switch BetterDays to parser scraper 2019-11-03 21:22:29 +01:00
Techwolf
5ca5da51fc Fix Curtailed 2019-11-03 21:20:23 +01:00
Techwolf
79618e2a2f Add first strip URL for PS238 2019-11-03 21:19:10 +01:00
Techwolf
764a8ce6f6 Fix SlightlyDamned 2019-11-03 21:17:31 +01:00
Tobias Gruetzmacher
e24c0ae557 Simplify voting code 
Not sure if I keep this feature, but at least I can now see if anybody
is still using it...
 2019-11-03 21:04:34 +01:00
Tobias Gruetzmacher
ac9d8db1e8 Make sure user agent is in all HTTP requests 2019-11-03 20:17:27 +01:00
Mikkel Høgh
0dea216851 Fix missing returns in RSS parsing (#137) 
Also added a very basic test for feed parsing.
 2019-10-28 16:48:21 +01:00
Tobias Gruetzmacher
328b3cd072 Add new namer "joinPathPartsNamer" 
Additionally, switch some comics which benefit from it to the new namer.
This fixes #127.
 2019-06-30 20:52:15 +02:00
Mikkel Høgh
b8b488670e New comic: The Rock Cocks (#138) 2019-06-30 19:46:39 +02:00
Mikkel Høgh
f29d14c3b4 Fix Gunnerkrigg Court prevSearch matcher (#135) 2019-06-26 23:25:01 +02:00
Mikkel Høgh
78ac7144b2 Fix Girls with Slingshots matchers (#136) 
Domain name and URLs have changed slightly.

Fixes #105.
 2019-06-26 23:22:45 +02:00
Tobias Gruetzmacher
1d910a5bbd Remove pbr from runtime 2019-06-19 07:31:34 +02:00
Gervásio Júnior
6c8814fe40 Fix multiple imgs for json flag & ZenPencils bouncer (#133) 
When using the JSON output flag, if the page has more than one image,
dictionary indexing cannot be used as list.

For the ZenPencils comic, the bouncer is missing, saving the page url
as the root url.
 2019-06-19 07:09:33 +02:00
Arthur Caranta
ffbf494765 Added support for Tripping Over You comic (#130) 2019-04-17 20:28:17 +02:00
Tobias Gruetzmacher
9c4a791853 Repair loader for PyInstaller 3.4 2019-03-08 23:46:50 +01:00
Damjan Košir
78e8f05360 added MonkeyUser 2018-08-28 22:13:48 +12:00
erpbridge
e7410ce26b Update for new LookingForGroup site format (#123) 
Site uses a WordPress build, but does not explicitly match
any existing scrapers. Fixes #120.
 2018-08-12 23:39:51 +02:00
erpbridge
62d3692d3b Update for ElGoonishShive and ElGoonishShiveNP (#122) 
Strip no longer supports ID numbers after May 21 site revamp per Dan Shive. Code here switched to ComicControl. Tested and verified locally.  (fixes #118)
 2018-08-10 09:13:43 +02:00
Tobias Gruetzmacher
49ec3cc3fa Fix (and simplify) GoComics expressions (fixes #117) 2018-07-14 11:00:27 +02:00
Tobias Gruetzmacher
6ba1e49bfd Kenneth Reitz’s Code Style™ 
See http://docs.python-requests.org/en/master/dev/contributing/#kenneth-reitz-s-code-style

Effectively, this removes "visual" indents.
 2018-06-29 19:26:17 +02:00
Tobias Gruetzmacher
fbb3a18c91 Enable warnings and fix some of them 2018-05-23 00:54:40 +02:00
Dirk Reiners
050a0dc97c MenageA3 naming fix 2018-04-23 08:07:41 +02:00
Dirk Reiners
cba9edbdec LifeAintNoPontFarm added 2018-04-23 08:06:13 +02:00
Dirk Reiners
01c1b04778 CyanideAndHappiness fix 2018-04-23 07:53:22 +02:00
Dirk Reiners
fbd2ac2246 Handle get_terminal_size() returning 0 (fixes #106) 2018-04-23 07:50:28 +02:00
Peter Janes
2a2ff2d545 GoComics no longer has nav on the comic's home page. 2018-04-06 14:09:13 -04:00
Tobias Gruetzmacher
1fe98d2f7f Use a diferent div class for GoComics (fixes #102). 2018-03-23 00:29:40 +01:00
Tobias Gruetzmacher
2dbd3382f7 Update LeastICouldDo (fixes #99) 2017-12-15 00:00:25 +01:00
Tobias Gruetzmacher
75aa7207ea Some minor fixes to make some modules work again. 2017-11-27 01:04:35 +01:00
Tobias Gruetzmacher
405c4c0b43 Recreate SluggyFreelance module (fixes #96). 2017-11-26 20:23:33 +01:00
Tobias Gruetzmacher
90685d9b0c Only support modern versions of PyCountry. 2017-11-26 19:29:48 +01:00
Damjan Košir
79a2516c61 deathbulge fix 2017-11-17 21:49:47 +13:00
Tobias Gruetzmacher
d88f6aeee3 Replace online tests with mocks. 
We want to test our code, not the comic modules.
 2017-10-15 14:54:44 +02:00
Tobias Gruetzmacher
f1b83748ed When testing the command line, call main method. 
Previously, we were spawning the main binary in a subprocess, which is
fragile and interacts poorly with some testing frameworks...
 2017-10-15 14:54:44 +02:00
Tobias Gruetzmacher
ac2ca54570 Remove handlers after director run. 2017-10-15 14:54:44 +02:00
Damjan Košir
24862715d5 realised we have a scraper for CMS MenageA3 uses 2017-10-03 21:47:32 +13:00
Damjan Košir
0e0dcf1f8e redoing MenageA3 with ParserScraper (previous search regex was broken) 2017-10-02 21:52:40 +13:00
Tobias Gruetzmacher
6369203bc0 Merge pull request #92 from clonejo/feature/commitstrip 
add a comic plugin for CommitStrip
 2017-09-20 22:46:46 +02:00
Damjan Košir
89a902651c Merge remote-tracking branch 'origin/master' 2017-09-19 22:36:48 +12:00
Damjan Košir
a9d7b4de12 added Deathbulge 2017-09-19 22:36:19 +12:00
clonejo
331faae3ea
add a comic plugin for CommitStrip 2017-09-18 21:31:15 +02:00
glyphy
ad8374d7b8 Fixing the Menagea3 plugin (#91) 
I've changed the menagea3 plugin so it should work with the
new directory structure found on the site.
 2017-09-04 21:19:46 +02:00
Tobias Gruetzmacher
7e0adf1d96 Unify more WordPress-based modules. 2017-05-22 01:17:05 +02:00
Tobias Gruetzmacher
42f66c07b0 Random module fixes. 2017-05-22 00:30:31 +02:00
Tobias Gruetzmacher
f8def5b9db Bugfix: StandardError does not exist in Python 3. 2017-05-21 23:37:09 +02:00
Tobias Gruetzmacher
a99098d5ad Update GoComics module. 2017-05-21 23:10:32 +02:00
Tobias Gruetzmacher
1400879dc8 Fix another set of modules (e, k). 2017-05-17 00:11:29 +02:00
Tobias Gruetzmacher
4ee99eb196 Merge pull request #85 from sizlo/improveordering 
Preserve the order of images in multi image strips for ordered symlink folders
 2017-05-16 23:09:46 +02:00
Tim Brier
95e48b8d8d Keep track of the order of images for multi-image strips in the JSON output 2017-05-15 10:56:47 +01:00
Tobias Gruetzmacher
8b90aa5cfb Some minor style fixes. 2017-05-15 00:54:02 +02:00
Tobias Gruetzmacher
b8484cde50 Fix some more modules. 2017-05-15 00:27:28 +02:00
Tobias Gruetzmacher
ddd3fb418c Remove some broken comics from ComicFury module. 2017-05-14 22:45:12 +02:00
Tobias Gruetzmacher
09687c91f4 Fix some SmackJeeves comics. 2017-05-12 00:32:25 +02:00
sizlo
a83911aa67 Favour the first image we found when we're not expecting multiple images 2017-04-18 21:59:04 +01:00
sizlo
8d84361de4 Preserve the order we found images in when removing duplicate images 2017-04-18 21:58:12 +01:00
Tobias Gruetzmacher
593975d907 Minor cleanups for new modules (see #84). 2017-04-16 01:28:17 +02:00
Tim Brier
233da3e052 Add support for SurvivingTheWorld and TumbleDryComics (#84) 2017-04-16 01:11:30 +02:00
Tobias Gruetzmacher
0973570295 Fix a bunch of modules. 2017-04-16 01:06:41 +02:00
Tobias Gruetzmacher
e6f18a2027 Clean up ComicGenesis 2017-02-27 18:20:54 +01:00
Tobias Gruetzmacher
23dccb184e Replace PyInstaller version hack with something better. 2017-02-14 22:07:52 +01:00
Tobias Gruetzmacher
abb72a3a24 Fix CloneManga modules. 2017-02-13 23:41:45 +01:00
Tobias Gruetzmacher
ebbb27d05d Move xpath_class to helpers module. 2017-02-13 22:41:17 +01:00
Tobias Gruetzmacher
20ab279cde Clean up SmackJeeves... 
Currently only covers already existing modules: Removed 11 broken
modules, added 2 and tried to update comic names and the adult and
endOfLife flags from their index. This isn't helped by the fact that
their search seems to skip some comics...
 2017-02-13 01:46:49 +01:00
Tobias Gruetzmacher
83187b0554 Fix ViiviJaWagner. 2017-02-12 20:29:57 +01:00
Tobias Gruetzmacher
657e61811d Update list of old and removed modules. 2017-02-12 20:17:07 +01:00
Tobias Gruetzmacher
3b6af33ecb Some small module fixes. 2017-02-12 20:15:25 +01:00
Tobias Gruetzmacher
5359dd8629 Update ComicFury again... 2017-02-12 19:50:51 +01:00
Tobias Gruetzmacher
9895014655 Fix PHD with an ugly hack... 2017-02-12 16:21:36 +01:00
Tobias Gruetzmacher
b57945efd1 Update GoComic modules. 2017-02-12 12:21:01 +01:00
Tobias Gruetzmacher
ebe98bc8ba Fix some modules. 2017-02-12 02:16:38 +01:00
Tobias Gruetzmacher
20ca5d7fc2 Fix some modules. 2017-02-06 00:05:05 +01:00
gruetzkopf
edb49faa8b Add support for 'The Monster under the Bed' 2017-01-22 00:11:05 +01:00
Tobias Gruetzmacher
c4a184d173 Remove some vanished modules. 2017-01-12 02:01:10 +01:00
Tobias Gruetzmacher
36ac459bed Add removed GoComics modules to old list. 2017-01-12 01:22:13 +01:00
Tobias Gruetzmacher
a183e812ae Update GoComics module for new site layout. 
(fixes #77)
 2017-01-11 02:21:05 +01:00
Tobias Gruetzmacher
061efaac6e New module for ComicSherpa (removed from GoComics) 2017-01-11 01:34:52 +01:00
John Safrit
969e633877 Fix pattern for The Devils Panties 2017-01-08 17:39:59 -05:00
Tobias Gruetzmacher
3f9feec041 Allow modules to ignore some HTTP error codes. 
This is neccessary since it seems some webservers out there are
misconfigured to deliver actual content with an HTTP error code...
 2016-11-01 18:25:02 +01:00
Tobias Gruetzmacher
46b7a374f6 Small GoComics update. 2016-11-01 02:51:00 +01:00
Tobias Gruetzmacher
f7f4e130bf Small fix to the WLP module. 2016-11-01 02:27:29 +01:00
Tobias Gruetzmacher
bc755d09a3 Apply link modifier to all links. 
This was previously only the "previous link modifier", now it can also
modify "next" and "latest" links. Additionally, the modifier is given
the current URL, so those cases can be distinguished.
 2016-11-01 01:50:44 +01:00
Tobias Gruetzmacher
7fc05f75f5 Remove broken PetiteSymphony comics. 2016-10-31 07:16:10 +01:00
Tobias Gruetzmacher
69e6318f87 Remove ScurryAndCover, too much JavaScript. 2016-10-31 07:04:00 +01:00
Tobias Gruetzmacher
47e2502ec7 Fix a bunch of comic modules. 2016-10-31 06:57:47 +01:00
Tobias Gruetzmacher
446b81fc45 Fix Wumo and friends. 2016-10-30 15:28:54 +01:00
Tobias Gruetzmacher
51ed898f5d Fix some SmackJeeves comics. 2016-10-30 14:30:45 +01:00
Tobias Gruetzmacher
b6d99945f6 Merge pull request #73 from acaranta/master 
Added several SmackJeeves Comics
 2016-10-30 11:55:17 +01:00
Tobias Gruetzmacher
3b9f30affd Update ComicFury modules. 2016-10-30 11:04:45 +01:00
Tobias Gruetzmacher
a02660a7d3 Replace custom @memoized with stdlib @lru_cache. 2016-10-29 00:46:49 +02:00
Tobias Gruetzmacher
9a6a310b76 Fixup copyright years. 2016-10-29 00:21:41 +02:00
acaranta
83880a3cbd corrected RainbowMansion 2016-10-27 09:53:34 +02:00
acaranta
0ed823175c Added even more Smackjeeves comics 2016-10-27 06:58:57 +02:00
acaranta
a5c9a3c35c Added several SmackJeeves Comics 2016-10-26 05:25:13 +02:00
Peter Brunner
19445a83ae Fix smbc 2016-10-18 21:28:42 -04:00
Tobias Gruetzmacher
f94caa8a16 Use terminal size calculation from standard library. 2016-10-14 23:55:10 +02:00
Tobias Gruetzmacher
06be2a026b Move some ex-KeenSpot comics to shorter names. 2016-10-14 14:23:33 +02:00
Tobias Gruetzmacher
b17d6e5f22 Rework/fix KeenSpot modules. 2016-10-14 00:14:53 +02:00
Tobias Gruetzmacher
064e7976ec Add namer for Extra Fabulous Comics. 2016-10-06 00:42:50 +02:00
mostlyuseful
fce7dfff19 Add "Extra Fabulous Comics" comic 2016-10-04 17:06:50 +02:00
Tobias Gruetzmacher
f342a93aa1 Update GoComics module. 2016-10-01 03:39:36 +02:00
Tobias Gruetzmacher
c0d945a563 Update ComicFury modules. 2016-10-01 02:52:33 +02:00
Tobias Gruetzmacher
98c98ddfab Fix some more comic modules (c-f). 2016-09-30 00:15:45 +02:00
Tobias Gruetzmacher
b1d2650615 Fix some modules (a&b). 2016-09-29 01:29:01 +02:00
Damjan Košir
c04c62e92b xkcd now hone with xpaths 2016-08-18 21:28:25 +12:00
Damjan Košir
9ba184eb43 fixing LoadingArtist 2016-08-16 21:20:35 +12:00
Hubert Figuière
afcd19bf5b Added Prince of Sartar Comic 2016-08-08 09:18:33 -04:00
Hubert Figuière
81821dc450 Added Space Junk Arlia comic 2016-08-08 09:18:33 -04:00
Tobias Gruetzmacher
fb37f946e0 Speed up comic module tests. 
This fakes an If-Modified-Since header, so most web servers don't need
to send comic images at all. This should also reduce the amount of data
that needs to be fetched for comic module tests.
 2016-08-01 00:44:34 +02:00
Tobias Gruetzmacher
4f80016bf0 Change robotparser import to make PyInstaller happy. 2016-06-06 22:42:01 +02:00
Tobias Gruetzmacher
64c8e502ca Ignore case for comic download directories. 
Since we already match comics case-insensitive on the command line, this
was a logical step, even if this means changing quite a bit of code that
all tries to resolve the "comic directory" in a slightly different
way...
 2016-06-06 00:08:29 +02:00
Tobias Gruetzmacher
215d597573 Remove DrunkDuck for now. 
- It's been disabled for ages
- Needs a major rework
- I don't want to add that many comics anyways...
- This also gets rid of make_scraper :)
 2016-06-05 22:22:17 +02:00
Tobias Gruetzmacher
67d0d38100 Migrate SnafuComics to single-class module. 2016-06-05 22:12:16 +02:00
Tobias Gruetzmacher
125c96e9dc Remove command to download ALL comics... 2016-06-05 21:57:56 +02:00
Tobias Gruetzmacher
df2048cb34 Keep track of removed and moved comics (fixes #41). 
I plan on keeping this list for at least ~ 2 releases and then purging
older entries...
 2016-06-05 21:47:58 +02:00
Tobias Gruetzmacher
9b755a7e6c Restore BobWhite. 2016-06-05 18:32:27 +02:00
Tobias Gruetzmacher
603fd62a1e Fix workaround for PyInstaller... 2016-06-05 16:01:35 +02:00
Tobias Gruetzmacher
295b53a2d3 Fix name overrides (broken by 51008a). 2016-06-05 10:03:29 +02:00
Tobias Gruetzmacher
844bec09ba Remove another dead comic from ComicFury. 2016-06-05 01:06:04 +02:00
Tobias Gruetzmacher
12123961a4 Fix error in PyInstaller packaged application. 2016-06-05 00:34:16 +02:00
André-Patrick Bubel
2b8e948868 Add String Theory comic 2016-06-01 11:19:17 +00:00
André-Patrick Bubel
192751073c Add KillSixBillionDemons comic 2016-05-31 07:28:32 +00:00
Tobias Gruetzmacher
807bee6342 Migrate GoComics to single-class module. 2016-05-23 00:01:10 +02:00
Tobias Gruetzmacher
2c8e57bdea Migrate Creators to single-class module. 2016-05-22 23:56:59 +02:00
Tobias Gruetzmacher
f5dff27b0a Migrate SmackJeeves to single-class module. 2016-05-22 23:54:21 +02:00
Tobias Gruetzmacher
1ea20e1743 Migrate WebcomicFactory to single-class module. 2016-05-22 23:40:58 +02:00
Tobias Gruetzmacher
c62a7283a2 Migrate ComicFury to single-class module. 2016-05-22 23:31:53 +02:00
Tobias Gruetzmacher
1834bf179f Migrate Arcamax to single-class module. 2016-05-22 23:17:24 +02:00
Tobias Gruetzmacher
f29472c143 Make auto-update script more flexible. 2016-05-22 23:06:05 +02:00
Tobias Gruetzmacher
e4650d5941 Remove make_scraper from Nitrocosm. 2016-05-21 14:35:53 +02:00
Tobias Gruetzmacher
b6eb8ab8ef Remove make_scraper from SandraAndWoo 2016-05-21 14:12:11 +02:00
Tobias Gruetzmacher
4630ea047c Implement Oglaf's strange navigation (fixes #33) 
(also should fix wummel#91)
 2016-05-21 02:38:07 +02:00
Tobias Gruetzmacher
51008a975b Refactor: Introduce generator methods for scrapers 
This allows one comic module class to generate multiple scrapers. This
change is to support a more dynamic module system as described in #42.
 2016-05-21 01:29:36 +02:00
Tobias Gruetzmacher
89cfd9d310 Add comics from catomix.com. 2016-05-16 23:55:41 +02:00
Tobias Gruetzmacher
a6cf4e7040 Fix some more comic modules. 2016-05-16 23:16:29 +02:00
Tobias Gruetzmacher
be1a63da0c Update GoComics comic list. 2016-05-16 18:26:45 +02:00
Tobias Gruetzmacher
6d3f74142c Move command line tool into package. 
This way we can use the default Python console_scripts install process.
 2016-05-16 14:57:47 +02:00
Tobias Gruetzmacher
b9d9564085 Fix Dilbert (fixes #44). 2016-05-16 01:21:23 +02:00
Tobias Gruetzmacher
e9b3c487c0 Remove some dead comics. 2016-05-16 01:10:20 +02:00
Tobias Gruetzmacher
bd60155d9f Some more ComicFury comics gone... 2016-05-16 00:53:22 +02:00
Tobias Gruetzmacher
849e60e795 Remove make_scraper magic from webcomiceu. 2016-05-07 03:20:01 +02:00
Tobias Gruetzmacher
975d2376bf Another round of comic module fixes. 2016-05-07 01:50:10 +02:00
Tobias Gruetzmacher
efe1308db2 Replace home-grown Python2/3 compat. with six. 2016-05-05 23:33:48 +02:00
Tobias Gruetzmacher
77ed0218e0 Fix some comic modules. 2016-05-05 20:55:14 +02:00
Tobias Gruetzmacher
bb2ac39639 Fix some URLs. 2016-05-05 10:12:03 +02:00
Tobias Gruetzmacher
d05316e3ac Seems ComicFury is deleting comics regularly... 
Well, there's nothing we can do: Remove them.
 2016-05-04 08:26:53 +02:00
Tobias Gruetzmacher
0c1aa9e8bd Move libxml < 2.9.3 workaround to base class. 2016-05-02 23:22:06 +02:00
Tobias Gruetzmacher
b93a8fde65 Move PensAndTales comics and fix them. 2016-05-02 22:32:14 +02:00
Tobias Gruetzmacher
4006ced43d Move all HijinksEnsue comics into alphabetic files. 2016-05-02 01:25:34 +02:00
Tobias Gruetzmacher
d5f91ecfd2 Fix some modules in m.py. 2016-04-30 01:59:28 +02:00
Tobias Gruetzmacher
1d52d33311 Remove missing SmackJeeves comics. 2016-04-30 00:56:20 +02:00
Tobias Gruetzmacher
d796f3476c Fix some modules in d.py. 2016-04-30 00:44:18 +02:00
Tobias Gruetzmacher
cc16fea880 Fix some modules in c.py 2016-04-29 00:35:02 +02:00
Tobias Gruetzmacher
1d94439715 Fix some more comic modules. 2016-04-27 00:31:27 +02:00
Tobias Gruetzmacher
8b1ac4eb35 Fix "tagsoup" on SmackJeeves 
Unfortunatly, browsers render < outside of HTML tags differently then
libXML until recently (libXML 2.9.3), so we need to preprocess pages
before parsing them...

(This was fixed in libXML commit 140c25)
 2016-04-26 08:05:38 +02:00
Tobias Gruetzmacher
035d6e94e4 Allow output level for warnings and errors. 2016-04-26 07:53:53 +02:00
Tobias Gruetzmacher
8ddf553eb4 Fix some more SmackJeeves modules. 2016-04-22 01:04:47 +02:00
Tobias Gruetzmacher
fd85c8583a Unify similar code in fetchUrl and fetchText 2016-04-22 00:42:46 +02:00
Tobias Gruetzmacher
6574997e01 Refactor: All the other class methods. 
Turns out, it would have been better if all methods had been instance
methods and not class methods. This finished a big chunk of the rework
needed for #42.
 2016-04-21 23:52:31 +02:00
Tobias Gruetzmacher
0d436b8ca9 Refactor: url modifiers to normal methods. 
As before, to implement #42 these might want to access information from
the instance, so they should be normal methods.
 2016-04-21 21:39:25 +02:00
Tobias Gruetzmacher
c3f32dfef7 Refactor: Make namer a method. 
When #42 is realized, the naming of files might differ between comic
modules, so the namer's logical location is the instance, not the class.
 2016-04-21 08:20:49 +02:00
Tobias Gruetzmacher
5bd2a49f48 Add debug output on matched XPath/CSS expression. 2016-04-20 23:51:54 +02:00
Tobias Gruetzmacher
fe51a449df Update SmackJeeves 
- Now uses _ParserScraper, which makes the pattern quite a bit more
  generic and IMHO more readable
- remove make_scraper magic
- No new comics, only fixed existing ones and removed some dead ones.
 2016-04-20 23:36:45 +02:00
Tobias Gruetzmacher
190cd3b063 Convert language & getDisabledReasons to methods. 
Both are more properties of a webcomic (this is part of the design
changes for #42)
 2016-04-19 23:53:46 +02:00
Tobias Gruetzmacher
df46907f39 Register EXSLT extensions by default. 
This allows comic module authors to use the full power of regular
expressions in XPath expression, see http://exslt.org/regexp/regexp.html
for usage. Please be aware that these use the prefix re: instead of
regexp: here.
 2016-04-19 23:48:14 +02:00
Tobias Gruetzmacher
4204f5f1e4 Send "If-Modified-Since" header for images. 2016-04-19 00:36:50 +02:00
Tobias Gruetzmacher
13a3409854 Remove some comics that are gone or block us. 2016-04-17 19:42:43 +02:00
Tobias Gruetzmacher
1fbc844077 Update GoComics. 2016-04-17 18:40:09 +02:00
Tobias Gruetzmacher
73e958670d Update ComicFury (again). 2016-04-17 16:19:44 +02:00
Tobias Gruetzmacher
b0481a01f7 Update languages. 2016-04-16 13:14:12 +02:00
Tobias Gruetzmacher
3329027e4b Update ComicFury. 2016-04-16 13:13:47 +02:00
Tobias Gruetzmacher
ee99c087d7 Remove prevUrlMatchesStripUrl. 
It was only used for one test.
 2016-04-16 01:14:26 +02:00
Tobias Gruetzmacher
92a688457a Remove useless indirection. 2016-04-15 23:42:24 +02:00
Tobias Gruetzmacher
52515b5fc5 Update GoComics. 2016-04-15 00:26:14 +02:00
Tobias Gruetzmacher
031a523846 Fix SnafuComics. 2016-04-14 23:52:35 +02:00
Tobias Gruetzmacher
7626b1e100 Webcomics Nation is gone. 2016-04-14 22:46:52 +02:00
Tobias Gruetzmacher
497653c448 Remove make_scraper magic from Arcamax. 2016-04-14 00:17:59 +02:00
Tobias Gruetzmacher
db87ed95e7 Use new features to make modules simpler. 2016-04-13 23:28:43 +02:00
Tobias Gruetzmacher
b266e28ae1 Remove debugging prints 😭 2016-04-13 22:59:06 +02:00
Tobias Gruetzmacher
ff3b824311 Fix variable shadowing... 2016-04-13 22:43:34 +02:00
Tobias Gruetzmacher
060281e5ff Use concrete scraper objects everywhere. 
This is a first step for #42. Since most access to the scraper classes
is through instances, modules can now dynamically override url and name
(name is now a property).
 2016-04-13 22:17:30 +02:00
Tobias Gruetzmacher
0468f2f31a Refactor: Convert starter to simple method. 2016-04-13 20:01:51 +02:00
Tobias Gruetzmacher
16004e43e4 Use default bounceStarter for site modules. 2016-04-13 01:24:13 +02:00
Tobias Gruetzmacher
9028724a74 Clean up update helper scripts. 2016-04-13 00:52:16 +02:00
Tobias Gruetzmacher
42e43fa4e6 Read starter parameters from class. 
This allows to specify starters in a more declarative and dynamic way.
 2016-04-12 23:11:39 +02:00
Tobias Gruetzmacher
b865a171f9 Remove some broken comics. 2016-04-12 08:21:06 +02:00
Tobias Gruetzmacher
4e2e4ac529 Prevent scraper from moving to a different comic. 2016-04-12 08:10:47 +02:00
Tobias Gruetzmacher
443ab119e9 Refresh GoComics list from online directory. 2016-04-12 00:36:33 +02:00
Tobias Gruetzmacher
0e385a3697 Update GoComics (no change in supported comics) 
- remove make_scraper magic
- switch to _ParserScraper
 2016-04-11 22:42:01 +02:00
Tobias Gruetzmacher
ad7a297964 Fix WLP comics. 2016-04-11 01:07:21 +02:00
Damjan Košir
af2e57d850 Added comic ScurryAndCover... 
- Yay, funky JavaScript parsing!
- Start page isn't latest comic...

Updated-by: Tobias Gruetzmacher <tobias-git@23.gs>
 2016-04-11 00:09:53 +02:00
Tobias Gruetzmacher
fa98f6ddbf Move more comics to common WordPressScraper. 2016-04-10 23:04:34 +02:00
Tobias Gruetzmacher
f6e605e146 Fix unicode error in text search. 2016-04-10 13:16:30 +02:00
Tobias Gruetzmacher
bc10bd9a4d Streamline color output. 
- Depend on external colorama instead of embedding an old copy.
- Move most output code into output module.
- Convert pager to context manager.
 2016-04-10 03:45:00 +02:00
Tobias Gruetzmacher
bb5b6ffcec Fix comics in module a.py. 2016-04-07 23:21:31 +02:00
Tobias Gruetzmacher
0033a8046b Fix creators module. 2016-04-07 00:20:03 +02:00
Tobias Gruetzmacher
8768ff07b6 Fix AhoiPolloi, be a bit smarter about encoding. 
HTML character encoding in the context of HTTP is quite tricky to get
right and honestly, I'm not sure if I did get it right this time. But I
think, the current behaviour matches best what web browsers try to do:

1. Let Requests figure out the content from the HTTP header. This
   overrides everything else. We need to "trick" LXML to accept our
   decision if the document contains an XML declaration which might
   disagree with the HTTP header.
2. If the HTTP headers don't specify any encoding, let LXML guess the
   encoding and be done with it.
 2016-04-06 22:22:22 +02:00
Tobias Gruetzmacher
183d18e7bc Skip non-image on xkcd. 2016-04-06 00:50:01 +02:00
Tobias Gruetzmacher
9feaf245f2 Fixed & removed some comics in s.py. 2016-04-06 00:40:13 +02:00
Tobias Gruetzmacher
6bbdcfb341 BloomingFaeries: Don't download every page twice. 
(Also, simplify namer, switch to _ParserScraper)
 2016-04-05 23:58:43 +02:00
Tobias Gruetzmacher
8db6f8e8b7 Fix ZapComics, remove ZebraGirl. 
- ZebraGirl is now ComicFury/ZebraGirl...
 2016-04-04 00:27:11 +02:00
Tobias Gruetzmacher
0bcfb8a82e Move ComicControl into common module. 
- Move all comics using ComicControl into alphabetical files.
- Add BalderDash & Picklewhistle
 2016-04-04 00:12:53 +02:00
Tobias Gruetzmacher
0d453a6858 Move Flowerlark Studios into alphabetical files. 2016-04-03 22:58:01 +02:00
Tobias Gruetzmacher
a9f0dfdce4 Merge pull request #39 from peterjanes/peterjanes/sherman-fix 
Fix Sherman's Lagoon
 2016-04-03 22:20:04 +02:00
Tobias Gruetzmacher
926439cd14 Every comic need an url. 2016-04-03 22:03:16 +02:00
Tobias Gruetzmacher
2c6decb7f5 Move WebcomicFactory in its own module. 
Also, add an updater script for it.
 2016-04-03 21:31:56 +02:00
Peter Janes
759bd0c360 Fix Sherman's Lagoon 2016-04-03 14:54:41 -04:00
Tobias Gruetzmacher
bb1f20d867 Remove make_scraper for most WordPress comics. 
- Dropped KatzenfutterGeleespritzer, because robots.txt.
- Move all WordPress/ComicPress scrapers into alphabetical files.
- Move _WordPressScraper & _ComicPress scraper into common.py.
- Some smaller PEP8 fixes.
 2016-04-02 00:19:53 +02:00
Tobias Gruetzmacher
7f1e136d8b Sort comics alphabetically & PEP8 style fixes. 2016-03-31 23:13:54 +02:00
Tobias Gruetzmacher
d6db1d0b81 Fix a conflict with IPython. 2016-03-20 23:57:07 +01:00
Tobias Gruetzmacher
90dfceaeb1 Remove dead modules (& format). 2016-03-20 20:48:42 +01:00
Tobias Gruetzmacher
f243096d49 Fix GastroPhobia, remove GeneralProtectionFault. 
(& formatting)
 2016-03-20 20:11:21 +01:00
Tobias Gruetzmacher
cfcfcc2468 Switch plugin loading to pkgutil. 
This should work with all PEP-302 loaders that implement iter_modules.
Unfortunatly, PyInstaller (which I plan to use for Windows releases)
does not support it, so we don't get around a special case. Anyways,
this should help for #22.
 2016-03-20 15:13:24 +01:00
Tobias Gruetzmacher
1af022895e Fix NuklearPower (fixes #38). 
Also remove make_scraper magic.
 2016-03-17 23:19:52 +01:00
Tobias Gruetzmacher
552f29e5fc Update ComicFury comics. (+871, -245) 
- Remove make_scraper magic
- Switch to HTML parser
- Update parsing of comic listing.
 2016-03-17 00:44:06 +01:00
Tobias Gruetzmacher
6727e9b559 Use vendored urllib3. 
As long as requests ships with urllib3, we can't fall back to the
"system" urllib3, since that breaks class-identity checks.
 2016-03-16 23:18:19 +01:00
Damjan Košir
615f094ef3 fixing EdmundFinney 2016-03-14 20:32:18 +13:00
Tobias Gruetzmacher
c4fcd985dd Let urllib3 handle all retries. 2016-03-13 21:30:36 +01:00
Tobias Gruetzmacher
78e13962f9 Sort scraper modules (mostly for test stability). 2016-03-13 20:24:21 +01:00
Tobias Gruetzmacher
017d35cb3c Fallback version if pkg_resources not available. 
This helps for Windows packaging.
 2016-03-03 01:05:36 +01:00
Johannes Schöpp
351fa7154e Modified maximum page size 
Fixes #36
 2016-03-01 22:19:44 +01:00
Damjan Košir
b0dc510b08 adding LastNerdsOnEarth 2016-01-03 14:16:58 +13:00
Damjan Košir
a1e79cbbf2 fixing Fragile 2016-01-03 14:08:49 +13:00
Tobias Gruetzmacher
81827f83bc Use GitHub releases API for update checks. 2015-11-06 23:07:19 +01:00
Tobias Gruetzmacher
a41574e31a Make version fetching a bit more robust (use pbr). 2015-11-06 22:08:14 +01:00
Tobias Gruetzmacher
64f7e313d5 Remove make_scraper magic from footloosecomic.py. 2015-11-05 00:03:13 +01:00
Tobias Gruetzmacher
7f7a69818b Remove make_scraper magic from creators module. 2015-11-04 23:43:31 +01:00
Tobias Gruetzmacher
94470d564c Fix import for Python 3. 2015-11-03 23:40:45 +01:00
Tobias Gruetzmacher
b819afec39 Switch build to PBR. 
This gets us:
- Automatic changelog
- Automatic authors list
- Automatic git version management
 2015-11-03 23:27:53 +01:00
Tobias Gruetzmacher
dc22d7b32a Add CatNine comic. 2015-11-02 23:29:56 +01:00
Tobias Gruetzmacher
10d9eac574 Remove support for very old versions of "requests". 2015-11-02 23:24:01 +01:00
MariusK
3e1ea816cc Fixed 'Ruthe' 2015-10-02 13:52:44 +02:00
Helge Stasch
48d8519efd Changed Goblins comic - moved to new scraper and fixed minor issues with some comics (old scrapper was unstable for some comics of Goblins) 2015-09-28 23:50:15 +02:00
Helge Stasch
17fbdf2bf7 Added comic "Ahoy Earth" 2015-09-27 00:44:47 +02:00
Tobias Gruetzmacher
d72ceb92d5 BloomingFaeries: Remove imageUrlModifier (not needed). 2015-09-04 00:37:05 +02:00
Tobias Gruetzmacher
abd80a1d35 Merge pull request #28 from KevinAnthony/master 
added comic Blooming Faeries
 2015-09-03 23:26:37 +02:00
Tobias Gruetzmacher
b737218182 ZenPencils: Allow multiple images per page. 2015-09-03 23:24:28 +02:00
Kevin Anthony
62ec1f1d18 Removed debugging print state 2015-09-02 11:22:24 -04:00
Kevin Anthony
d7180eaf99 removed bad whitespace 2015-09-02 11:04:32 -04:00
Kevin Anthony
6e8231e78a Added Namer to BloomingFaeries since the web comic author doesn't seem intrested in sticking to any kind of file naming convention 2015-09-02 11:01:48 -04:00
Kevin Anthony
1045bb7d4a added comic Blooming Faeries 2015-09-02 10:13:42 -04:00
Damjan Košir
11f0aa3989 created Wordpress Scraper class 2015-08-11 21:31:45 +12:00
Damjan Košir
0a5b792c32 added Fragile (English and Spanish) 2015-08-07 23:37:10 +12:00
Damjan Košir
fd9c480d9c adding bonus panel to SWBC and multiple images flag to ParserScraper 2015-08-03 22:58:44 +12:00
Damjan Košir
f8a163a361 added a CMS ComicControl, moved some existing comics there, added StreetFighter and Metacarpolis 2015-08-03 22:40:06 +12:00
Damjan Košir
648a84e38e added Sharksplode 2015-08-03 22:20:17 +12:00
Damjan Košir
c19806b681 added AoiHouse 2015-07-31 23:33:30 +12:00
Damjan Košir
2201c9877a added KiwiBlitz 2015-07-31 23:09:56 +12:00
Damjan Košir
fe22df5e5b added LetsSpeakEnglish 2015-07-31 23:06:06 +12:00
Damjan Košir
79ec427fc0 added CatVersusHuman 2015-07-30 22:16:34 +12:00
Tobias Gruetzmacher
303432fc68 Also use css expressions for textSearch. 2015-07-18 01:22:40 +02:00
Tobias Gruetzmacher
6a70bf4671 Enable some comics based on current policy. 2015-07-18 01:21:29 +02:00
Tobias Gruetzmacher
6b0046f9b3 Fix small typos. 2015-07-18 00:11:44 +02:00
Tobias Gruetzmacher
68d4dd463a Revert robots.txt handling. 
This brings us back to only honouring robots.txt on page downloads, not
on image downloads.

Rationale: Dosage is not a "robot" in the classical sense. It's not
designed to spider huge amounts of web sites in search for some content
to index, it's only intended to help users keep a personal archive of
comics he is interested in. We try very hard to never download any image
twice. This fixes #24.

(Precedent for this rationale: Google Feedfetcher:
https://support.google.com/webmasters/answer/178852?hl=en#robots)
 2015-07-17 20:46:56 +02:00
Tobias Gruetzmacher
7d3bd15c2f Remove AbleAndBaker, site is gone. 2015-07-16 00:49:48 +02:00
Tobias Gruetzmacher
472afa24d3 GoComics doesn't allow spiders, disable them... 
This removes 757 comics, including quite popular ones like Calvin and
Hobbes, Garfield, FoxTrot, etc. :(
 2015-07-16 00:36:10 +02:00
Tobias Gruetzmacher
7c15ea50d8 Also check robots.txt on image downloads. 
We DO want to honour if images are blocked by robots.txt
 2015-07-15 23:50:57 +02:00
Tobias Gruetzmacher
5affd8af68 More relaxed robots.txt handling. 
This is in line with how Perl's LWP::RobotUA and Google handles server
errors when fetching robots.txt: Just assume access is allowed.

See https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
 2015-07-15 19:11:55 +02:00
Tobias Gruetzmacher
88e387ad15 Add Sleepless Domain. 2015-07-12 18:31:21 +02:00
Tobias Gruetzmacher
0b6d7425e1 Remove BladeKitten. 
It's not available online anymore, only in print or as a PDF download.
 2015-07-11 01:29:21 +02:00
Tobias Gruetzmacher
808b624e5f Remove hard dependency on pycountry again. 
This basically reverts commit 86b31dc12b.

It now works like this: If the use has pycountry installed, it is used.
If not, Dosage falls back to a small internal list generated from
pycountry by scripts/mklanguages.py.

This means additional work if we ever decide to translate Dosage, since
pycountry already has all the translations for language names...

This fixes #23.
 2015-07-11 01:27:39 +02:00
Tobias Gruetzmacher
d97a9c63e4 Add Erstwhile. 2015-07-10 01:14:56 +02:00
Damjan Košir
7abca1222b added NerfNow 2015-07-07 22:18:06 +12:00
Damjan Košir
119a3cd13a added text to ScandinaviaAndTheWorld 2015-07-07 19:48:25 +12:00
Damjan Košir
5f243e3868 not a comic 2015-07-05 18:33:14 +12:00
Damjan Košir
5e7ad33fc8 Nnewts disabled 2015-07-05 18:32:33 +12:00
Damjan Košir
45012ff9c3 BladeKitten disabled 2015-07-05 18:31:38 +12:00
Tobias Gruetzmacher
0c6feec8cd Fix module name EastCoastVsWestCoast. 2015-06-24 00:51:42 +02:00
Damjan Košir
96572e8cba added TheMelvinChronicles 2015-06-12 21:00:11 +12:00
Damjan Košir
6412e6e542 fixed Spinnerette 2015-06-08 20:31:13 +12:00
Damjan Košir
3d8a49d228 realised TheWebcomicFactory is actually 28 comics... added them 2015-06-07 21:33:59 +12:00
Damjan Košir
05bb22b3ef added TheWebcomicFactory 2015-06-06 14:25:32 +12:00
Damjan Košir
c98800388e added Sithrah 2015-06-04 19:24:55 +12:00
Damjan Košir
010b4bf669 renaming comicpress to wordpress (as it's not just for the comicpress theme) 2015-06-04 19:12:40 +12:00
Damjan Košir
bc91f5f1fb added MistyTheMouse 2015-06-04 19:06:40 +12:00
Damjan Košir
e2d01e4924 fixed ScandinaviaAndTheWorld 2015-06-04 18:58:59 +12:00
Damjan Košir
545a67111e fixed Alice 2015-06-01 15:15:34 +12:00
Damjan Košir
a08ad2dc80 fixed GoGetARoomie 2015-06-01 15:11:16 +12:00
Damjan Košir
ceb19ed2bc fixed Wulffmorgenthaler (now Wumo), added TruthFacts and MeAndDanielle 2015-06-01 12:14:52 +12:00
Damjan Košir
4cd88ecdc0 fixed WormWorldSaga 2015-06-01 11:45:22 +12:00
Damjan Košir
ea6cb925a6 fixed LoadingArtist 2015-06-01 11:33:50 +12:00
Damjan Košir
e268b09567 fixed EarthsongSaga 2015-06-01 11:19:02 +12:00
Damjan Košir
29c8d2eea0 fixed Meek 2015-05-31 23:41:12 +12:00
Damjan Košir
9be6f613e4 fixed MysteriesOfTheArcana 2015-05-31 23:39:04 +12:00
Damjan Košir
3ea8236224 fixed FowlLanguage 2015-05-31 23:29:34 +12:00
Damjan Košir
c1245a85ad moved Footloose, added Cherry, Desigaspring 2015-05-31 23:23:02 +12:00
Damjan Košir
01aeebfbe4 fixed Footloose 2015-05-31 23:16:12 +12:00
Damjan Košir
029fa74067 fixed Bardsworth 2015-05-31 23:03:40 +12:00
Damjan Košir
f3036de8fd fixed Pimpette 2015-05-31 22:57:25 +12:00
Damjan Košir
df7404fd7c fixed CatsAndCameras 2015-05-31 22:50:17 +12:00
Damjan Košir
d4cc8ac857 added buni 2015-05-27 20:36:11 +12:00
Damjan Košir
9beeceffad added BusinessCat and HappyJar 2015-05-27 20:34:51 +12:00
Damjan Košir
d970d27b14 removing duplicate 2015-05-27 00:10:46 +12:00
Damjan Košir
33abd95348 fixed TheGentlemansArmchair 2015-05-26 23:48:22 +12:00
Damjan Košir
5e123ae79e fixed DarkWings (now available under the real name Eryl as well), added Ashes, Laiyu, NoMoreSavePoints and EasilyAmused 2015-05-26 23:43:15 +12:00
Damjan Košir
9adb020fc2 fixed DemolitionSquad 2015-05-26 22:59:25 +12:00
Damjan Košir
605c5f8619 fixed PokeyThePenguin 2015-05-26 22:31:43 +12:00
Damjan Košir
766b7ba99d fixed ProperBarn, added 2214 and OTE 2015-05-26 22:16:55 +12:00
Damjan Košir
2c41435ceb fixing HijiNKS ENSUE and added all 4 comics on that page 2015-05-26 22:06:55 +12:00
Damjan Košir
465e7eaf6f fixing CowboyJedi kinda... there is currently no comic on the front page and the author knows it 2015-05-26 21:35:36 +12:00
Damjan Košir
529a41397a fixing CorydonCafe 2015-05-26 21:32:25 +12:00
Damjan Košir
c3abb93e99 fixing ChainsawSuit 2015-05-26 19:53:04 +12:00
Damjan Košir
f8690af029 fixing Curvy 2015-05-26 19:47:31 +12:00
Damjan Košir
36c790fa4b fixing CraftedFables 2015-05-26 19:32:12 +12:00
Damjan Košir
7067c51056 fixed CheckerboardNightmare 2015-05-25 22:19:36 +12:00
Damjan Košir
5569439c43 fixed 16 comics 2015-05-25 21:57:06 +12:00
Damjan Košir
3edaa97fb9 fixing KatzenfutterGeleespritzer 2015-05-25 20:06:58 +12:00
Damjan Košir
8a245e1d10 fixing BloodBound 2015-05-21 00:04:07 +12:00
Damjan Košir
dc2349951a moving BroodHollow to comicpress 2015-05-21 00:00:35 +12:00
Damjan Košir
a05ae9c75d fixing PandyLand 2015-05-20 23:56:49 +12:00
Damjan Košir
fd60065591 fixing OnTheEdge 2015-05-20 23:50:18 +12:00
Damjan Košir
80b783c016 fixing CourtingDisaster 2015-05-20 23:16:54 +12:00
Damjan Košir
ff239ff58e Merge branch 'comicpress' 2015-05-20 23:12:03 +12:00
Damjan Košir
77c5dbce9b better prevSearch for comic press 2015-05-20 23:08:02 +12:00
Damjan Košir
bc4e7a03f2 fixed BroodHollow 2015-05-20 23:03:15 +12:00
Damjan Košir
8de620c78b fixed CigarroAndCerveja 2015-05-20 22:58:13 +12:00
Damjan Košir
4529fdee3b adding no downsize option 2015-05-20 22:38:29 +12:00
Damjan Košir
77a9cce00d fixing Hipsters 2015-05-19 19:49:45 +12:00
Damjan Košir
79d775a8d9 adding comicpress scraper 2015-05-16 00:15:32 +12:00
Damjan Košir
962286d391 fixed OctopusPie 2015-05-14 23:06:12 +12:00
Damjan Košir
3bbf2d5c23 fixing neko the kitty 2015-05-14 22:42:04 +12:00
Damjan Košir
f75fc62e84 fixing pebbleversion 2015-05-14 22:33:46 +12:00
Helge Stasch
5a1ef9b791 Fixed problem with LookingForGroup comic 2015-05-07 13:57:10 +02:00
Damjan Košir
9a009018c7 adding strip Moonsticks 2015-05-07 23:00:55 +12:00
Helge Stasch
64a875388f Added Comic MaxOveracts 2015-05-04 14:06:01 +02:00
Marc Winkelmann
69e5b8ad93 Shermans Lagoon and On The Fastrack working again. Also corrected name. 2015-05-02 22:27:08 +02:00
DirkReiners
1438330a94 Fixes and Additions... 
Fixed SabrinaOnline
Fixed SMBC
Added StandStillStaySilent (partial, prevsearch not working yet)
 2015-04-29 10:37:14 -05:00
DirkReiners
749beff7a3 Added MareInternum (marecomic.com) 2015-04-29 10:36:12 -05:00
DirkReiners
273b429fcd Merge branch 'master' of https://github.com/webcomics/dosage 2015-04-29 09:51:47 -05:00
Damjan Košir
391313972c fixed ManlyGuysDoingManlyThings 2015-04-26 23:47:38 +12:00
Damjan Košir
9837a87a43 fixed omake teather 2015-04-26 23:32:22 +12:00
Damjan Košir
8df9d20556 added doctor cat 2015-04-26 22:32:52 +12:00
Damjan Košir
dc427d6066 fixed the gamercat 2015-04-26 21:52:31 +12:00
Damjan Košir
561005887a unneeded max 2015-04-26 00:23:45 +12:00
Damjan Košir
ac7b0d7e0e adding parallel run option 2015-04-26 00:19:08 +12:00
Damjan Košir
1e94a3c7c5 now the same as offical version 2015-04-25 20:52:03 +12:00
Damjan Košir
dae2698102 removing mismerge 2015-04-25 20:40:28 +12:00
Damjan Košir
dc014a7cb4 Merge remote-tracking branch 'upstream/master' 
Conflicts:
	dosagelib/plugins/e.py
	dosagelib/plugins/i.py
	dosagelib/plugins/n.py
	dosagelib/plugins/s.py
	dosagelib/plugins/t.py
	dosagelib/plugins/w.py
 2015-04-25 20:28:27 +12:00
DirkReiners
b8ef6958b9 Merge branch 'master' of https://github.com/webcomics/dosage 2015-04-24 15:38:36 -05:00
Helge Stasch
4cdd92dcd7 Added comic Magellan 2015-04-23 09:12:24 +02:00
Tobias Gruetzmacher
9f33c31c68 Merge pull request #12 from Freestila/master 
Changed comic name, since comic is named FowlLanguage instead of FoulLan...

Conflicts:
	dosagelib/plugins/f.py
 2015-04-22 22:24:26 +02:00
Tobias Gruetzmacher
bf9f45b380 Switch to setuptools and cleanup metadata. 
py2exe support is gone for now, will be restored later.
 2015-04-22 22:22:03 +02:00
Helge Stasch
8218e805b2 Changed comic name, since comic is named FowlLanguage instead of FoulLanguage 2015-04-22 21:25:10 +02:00
Tobias Gruetzmacher
bf9bf5e9b0 Merge pull request #11 from Freestila/master 
Added "Ralf the Destroyer"
 2015-04-21 23:46:11 +02:00
Tobias Gruetzmacher
86b31dc12b Depend on pycountry directly. 2015-04-21 21:56:54 +02:00
Helge Stasch
d7e9c8eb94 Added "Ralf the Destroyer" 2015-04-21 19:12:40 +02:00
Tobias Gruetzmacher
d5e7690419 Fix size comparison for RSS & HTML output. 
This was always broken, but somehow worked with Python 2.7 (WTF?). Now
that we test with Pillow, this code path runs with Python 3 and throws
an error.
 2015-04-21 00:01:23 +02:00
Tobias Gruetzmacher
ff21df596b Remove descriptions and genres (closes #9). 
Maintaining the descriptions creates quite a bit of overhead (finding
them, copying them, checking if they are still correct) for a minimal
user benefit.

PS: Viewing this diff should be easier in a difftool that shows changes
in a line, for example kdiff3.
 2015-04-20 20:29:09 +02:00
Tobias Gruetzmacher
3b33129e58 Fix ViiviJaWagner. 2015-04-18 22:45:13 +02:00
Tobias Gruetzmacher
e8af5adcb8 Update list of supported GoComics comics. 2015-04-18 02:04:31 +02:00
Tobias Gruetzmacher
f0831a1f0f Fix and update ArcaMax (fixes #8). 2015-04-17 21:53:13 +02:00
DirkReiners
99f33151e2 Merge branch 'master' of https://github.com/webcomics/dosage 2015-04-16 18:36:42 -05:00
DirkReiners
8f3a9f660a Fixed ASofterWorld 2015-04-16 18:35:21 -05:00
DirkReiners
49b964cb3c Added PS238 2015-04-16 18:20:14 -05:00
Manabi
65c021ef2b Fixed IAmArg 2015-04-15 14:43:06 -04:00
Manabi
475739ea60 Fixing DogHouseDiaries 2015-04-15 12:56:03 -04:00
Manabi
c0619e8dca Fixing DogHouseDiaries 2015-04-15 12:51:45 -04:00
Manabi
2b98a9023e Added Peanuts Begins & Wizard of Id Classics 2015-04-13 22:26:12 -04:00
Tobias Gruetzmacher
974752951b Fix xkcd (closes #3), remove adult tag (fixes wummel#85). 2015-04-12 20:06:34 +02:00
Tobias Gruetzmacher
5934f03453 Merge branch 'htmlparser' - I think it's ready. 
This closes pull request #70.
 2015-04-01 22:13:55 +02:00
Tobias Gruetzmacher
614c25e278 Fix coding style. 2015-03-22 17:13:53 +01:00
Tobias Gruetzmacher
e94e2ae432 Merge pull request #95 from serenitas50/master 
Added comic Beetlebum (http://blog.beetlebum.de/).
 2015-03-22 17:04:36 +01:00
Tobias Gruetzmacher
b5ed4c56b6 Merge pull request #94 from Manabi/master 
Added definition for Drive comic

Conflicts:
	dosagelib/plugins/g.py
 2015-03-22 16:34:07 +01:00
Tobias Gruetzmacher
b5368b366a Merge Gaia(German), SandraAndWoo(German) into common base. 
This also fixes #97 by correcting the imageSearch regex.
 2015-02-04 19:41:52 +01:00
Manabi
f85464ccb2 Fixed unclosed ' error 
Lines 293/294 should have been one line, this is now fixed.
 2015-02-02 04:35:49 -05:00
Manabi
190f53ee4d Fixing name of GunnkriggCourt 
Existing name was missing a g.
 2015-02-02 04:24:32 -05:00
Serenitas50
94004846cd Added comic Beetlebum (http://blog.beetlebum.de/). 2015-01-31 22:07:35 -02:00
Manabi
a5b0d0c5de Added definition for Drive comic 2015-01-26 04:21:24 -05:00
Dirk Reiners
b710d3fa81 Merge branch 'master' of https://github.com/wummel/dosage 2015-01-16 13:24:48 -06:00
Dirk Reiners
c6f0dd6117 PiledHigherAndDeeper: Fix for new website format 2015-01-16 12:06:17 -06:00
Dirk Reiners
e25270c866 Dilbert: Fix for new websitre format 2015-01-16 12:05:53 -06:00
Dirk Reiners
3724eba835 Cyanide And Happiness: Fix for new website format 2015-01-16 12:05:36 -06:00
Tobias Gruetzmacher
f8531eca57 Move SinFest back to KeenSpot namespace. 2015-01-16 00:16:28 +01:00
Tobias Gruetzmacher
4733153d01 Merge pull request #87 from rpglover64/master 
Update SinFest to work with new website.
 2015-01-16 00:15:04 +01:00
Alex Rozenshteyn
a0506b22f0 Update ZenPencils URL. 2014-12-16 13:51:52 -05:00
Alex Rozenshteyn
51996e45ed Update SinFest to work with new website. 2014-12-16 12:01:54 -05:00
Tobias Gruetzmacher
2c1ff889fa Fix scope in HTML output. 2014-12-10 00:57:17 +01:00
Tobias Gruetzmacher
b7bc16650a Merge branch 'carlosefonseca/master' 2014-12-10 00:07:21 +01:00
Tobias Gruetzmacher
5af4f45505 Merge branch 'zac9/patch-2' 2014-12-10 00:03:08 +01:00
Tobias Gruetzmacher
32265c99d7 Merge branch 'zac9/patch-1' 2014-12-10 00:00:51 +01:00
Carlos Fonseca
04cc07a466 Added comic Nimona 2014-12-08 13:28:37 +00:00
mbrandis
25cf4888ae - Adapted ShermansLagoon 
- Better version of OnTheFastTrack
 2014-11-14 20:37:06 +01:00
mbrandis
c63f927e5c - Modified OnTheFasttrack adapting the new API. 2014-11-14 20:09:42 +01:00
mbrandis
cd48801b0d - Added next and previous day at end of page. 2014-11-14 15:39:42 +01:00
Dirk Reiners
fda654b5e0 Some fixes... 
AbstruseGoose: fixed prev
Carciphona: fixed latest
Curtailed: fixed image and prev (moved to WP)
DorkTower: fixed image search
GrrlPower: fixed site name issue
MadamAndEve: archive not updated in a long time, but current strip is.
Works, but needs to be run daily.
PennyArcade: fixed namer
PvPonline: fixed prev
 2014-10-24 16:42:32 -05:00
Dirk Reiners
77a5e09c10 Minor fix for using pathes to pick comics 2014-10-24 16:39:40 -05:00
Tobias Gruetzmacher
6769e1eb36 Add StrongFemaleProtagonist. 
This uses the _ParserScraper and CSS selectors.
 2014-10-13 23:39:50 +02:00
Tobias Gruetzmacher
1d52d6a152 Add support for CSS selectors to HTML parser. 
Each comic module author can decide if she wants to use CSS or XPath,
not a mix of both. Using CSS needs the cssselect python module and the
module gets disabled if it is unavailable.
 2014-10-13 22:43:06 +02:00
Tobias Gruetzmacher
17bc454132 Bugfix: Don't assume RE patterns in base class. 2014-10-13 22:29:47 +02:00
Tobias Gruetzmacher
e92a3fb3a1 New feature: Comic modules ca be "disabled". 
This is modeled parallel to the "adult" feature, except the user can't
override it via the command line. Each comic module can override the
classmethod getDisabledReasons and give the user a reason why this
module is disabled. The user can see the reason in the comic list (-l or
--singlelist) and the comic module refuses to run, showing the same
message.

This is currently used to disable modules that use the _ParserScraper if
the LXML python module is missing.
 2014-10-13 21:43:46 +02:00
Tobias Gruetzmacher
d495d95ee0 Refactor: Move repeated check into its own function. 2014-10-13 21:29:54 +02:00
Tobias Gruetzmacher
3235b8b312 Pass unicode strings to lxml. 
This reverts commit fcde86e9c0 & some
more. This lets python-requests do all the encoding stuff and leaves
LXML with (hopefully) clean unicode HTML to parse.
 2014-10-13 19:39:48 +02:00
zac9
6ca200419a Update s.py 2014-09-28 19:48:26 -07:00
zac9
5b7ab5a711 Update o.py 2014-09-28 19:41:29 -07:00
zac9
491b5457b2 Added comic ShotgunShuffle 2014-09-28 06:29:02 -07:00
Bastian Kleineidam
731291979d Fixed RedMeat. 2014-09-22 22:14:31 +02:00
Bastian Kleineidam
e43694c156 Don't crash on multiple HTML output runs per day. 2014-09-22 22:00:16 +02:00
Bastian Kleineidam
e87f5993b8 Merge branch 'master' into htmlparser 2014-08-07 18:10:15 +02:00
Tobias Gruetzmacher
08175d28c9 Fix Ruthe (see #73). 2014-07-31 21:27:49 +02:00
Tobias Gruetzmacher
ca2d722d39 Fix DieFruehreifen (closes #73). 2014-07-31 21:18:15 +02:00
Tobias Gruetzmacher
6c7fb176b1 Add Blade Kitten as an example for the new parser. 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
f9f0b75d7c Create new HTML parser based scraper class. 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
fcde86e9c0 Change getPageContent to (optionally) return raw text. 
This allows LXML to do its own "magic" encoding detection
 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
0e03eca8f0 Move all regular expression operation into the new class. 
- Move fetchUrls, fetchUrl and fetchText.
- Move base URL handling.
 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
fde1fdced6 Fix some typos. 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
2567bd4e57 Convert starters and other helpers to new interface. 
This allows those starters to work with future scrapers.
 2014-07-26 11:28:43 +02:00
Tobias Gruetzmacher
4265053846 Refactor: Move regualar expression scraping into a new class. 
- This also makes "<base href>" handling an internal detail of the regular
  expression scraper, future scrapers might not need that or handle it in
  another way.
 2014-07-26 11:28:43 +02:00
Bastian Kleineidam
3a929ceea6 Allow comic text to be optional. Patch from TobiX 2014-07-24 20:49:57 +02:00
Bastian Kleineidam
950dd2932c Remove stray print statement. 2014-07-21 20:20:15 +02:00
Tobias Gruetzmacher
ea5d533e30 Fix index lookups for SnowFlame and SnowFlakes. 2014-07-19 13:23:42 +02:00
Bastian Kleineidam
4d49d4394b Fix doc 2014-07-03 18:42:06 +02:00
Bastian Kleineidam
f194e430bc TheThinHLine: fetch bigger images and name image files from sequence number. 2014-07-03 18:41:25 +02:00
Bastian Kleineidam
4845a4ccc1 Merge branch 'master' of github.com:wummel/dosage 2014-07-03 17:12:42 +02:00
Bastian Kleineidam
641daa738b Updated list of comics 2014-07-03 17:12:25 +02:00
Bastian Kleineidam
93fe5d5987 Minor useragent refactoring 2014-07-03 17:12:25 +02:00
Bastian Kleineidam
4c2a339e25 Fix some comics. 2014-07-02 19:51:53 +02:00
Luc Fouin
cb76198da7 added the thin H line, fixes #67 2014-07-02 17:14:33 +02:00
Luc Fouin
763f9b02a2 added the thin H line 2014-07-02 17:11:33 +02:00
Bastian Kleineidam
b03ba158ef Fixed LookingForGroup 2014-07-01 23:44:01 +02:00
Bastian Kleineidam
3485e2ac54 Added Whomp. 2014-06-24 20:48:49 +02:00
wummel
a0086bfcd8 Merge pull request #63 from sehrgut/master 
Updated GirlGenius to new markup
 2014-06-24 20:40:15 +02:00
Peter B
8f1c864ec3 Added Safely Endangered 2014-06-17 01:05:11 -04:00
Keith Beckman
236b840363 Updated GirlGenius to new markup 
GG markup has changed, so I fixed the prevSearch regex to find the
"previous" button on the redesigned page.

As well, I set multipleImagesPerStrip to true, since there are quite a
few comics with multiple images that were being discarded.
 2014-06-13 16:43:40 -04:00
Bastian Kleineidam
68afeaf82d Make appname lowercase. 2014-06-09 13:24:58 +02:00
Bastian Kleineidam
00e424aed0 Fix zenpencils. 2014-06-08 13:40:42 +02:00
Bastian Kleineidam
687d27d534 Stripping should be done in normaliseUrl. 2014-06-08 10:12:33 +02:00