Commit graph

32 commits

Author SHA1 Message Date
Tobias Gruetzmacher
0d8e1b4a6f Fix basepath prefix removal
This was a funny bug that has existed for 9 years. It only manifests if
a scraper name starts with the configured basepath... Since the default
basepath is `Comics`, this might manifest if the user wants to fetch
something like `ComicsKingdom/Tiger`. Internally, dosage was removing
the prefix matching the basepath, so we end up with `Kingdom/Tiger`.
This won't cause any problems as long as this substring is still unique
for all supported comics, but if this now matches multiple comics, an
error is thrown...
2022-06-05 23:55:18 +02:00
Tobias Gruetzmacher
99b72c90be Remove unused multi-match logic 2022-06-04 10:56:25 +02:00
Tobias Gruetzmacher
9b95171f37 Add some basic type annotations 2022-05-28 19:33:16 +02:00
Tobias Gruetzmacher
e64635e86b Stricter style checking & related style fixes 2020-10-11 20:15:27 +02:00
Tobias Gruetzmacher
9237bd62b2 Convert scraper cache to a class
This should make it easier to extend with additional entries.
2020-10-04 22:28:51 +02:00
Tobias Gruetzmacher
27d28b8eef Update file headers
The default encoding for source files is UTF-8 since Python 3, so we can
drop all encoding headers. While we are at it, just replace them with
SPDX headers.
2020-04-18 13:45:44 +02:00
Tobias Gruetzmacher
44791439a5 Drop Python 2 support: Obsolete future statements 2020-02-04 01:06:19 +01:00
Tobias Gruetzmacher
9c65c3e05f Drop Python 2 support: six & other imports 2020-02-03 01:03:31 +01:00
Techwolf
eb69210e58 Add self to authors list, update copyright headers 2020-01-28 00:23:15 +01:00
Tobias Gruetzmacher
6e14e8709b Reduce allowed line length 2020-01-13 00:36:46 +01:00
Techwolf
c44a5a3b43 Mark endOfLife comics as completed when checking for updates 2020-01-12 21:29:47 +01:00
Tobias Gruetzmacher
2afc0cb645 Director: Always cleanup jobs & threads
This shouldn't make a difference in normal dosage usage (since dosage
exits right after this anyways), but it removes hidden dependencies
between tests and might help when embedding dosage into other Python
software. As a side effect this gets rid of some global state, so
running dosage multiple times in the same process is more reliable...
2020-01-04 13:39:14 +01:00
Tobias Gruetzmacher
f5a5106381 Allow combining -n with -c or -a (related to #90)
This allows fetching "all" comics (or catch up until the last existing
one) while setting an upper bound on how many pages to fetch at the same
time.
2019-12-01 23:19:06 +01:00
Tobias Gruetzmacher
ac2ca54570 Remove handlers after director run. 2017-10-15 14:54:44 +02:00
Tobias Gruetzmacher
9a6a310b76 Fixup copyright years. 2016-10-29 00:21:41 +02:00
Tobias Gruetzmacher
64c8e502ca Ignore case for comic download directories.
Since we already match comics case-insensitive on the command line, this
was a logical step, even if this means changing quite a bit of code that
all tries to resolve the "comic directory" in a slightly different
way...
2016-06-06 00:08:29 +02:00
Tobias Gruetzmacher
125c96e9dc Remove command to download ALL comics... 2016-06-05 21:57:56 +02:00
Tobias Gruetzmacher
df2048cb34 Keep track of removed and moved comics (fixes #41).
I plan on keeping this list for at least ~ 2 releases and then purging
older entries...
2016-06-05 21:47:58 +02:00
Tobias Gruetzmacher
efe1308db2 Replace home-grown Python2/3 compat. with six. 2016-05-05 23:33:48 +02:00
Tobias Gruetzmacher
b266e28ae1 Remove debugging prints 😭 2016-04-13 22:59:06 +02:00
Tobias Gruetzmacher
ff3b824311 Fix variable shadowing... 2016-04-13 22:43:34 +02:00
Tobias Gruetzmacher
060281e5ff Use concrete scraper objects everywhere.
This is a first step for #42. Since most access to the scraper classes
is through instances, modules can now dynamically override url and name
(name is now a property).
2016-04-13 22:17:30 +02:00
Damjan Košir
4529fdee3b adding no downsize option 2015-05-20 22:38:29 +12:00
Damjan Košir
561005887a unneeded max 2015-04-26 00:23:45 +12:00
Damjan Košir
ac7b0d7e0e adding parallel run option 2015-04-26 00:19:08 +12:00
Tobias Gruetzmacher
5934f03453 Merge branch 'htmlparser' - I think it's ready.
This closes pull request #70.
2015-04-01 22:13:55 +02:00
Dirk Reiners
77a5e09c10 Minor fix for using pathes to pick comics 2014-10-24 16:39:40 -05:00
Tobias Gruetzmacher
e92a3fb3a1 New feature: Comic modules ca be "disabled".
This is modeled parallel to the "adult" feature, except the user can't
override it via the command line. Each comic module can override the
classmethod getDisabledReasons and give the user a reason why this
module is disabled. The user can see the reason in the comic list (-l or
--singlelist) and the comic module refuses to run, showing the same
message.

This is currently used to disable modules that use the _ParserScraper if
the LXML python module is missing.
2014-10-13 21:43:46 +02:00
Tobias Gruetzmacher
d495d95ee0 Refactor: Move repeated check into its own function. 2014-10-13 21:29:54 +02:00
Bastian Kleineidam
348dd5e6c0 Add documentation 2014-03-04 20:53:19 +01:00
Bastian Kleineidam
3108c9124a Fix thread import for py3 2014-03-04 20:50:34 +01:00
Bastian Kleineidam
15ef59262a Make threads interruptable. 2014-03-04 18:38:46 +01:00