|
|
a9fcda794c
|
use a new db connection for each scrape thread
|
2019-09-13 12:11:01 +10:00 |
|
|
|
af1f8c6d8b
|
warn about rate limit
|
2019-09-13 12:03:59 +10:00 |
|
|
|
f982e54a2d
|
fixed some more issues that didn't happen before
|
2019-09-11 18:16:36 +10:00 |
|
|
|
a981c7a171
|
print a 'finished' message when done scraping a given user's posts
|
2019-09-11 15:47:48 +10:00 |
|
|
|
de0505004d
|
oops
|
2019-09-11 15:43:44 +10:00 |
|
|
|
ac29216973
|
make scrape.py commit more frequencly
|
2019-09-11 15:28:50 +10:00 |
|
|
|
cfa32589d4
|
handle case where we haven't scraped any posts yet i guess
|
2019-09-11 15:26:28 +10:00 |
|
|
|
bb7c50a648
|
commit to db when done
|
2019-09-11 15:23:00 +10:00 |
|
|
|
38463fefb2
|
enable db unicode support
|
2019-09-10 22:21:22 +10:00 |
|
|
|
bc6745ae02
|
use service_threads instead of hardcoding 8
|
2019-09-10 14:22:31 +10:00 |
|
|
|
f2d15b7d80
|
moved scraping functions to a separate file
|
2019-09-10 12:07:53 +10:00 |
|
