12 commits

Author	SHA1	Message	Date
Lynne	5d35519818	fixed indentation issue, hopefully #21 is fixed now	2019-09-17 18:12:32 +10:00
Lynne	a9fcda794c	use a new db connection for each scrape thread	2019-09-13 12:11:01 +10:00
Lynne	af1f8c6d8b	warn about rate limit	2019-09-13 12:03:59 +10:00
Lynne	f982e54a2d	fixed some more issues that didn't happen before	2019-09-11 18:16:36 +10:00
Lynne	a981c7a171	print a 'finished' message when done scraping a given user's posts	2019-09-11 15:47:48 +10:00
Lynne	de0505004d	oops	2019-09-11 15:43:44 +10:00
Lynne	ac29216973	make scrape.py commit more frequencly	2019-09-11 15:28:50 +10:00
Lynne	cfa32589d4	handle case where we haven't scraped any posts yet i guess	2019-09-11 15:26:28 +10:00
Lynne	bb7c50a648	commit to db when done	2019-09-11 15:23:00 +10:00
Lynne	38463fefb2	enable db unicode support	2019-09-10 22:21:22 +10:00
Lynne	bc6745ae02	use service_threads instead of hardcoding 8	2019-09-10 14:22:31 +10:00
Lynne	f2d15b7d80	moved scraping functions to a separate file	2019-09-10 12:07:53 +10:00