Handle network errors when getting links lists

flowingblaze commented

2025-01-24 15:45:11 -05:00

(Migrated from github.com)

This is how I encountered the bug:

'd' to display the menu again
please enter your choice, or 'q' to quit:
l
do you want to start downloading from the page you stopped on last time? (y/n)
y
do you want to get works from all encountered series links? (bookmarked series will always be downloaded, regardless of this option) (y/n)
y
please enter page number to stop on. enter 0 to download all pages.
0
do you want to include work metadata? (y/n)
y
do you want to log in to ao3? (y/n)
y
logging in
list index out of range

I was downloading my ao3 history as links and was able to get links for 376 pages while getting an "Error encountered while getting links list. List may not be complete." after page 253. I have 567 pages of ao3 history

This is how I encountered the bug: ``` 'd' to display the menu again please enter your choice, or 'q' to quit: l do you want to start downloading from the page you stopped on last time? (y/n) y do you want to get works from all encountered series links? (bookmarked series will always be downloaded, regardless of this option) (y/n) y please enter page number to stop on. enter 0 to download all pages. 0 do you want to include work metadata? (y/n) y do you want to log in to ao3? (y/n) y logging in list index out of range ``` I was downloading my ao3 history as links and was able to get links for 376 pages while getting an "Error encountered while getting links list. List may not be complete." after page 253. I have 567 pages of ao3 history

nianeyna commented

2025-01-24 18:02:35 -05:00

(Migrated from github.com)

Hi there! Can you please check your log file (using menu option "v") and see if there are any error messages there?

flowingblaze commented

2025-01-24 19:14:36 -05:00

(Migrated from github.com)

Here is the information from the log file

timestamp	message	error	success	stacktrace	link
01/24/2025, 15:00:52	Error encountered while getting links list. List may not be complete.	HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30)	False	Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connection.py", line 464, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1378, in getresponse response.begin() File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\socket.py", line 706, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1278, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1134, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\adapters.py", line 589, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 843, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\util\retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\util\util.py", line 39, in reraise raise value File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 538, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 369, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 61, in get_work_links self.get_work_links_recursive(links_list, link, visited_series, metadata) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 96, in get_work_links_recursive self.get_work_links_recursive(links_list, url, visited_series, metadata, thesoup) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 82, in get_work_links_recursive series_soup = self.repo.get_soup(link) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\repo.py", line 44, in get_soup html = self.my_get(url).text ^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\repo.py", line 59, in my_get response = self.session.get(url, headers=self.headers, timeout=(30, 30)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\adapters.py", line 635, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30)
01/24/2025, 15:00:52		Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find'			https://archiveofourown.org/works/51383221
01/24/2025, 15:00:52		Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find'			https://archiveofourown.org/works/43271898
01/24/2025, 15:00:52		Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find'			https://archiveofourown.org/works/34316548

Here is the information from the log file timestamp | message | error | success | stacktrace | link | | -------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | 01/24/2025, 15:00:52 | Error encountered while getting links list. List may not be complete. | HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30) | False | Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connection.py", line 464, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1378, in getresponse response.begin() File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\socket.py", line 706, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1278, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1134, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\adapters.py", line 589, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 843, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\util\retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\util\util.py", line 39, in reraise raise value File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 538, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\urllib3\connectionpool.py", line 369, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 61, in get_work_links self.get_work_links_recursive(links_list, link, visited_series, metadata) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 96, in get_work_links_recursive self.get_work_links_recursive(links_list, url, visited_series, metadata, thesoup) File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\ao3.py", line 82, in get_work_links_recursive series_soup = self.repo.get_soup(link) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\repo.py", line 44, in get_soup html = self.my_get(url).text ^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\repo.py", line 59, in my_get response = self.session.get(url, headers=self.headers, timeout=(30, 30)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\venv\Lib\site-packages\requests\adapters.py", line 635, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='archiveofourown.org', port=443): Read timed out. (read timeout=30) | | | 01/24/2025, 15:00:52 | | Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find' | | | https://archiveofourown.org/works/51383221 | | 01/24/2025, 15:00:52 | | Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find' | | | https://archiveofourown.org/works/43271898 | | 01/24/2025, 15:00:52 | | Traceback (most recent call last): File "C:\Users\User\Downloads\ao3downloader-main\ao3downloader-main\ao3downloader\parse_soup.py", line 242, in get_work_metadata_from_list tags = blurb.find('ul', class_='tags') ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'find' | | | https://archiveofourown.org/works/34316548 |

nianeyna commented

2025-01-26 14:59:23 -05:00

(Migrated from github.com)

Thanks @flowingblaze . It looks like this is probably some poor error handling on my part. I'll see what I can do to make sure the program doesn't stumble when it hits a timeout like this. As usual, you can track this issue to know when I've pushed a fix.

SorceressHeart commented

2025-03-06 12:53:44 -05:00

(Migrated from github.com)

I'm getting the same timeout error when the downloader tries to lig in to ao3. I believe I posted about it here, but I don't see the issue opened. I'll stay tuned here. Thanks as always.

Rows
Columns

Handle network errors when getting links lists #192