Add persistent "seen" ID tracking to skip already-downloaded AO3 works

8ullyMaguire commented

2025-10-14 09:58:00 -04:00

(Migrated from github.com)

When I download works from my AO3 History, "Marked for later", or any other search, ao3downloader saves them to a folder. I then move the files into Calibre and delete them from that folder because Calibre keeps its own copy and I don’t want duplicates. However, ao3downloader detects already-downloaded works by checking the files that remain in that folder. Since I remove the files after importing to Calibre, ao3downloader repeatedly re-downloads the same works.

I'm thinking of adding the “seen” functionality used in lemmy-rss-pybot (which tracks seen item IDs) to ao3downloader so it records downloaded AO3 work IDs separately (e.g., in a local database or file) and skips re-downloading those IDs even if the original files are no longer present.

When I download works from my AO3 History, "Marked for later", or any other search, ao3downloader saves them to a folder. I then move the files into Calibre and delete them from that folder because Calibre keeps its own copy and I don’t want duplicates. However, ao3downloader detects already-downloaded works by checking the files that remain in that folder. Since I remove the files after importing to Calibre, ao3downloader repeatedly re-downloads the same works. I'm thinking of adding the “seen” functionality used in [lemmy-rss-pybot](github.com/8ullyMaguire/lemmy-rss-pybot) (which tracks seen item IDs) to ao3downloader so it records downloaded AO3 work IDs separately (e.g., in a local database or file) and skips re-downloading those IDs even if the original files are no longer present.

mjharwood commented

2025-10-14 13:03:01 -04:00

(Migrated from github.com)

I've been semi-manually curating an ignore file to manage it, so this would be helpful.

It would need some way to allow redownload if the fic updated though, which ignore does not. Text files make it easier to review than a database - sqlite would work fine but takes additional skills to check the data. CSV would allow view and edit with a text editor or spreadsheet.

I've been semi-manually curating an ignore file to manage it, so this would be helpful. It would need some way to allow redownload if the fic updated though, which ignore does not. Text files make it easier to review than a database - sqlite would work fine but takes additional skills to check the data. CSV would allow view and edit with a text editor or spreadsheet.

8ullyMaguire commented

2025-10-31 13:42:49 -04:00

(Migrated from github.com)

I wrote a Python script as a workaround since I couldn't find where I should make the changes.

How to use it

cd to /path/to/ao3_archive
run ao3downloader
run python import_new_fics.py
import files in latest folder to calibre
remove files in latest folder

import_new_fics.py

import os
import shutil

# Base folder (change this if you move the folder)
BASE_DIR = "/path/to/ao3_archive"

# Paths
downloader_dir = os.path.join(BASE_DIR, "downloads")
import_dir = os.path.join(BASE_DIR, "latest")
seen_ids_file = os.path.join(BASE_DIR, "seen_ids.txt")

# Ensure import directory exists
os.makedirs(import_dir, exist_ok=True)

# Load imported IDs
if os.path.exists(seen_ids_file):
    with open(seen_ids_file, "r", encoding="utf-8") as f:
        seen_ids = set(line.strip() for line in f if line.strip())
else:
    seen_ids = set()

# Process files
new_ids = []
for filename in os.listdir(downloader_dir):
    if not filename.lower().endswith(".epub"):
        continue
    fic_id = filename.split(" ", 1)[0]
    if fic_id not in seen_ids:
        src = os.path.join(downloader_dir, filename)
        dst = os.path.join(import_dir, filename)
        shutil.copy2(src, dst)
        new_ids.append(fic_id)

# Update IDs file
if new_ids:
    with open(seen_ids, "a", encoding="utf-8") as f:
        for fic_id in new_ids:
            f.write(fic_id + "\n")

print(f"{len(new_ids)} new files.")

I wrote a Python script as a workaround since I couldn't find where I should make the changes. ### How to use it cd to `/path/to/ao3_archive` run ao3downloader run `python import_new_fics.py` import files in latest folder to calibre remove files in latest folder `import_new_fics.py` ```python import os import shutil # Base folder (change this if you move the folder) BASE_DIR = "/path/to/ao3_archive" # Paths downloader_dir = os.path.join(BASE_DIR, "downloads") import_dir = os.path.join(BASE_DIR, "latest") seen_ids_file = os.path.join(BASE_DIR, "seen_ids.txt") # Ensure import directory exists os.makedirs(import_dir, exist_ok=True) # Load imported IDs if os.path.exists(seen_ids_file): with open(seen_ids_file, "r", encoding="utf-8") as f: seen_ids = set(line.strip() for line in f if line.strip()) else: seen_ids = set() # Process files new_ids = [] for filename in os.listdir(downloader_dir): if not filename.lower().endswith(".epub"): continue fic_id = filename.split(" ", 1)[0] if fic_id not in seen_ids: src = os.path.join(downloader_dir, filename) dst = os.path.join(import_dir, filename) shutil.copy2(src, dst) new_ids.append(fic_id) # Update IDs file if new_ids: with open(seen_ids, "a", encoding="utf-8") as f: for fic_id in new_ids: f.write(fic_id + "\n") print(f"{len(new_ids)} new files.") ```

Rows
Columns

Add persistent "seen" ID tracking to skip already-downloaded AO3 works #265

How to use it