IP bans, dev/shm crashes, and teaching Quivya to throw punches

Three threads this cycle, and only two of them were planned. The Yu-Gi-Oh price/pop scraping pipeline (the guts behind the TCG Data Platform and PSA-to-eBay projects) finally left my laptop for a real server; that server's site then grew a public page fed by a Discord bot that has no idea it's a website backend; and Quivya got its first taste of actual PvP.

The scrapers meet a real server, and the server meets a ban hammer

The whole pipeline — card_db, recent_sales, ars_scrap, pop_reports, psa_vault — moved onto a Hetzner box, orchestrated with plain systemd units, Postgres running as a podman quadlet, and Caddy in front of the FastAPI site. I wrote a deploy/bootstrap.sh that does the entire setup end to end: system deps, matched Chrome/chromedriver, both repos cloned side by side, migrations, a one-shot referential seed, units enabled. Idempotent and fail-fast, so a botched run leaves nothing to clean up by hand.

Then, within days, eBay and 130point.com both started serving quiet blocks that never happened from home. Same script, same code, different fate depending on whose IP asked. Confirmed it wasn't a fluke by running the identical scrape from both places back to back. The fix ended up two-pronged: a run_scrapers.py orchestrator to scrape locally and sync results up to the server, and per-site proxy routing so the ones that must run server-side don't share a fingerprint.

# recent_sales/.env
EBAY_PROXY_URL=http://residential-proxy:8080
PTS130_PROXY_URL=http://residential-proxy:8081
# separate proxies per site so a burned cookie jar
# doesn't take both sessions down with it

Underneath that, headless Chrome inside the container was crashing on /dev/shm, fixed by disabling image loading and giving every retry a fresh incognito driver instead of a shared profile — I tried the shared-profile "optimization" first, made things worse, and reverted it the same day. On the product side: manager/editor/viewer roles now gate the quarantine and ingest queues, the pop-report diff view got unified across sources, sales pages are paginated, and results can go out via share links instead of handing out server credentials.

The Discord bot writes, the website reads

The community runs a photo contest on Discord — members post a picture on a theme, native polls settle the head-to-head matches, somebody gets crowned — and the site had a hall of fame of exactly nobody. Joining the two sounds like a nightly job: poll for past winners, render them. It can't be. Discord attachment URLs are signed and expire within about a day, and the bot's reset_tournament() deletes the submission row outright. By the time anything could ask who won last month, there is neither an image nor a theme left to answer with. So publishing happens at crowning time, the one moment the data still exists: the bytes are copied into a shared folder and only the filename goes into Postgres.

# the bot's .env and the site's .env, same box
CONTEST_MEDIA_DIR=/srv/contest_media
# the row stores "contest-7-937478581880225792.jpg",
# never a discordapp.com URL that expires by tomorrow

Two services, one host, one directory. The bot's database role has INSERT and SELECT on a single table and nothing else; every query on the site's side stays SELECT-only. And nothing in the publish path is allowed to break a contest — asyncpg is an optional import, an unset DATABASE_URL logs and skips, and every failure returns instead of raising. Postgres being down costs a website row, never a champion. The insert is ON CONFLICT DO NOTHING rather than an upsert, because crowning gets retried if the announcement crashes partway and the first row written for a contest is the correct one.

The winner's display name, avatar and win count are all copied into the row instead of being looked up later: people rename themselves and reskin their avatars, and a hall of fame entry should keep saying who won it as they were known then. The avatar is downloaded and served off the box rather than hotlinked, too — /hall-of-fame is public, so a cdn.discordapp.com <img> would hand Discord the IP address of every anonymous visitor.

That public-ness is the whole point. The hall of fame is the one page open to everyone — Instagram-style cards with hearts, comments, a search/sort bar and the deciding match's score — and it's the shop window for the Patreon subscription that pays for the scraping and the hosting, while the sales and pop-report views stay behind it. Which needed a new tier between "anonymous" and "paying": a logged-in server member can heart and comment, and wandering into a data view gets them a paywall page instead of a login redirect, because logging in again was never going to be the fix.

One wrinkle: the first three tournaments had never crowned anyone. A wildcard rule kept handing the runner-up straight back into an already-two-person final, so the final replayed until an admin reset it — 40 participations recorded, zero wins. Fixed in the bot, then a one-off script rebuilt those three results from the finalised poll counts and re-downloaded the photos from their original messages. It runs under the database owner's credentials rather than the bot's, since refreshing an existing row needs UPDATE and the bot has no business carrying that permission the rest of the time.

Quivya learns to throw down

Quivya shipped to the App Store, then Android's EAS build promptly refused to cooperate until I bumped the target SDK to 35 and patched expo-modules-core to match. Two full security/correctness audit passes followed — 26 issues, then 22 more on the second sweep — before either store build felt trustworthy enough to call 1.0.2, then 1.0.3.

The actual feature people asked for: 1v1 friend challenges — a direct invite with a push notification to join, a global banner so a challenge doesn't get buried under a Friends tab nobody checks, and online/in-game status so you know who's actually reachable. New quiz categories landed too, mostly because testers kept asking for them: Naruto, Hunter x Hunter, Jujutsu Kaisen, meme-origin trivia, and — very on brand for a French side project — Préfectures de France.

"The server doesn't get IP-banned for running your code. It gets IP-banned for running your code at scale, predictably, forever — which, in hindsight, is a pretty accurate description of what a server is for."

All three are live and getting used, which is the only metric that matters. Check the Projects page for the write-ups on the TCG Data Platform, the PSA-to-eBay sync, and Quivya itself, or hit the Contact page if any of this sounds like a problem you're currently having.