TCG & Web Scraping, pt2: Pop Reports
"Automating population report tracking across international grading services."
A selection of PSA graded cards.
0. What the hell is a graded card?
If you’ve spent any time in the TCG world lately, you’ve seen them: cards encased in thick, sonically-welded plastic slabs with a numeric grade at the top. To the uninitiated, it looks like someone took a perfectly good game piece and turned it into a museum artifact. But in this market, that plastic "slab" is everything.
Grading is the process of sending a card to a third-party company—like PSA, BGS, or CGC—where an expert examines it under a microscope to judge its condition on a scale from 1 to 10. A "Gem Mint 10" Charizard can be worth ten times more than a "Near Mint 9," even if the human eye can't tell the difference.
This leads us to the Population Report (or "Pop Report"). This is the official census of how many copies of a specific card exist in each grade. If a card has a "Low Pop," it’s scarce, and the price skydives upward. But here’s the kicker: these numbers change every single day as more cards get graded. Tracking that evolution across three or four different international grading companies by hand? That's not just tedious—it's impossible. That's exactly where the script comes in.
1. The Genesis
Following the ongoing collaboration with Strictly Sealed, he asked me if I could generate Excel files that would contain the population reports for specific cards, initially from PSA and later ARS. Ideally, these reports would allow users to see live updates in population counts by grade.
2. The First Data Source: PSA
PSA is the largest and most well-known grading service. Tracking their trends says a lot about the current market. If the market is growing, we see a global population increase. Knowing the rarity of a card at a glance—and the specific ratio of PSA 9s to 10s—gives you a great idea of how difficult a card is to "Gem" (achieve a 10).
The first step was setting up the Excel sheet with the "interesting" cards. Since I'm lazy, I decided to automate this for the main sets. For early Yu-Gi-Oh!, "interesting" is easy to define: any card rarer than a base rarity (Super, Ultra, Ultimate, or Ghost Rare). I wrote a script to scrape Yugipedia to gather card names and set lists, ordering them all into a master spreadsheet.
The Yugipedia lists scraped for each relevant set.
With the foundation built, I moved on to the PSA website. My strategy was as follows: open the latest report, store all existing card data (names, sets, previous grade counts) in memory, and then crawl the PSA site for current numbers. Finally, the script backs up the previous file and edits the Excel sheet with the new data. If a population count increases, the text turns green, and the previous count is shown in parentheses for context.
PSA's official population report for a specific card set.
The main hurdle was PSA's anti-bot measures. I had to implement some proprietary logic to bypass Cloudflare protections. Additionally, mapping the scraped names to the Excel list was painful; PSA's wording is inconsistent, often shifting between different labels for rarities or 1st Edition specs. Refining this took time to ensure the process was robust enough to handle these naming quirks.
3. ARS Grading
While BGS or CGC are popular, ARS Grading is highly regarded within the Yu-Gi-Oh! community for its reliability and premium aesthetic. However, ARS is based in Japan, and their reports are entirely untranslated. Searching for specific cards requires digging through Japanese names, which is a significant barrier for Western collectors.
Unlike PSA, ARS doesn't store entire sets on one page; they use "single card" pages. To solve this, I returned to Yugipedia to collect the Japanese names for our target sets. I then built a utility script to navigate the ARS search feature, identify the correct card pages, and map the URLs to our database.
Navigating the Japanese-only ARS Grading search interface.
Once the URLs were harvested, I reused my core parsing functions to open, scrape, and update the master Excel. The result was a game-changer: English-speaking collectors could finally track the rarity of ARS-graded cards with a single click, providing a much clearer picture of the global supply for high-end TCG items.
4. Conclusion
Building these tools was an exercise in turning raw, unorganized data into a competitive advantage. Between bypassing anti-bot measures, handling multi-language translation, and cleaning up inconsistent naming conventions, the project was a technical puzzle from start to finish. But the result, moving from a world of manual "refreshing" to automated "reporting" is exactly why I love web scraping.
5. A Word from Strictly Sealed
"Exqd1a did an unbelievable job optimizing the scraping/integration of the pop report data. But the more impressive work was the accuracy and potency of the sales data accumulation. He was not only able to create a protocol that rips sales data that is accurate and timely, he was able to transcend that by including item categories and products that I personally had never thought to include in the sales data accumulation in the first place. Amazing work."
