Built the Python vessel‑data scrapers (MarineTraffic, Maritime‑Database) and a repeatable import that seeds the platform's reference data — 184,197 rows, including 698 companies and 56,149 vessels.
Work carried out: 2025
Situation. A maritime networking‑and‑reviews site is dead on arrival if it’s empty. Nobody joins a directory with no companies in it. So before any of the social features mattered, the platform needed a real body of maritime companies and vessels already sitting there, ready to be found.
Task. Go and get that data — real companies and ships, at enough scale to feel populated — and get it into the database in a way that could be rerun, not a one‑off scrape nobody could reproduce.
Action. The scrapers are written in Python. One drives MarineTraffic with Playwright; another pulls from Maritime‑Database over async httpx; there’s a ClassNK fetcher in there too. They write out CSVs, and an import step cleans and normalises those and loads them into the Postgres schema through a single task, so seeding the database is one command rather than an afternoon of manual work. What went in came to 184,197 rows: 698 companies, 56,149 vessels and 33,074 cities, with a later refresh replacing 74,794 vessel rows.
Result. The platform launched with a populated directory instead of empty tables, and a base of reference data the networking, jobs and review features could all build on top of. Because the pipeline is repeatable, refreshing or extending it later is just running it again — which is how the vessel refresh happened without anyone rebuilding the tooling. Scraped data ages, and keeping it current is an ongoing cost rather than a solved problem.