# Automation & CI/CD

> Selected Engineer ApS work demonstrating Automation & CI/CD — the achievements that prove it.

- [Engineered hourly electricity consumption aggregation pipeline in Python / SQL / Bash + Jq, achieving 180ms for 30‑day datasets across heterogeneous JSONL sources.](https://data.engineer.company/portfolio/engineered-hourly-electricity-consumption-aggregation-pipeline-in-python-4/)
- [Resolved 1,000 issues in geographical data and time‑series data, using GDAL, ArcGIS, PostGIS, Mapbox, QGIS, SQL (PL/pgSQL, Transact‑SQL), Bash, ensuring high‑quality big data processing.](https://data.engineer.company/portfolio/resolved-1-000-issues-in-geographical-data-and-7/)
- [Designed, implemented, and administered 6 ETL/ELT pipelines, utilizing Google BigQuery, MSSQL, PostgreSQL, Shell scripting, PL/pgSQL, and Transact‑SQL, integrating data for efficient Python API processing.](https://data.engineer.company/portfolio/designed-implemented-and-administered-6-etl-elt-pipelines-8/)
- [Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.](https://data.engineer.company/portfolio/developed-and-launched-the-company-s-first-observability-9/)
- [Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.](https://data.engineer.company/portfolio/designed-deployed-and-maintained-10-postgresql-and-ms-13/)
- [Enhanced data security by implementing 1,000 RBAC rules for developers, application instances, PostgreSQL, MS SQL, and other Linux servers, preventing unauthorized access; documented with Ansible automation.](https://data.engineer.company/portfolio/enhanced-data-security-by-implementing-1-000-rbac-14/)
- [Automated GIS SaaS application deployment, data processing, and reporting system using GitHub Actions CI/CD, Python, Bash, and SQL.](https://data.engineer.company/portfolio/automated-gis-saas-application-deployment-data-processing-and-16/)
- [Automated delivery of 20 GIS data pipelines and app data ETL processes, streamlining infrastructure automation and reporting.](https://data.engineer.company/portfolio/automated-delivery-of-20-gis-data-pipelines-and-17/)
- [Automated 100 critical data backups using Barman, Google Cloud, Bash, and Python, ensuring data integrity across databases.](https://data.engineer.company/portfolio/automated-100-critical-data-backups-using-barman-google-18/)
- [Streamlined data analysis and software development processes, saving 4,000 hours by introducing GitHub, GitLab, Bash, and Python CI/CD practices.](https://data.engineer.company/portfolio/streamlined-data-analysis-and-software-development-processes-saving-43/)
- [Developed a Data Analytics reporting system, increasing quarterly software revenue by 400% through Python‑based PDF reports.](https://data.engineer.company/portfolio/developed-a-data-analytics-reporting-system-increasing-quarterly-44/)
- [Automated data processing tasks using Shell scripting, PL/pgSQL, Python, and Transact‑SQL, increasing productivity and efficiency.](https://data.engineer.company/portfolio/automated-data-processing-tasks-using-shell-scripting-pl-45/)
- [Engineered 600 PL/pgSQL‑based ETL/ELT pipelines to streamline complex data processing workflows across multiple PostgreSQL development and production environments.](https://data.engineer.company/portfolio/engineered-600-pl-pgsql-based-etl-elt-pipelines-54/)
- [Built the Python vessel‑data scrapers (MarineTraffic, Maritime‑Database) and a repeatable import that seeds the platform's reference data — 184,197 rows, including 698 companies and 56,149 vessels.](https://data.engineer.company/portfolio/built-a-python-vessel-data-scraper-marinetraffic-maritime-66/)
- [Designed a time‑management and reporting system that remained in production use for years without significant modification.](https://data.engineer.company/portfolio/designed-a-time-management-and-reporting-system-that-88/)
- [Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.](https://data.engineer.company/portfolio/moved-slow-work-onto-a-river-job-queue-97/)
- [Wrote a scope rule into the repository after a restructure carried another company's inventory, firewall allowances and prose into it — and kept the quarantined residue under the secret scanner rather than excluding it.](https://data.engineer.company/portfolio/wrote-a-scope-rule-after-a-credential-leak-124/)
- [Replaced 963 anonymous structured‑data blocks that restated the company 1,671 times with one linked graph of 16 types minted from stable origin identifiers.](https://data.engineer.company/portfolio/replaced-963-structured-data-blocks-with-one-graph-138/)
- [Built a database‑driven CV, references, portfolio and cover‑letter generator in Python — 41 modules, 10,580 lines — rendering six output formats from one 23‑table SQLite source assembled by a 19‑step idempotent pipeline.](https://data.engineer.company/portfolio/built-a-database-driven-cv-and-portfolio-generator-144/)
- [Wrote tests for the checkers themselves after establishing that a checker fed only clean input will one day report clean because it read nothing — planting a misspelling to confirm the spell‑check finds it, and taking an id range from the database rather than from a number in the test.](https://data.engineer.company/portfolio/wrote-tests-for-the-checkers-themselves-149/)
- [Built a cross‑language content check that fails when a translation drops a figure the English states, and when a language uses notation it does not use — finding two Danish descriptions missing a metric and sixteen Ukrainian spans quoting in the English style.](https://data.engineer.company/portfolio/built-a-cross-language-figure-and-notation-check-153/)
- [Audited 644 Rust crates for licence compatibility on every build, and proved the check fires by rewriting one crate's licence and by moving the pinned core revision without regenerating.](https://data.engineer.company/portfolio/audited-644-rust-crates-for-licence-compatibility-163/)

<https://data.engineer.company/categories/automation/>
