Junior Software Engineer

Role Summary:
We are seeking a highly experienced (3+ years) and independent Senior Data Engineer to take a leading role in enhancing the quality, reliability, and security of our primarily self-hosted data infrastructure and codebase. This role involves significant ownership of refactoring complex Python code and Playwright scrapers, optimizing databases (PostgreSQL/ClickHouse), managing containerized environments (Docker), setting up testing infrastructure, developing and managing data workflows (Prefect), implementing data transformations (dbt, SQLMesh), and contributing to the overall robustness and security of our systems. The ideal candidate thrives on autonomy, excels in managing self-hosted services, tackles complex technical challenges proactively, and possesses a deep understanding of modern data engineering, development, and operations practices.

Key Responsibilities:
- Code Refactoring & Development: Lead efforts in analyzing, refactoring, and optimizing existing Python scripts/applications and Playwright web scrapers for improved performance, scalability, maintainability, and adherence to best practices.
Design, develop, test, and deploy robust automation scripts and tools.
- Infrastructure & Environment Management: Autonomously design, set up, configure, maintain, and secure self-hosted development and testing environments, including database setup (with dummy data) and dependency management.
Expert-level use of containerization technologies (Docker) for managing and deploying applications, particularly in a self-hosted context.
- Perform system maintenance, updates, and dependency management for Windows and Linux virtual machines, ensuring high availability and performance.
- Manage, optimize, and ensure the operational health of self-hosted containerized platforms (e.g., for data visualization/reporting).
- Data Platform, Pipeline & Database Management:
- Design, implement, manage, and optimize data pipelines and workflows using Prefect.
- Develop, test, and maintain data transformation models using dbt and/or SQLMesh.
- Oversee the monitoring, testing, and troubleshooting of data pipelines and replication processes.
- Lead database administration tasks for PostgreSQL and ClickHouse, including advanced performance tuning, indexing strategies, capacity planning, and maintenance.
- Develop and implement sophisticated data quality scripts and data observability solutions within our stack.
- Manage the deployment lifecycle for data models and data marts generated via dbt/SQLMesh.
- Security & Compliance: Implement and advocate for security best practices across all development and infrastructure tasks within our self-hosted environment.

Required Skills & Experience:
- 3+ years of proven experience in relevant roles, with demonstrable progression in responsibilities and significant experience managing self-hosted systems.
- Must have github (share access to github account)
- Expert proficiency in Python development...
- Significant experience with web scraping frameworks, specifically Playwright...
- Deep expertise with Docker... including securing and managing containers in self-hosted environments.
- Proven ability to independently design, build, manage, and secure development/testing environments from the ground up.
- Advanced skills in database administration and optimization for both PostgreSQL and ClickHouse...
- Solid experience with Linux and Windows system administration in production, self-hosted environments.
- Hands-on experience with workflow orchestration tools, specifically Prefect.
- Proficiency with data transformation tools, specifically dbt and/or SQLMesh.
- Strong understanding and practical experience with data warehousing, data modeling, and ETL/ELT processes.
- Experience with cloud platforms.
- Strong theoretical and practical understanding of infrastructure security, PKI, TLS/SSL certificate lifecycle management, especially relevant to securing self-hosted applications and networks.
- Familiarity with security tools like Hashicorp Vault or similar secrets management.
- Expertise with version control systems (e.g., Git) and branching strategies.
- Exceptional ability to work independently, define technical direction, troubleshoot highly complex issues in potentially complex, self-hosted environments...
- Excellent analytical and problem-solving skills.