Senior Data Engineer (GCP, PySpark, Dataproc)

We are looking for a Senior Data Engineer with strong hands-on experience in Apache Spark, PySpark and GCP Dataproc to join Implex and work on a long-term data platform project for a public-sector in the Middle East.

The engineer will contribute to the development and support of data-processing pipelines that integrate multiple enterprise data sources, including PostgreSQL, SAP HANA and S3-compatible object storage. The role involves not only writing PySpark jobs but also troubleshooting distributed workloads, configuring Dataproc environments, securing external connections and ensuring reliable delivery of data into the target data warehouse.

Responsibilities

• Develop, maintain, and optimize ETL pipelines using Apache Spark and PySpark.
• Configure, run, and troubleshoot Spark workloads on GCP Dataproc.
• Package and submit Spark jobs, manage dependencies, and analyze driver and executor logs.
• Identify and resolve performance issues related to memory usage, shuffling, partitioning, data skew, and distributed data processing.
• Integrate Spark workloads with S3-compatible object storage using the Hadoop S3A connector.
• Configure and troubleshoot Spark connectivity with MinIO, including bucket policies, custom endpoints, path-style access, TLS, signature compatibility, and redirect handling.
• Configure custom CA certificates and Java truststores for Spark drivers and executors.
• Implement and optimize scalable Spark JDBC reads and writes for PostgreSQL and SAP HANA, including partitioning, batching, pushdown, and data type mapping.
• Build reliable incremental data loads, retries, backfills, idempotent processing, and data reconciliation mechanisms.
• Design and support staging-to-publish data flows, bulk data loads, and data warehouse integration patterns.
• Contribute to data modeling, schema evolution, Slowly Changing Dimension patterns, data lineage, technical documentation, and data quality practices.
• Apply secure secrets management and least-privilege access principles across data integrations.

Must-Have Technical Skills

• Apache Spark / PySpark development (Dataproc): driver/executor behavior, job packaging/submission, performance tuning
• GCP Dataproc operations: cluster configuration, init actions, dependency management, troubleshooting via logs/metrics
• Hadoop S3A connector: `fs.s3a.*` configuration, endpoint/path-style access, credential providers, S3 semantics
• MinIO (S3-compatible) integration: bucket policies, TLS endpoints, signature/redirect troubleshooting
• TLS/SSL & PKI with custom CA: certificate chains, SAN/hostname validation, diagnosing handshake/PKIX errors
• Java truststores (JKS/PKCS12) & JVM SSL config: `keytool`, distributing truststores, setting driver/executor JVM options
• PostgreSQL integration: Spark JDBC reads/writes at scale, indexing/performance basics, data type mapping
• SAP HANA integration: JDBC/ODBC connectivity, driver management, calculation views vs tables, pushdown/performance tuning
• ETL engineering: incremental loads/CDC concepts, idempotency, retries, backfills, data quality/reconciliation
• Data Warehousing integration: strong SQL, staging-to-publish patterns, SCD concepts, bulk load strategies
• Data modeling & governance basics: dimensional modeling, schema evolution, lineage/documentation practices

Nice-to-Have Skills (but not must)

• Linux + networking fundamentals: DNS, routing, firewall/LB/proxy basics; tools like `curl`/`openssl s_client` for validation
• Secure secrets handling: GCP Secret Manager (or equivalent), least-privilege access, avoiding hardcoded credentials
• KAFKA knowledge if we ever bring KAFKA into the architecture again

Hiring process

• HR interview
• PM/Technical interview with Implex
• Dev Lead interview on the client side

What we offer

• A long-term international project
• Opportunity to work on a national-scale digital platform used by thousands of users
• Remote full-time collaboration
• Professional and supportive team environment
• Challenging technical tasks and a meaningful product with real-world impact