Our expectations

Tech stack:

• Apache Spark on Azure Databricks

• Apache Kafka

• Delta Lake

• Java

• MS SQL Server and NoSQL storages like Elastic Search, Redis, Azure Data Explorer

• Docker containers

• Azure DevOps and fully automated CI/CD pipelines with Databricks Asset Bundles, ArgoCD, GitOps, Helm charts

• Automated tests 

• Proficient in programming, not just data processing, we are not considering candidates with an ETL profile.

Personal requirements:

• Have 4+ years of Apache Spark experience and have faced various data engineering challenges in batch or streaming

• Have an interest in stream processing with Apache Spark Structured Streaming on top of Apache Kafka

• Have experience leading technical solution designs

• Have experience with distributed systems on a cloud platform

• Have experience with large-scale systems in a microservice architecture

• Are familiar with Git and CI/CD practices and can design or implement the deployment process for your data pipelines

• Possess a proactive approach and can-do attitude

• Are excellent in English and Polish, both written and spoken

• Have a higher education in computer science or a related field

• Are a team player with strong communication skills