Data Engineer

As a Data Engineer, you will be responsible for designing, building, and maintaining our data infrastructure, which includes streaming pipelines for applications and data lakes. You will collaborate with cross-functional teams, including business intelligence, analysts, product managers, and software engineers, to ensure the availability, reliability, and scalability of our data systems

Responsibilities

• Leading the development and implementation of streaming and batch data pipelines to extract, transform, and load data from various sources into our data lakes/data warehouse.

• Implement self-service tools in the data platform (orchestration tools and data quality tools)

• Implement a data validation process to improve data quality in data pipelines

• Monitor the data platform cluster (Kubernetes)

• Implement a data observability process to monitor data pipelines

• Implement data lineage to tracking data movement

• Implement data dictionary, relationship and its interpretation

• Developing and maintaining data processing and analysis frameworks

• Ensuring data security, privacy, and compliance with relevant regulations and policies

• Collaborating with data analysts to understand data requirements and perform database design based on data understanding and requirements with consideration for performance

• Manipulating complex data from different sources (MySQL, PostgreSQL, MongoDB, Excel, CSV, and Google Sheets)

• Monitoring data quality and troubleshooting data-related issues

• Identifying and implementing process improvements to enhance the efficiency and effectiveness of our data systems

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field

• 3-5 years of experience in data engineering or related field

• Experience with streaming and batch data processing, ETL, and data warehousing

• Experience in Cloud-based data platforms (Azure, AWS, GCP, or others)

• Experience in Git, Docker, BigQuery, Airflow

• Experience with databases (e.g., MySQL, PostgreSQL, MongoDB)

• Strong programming skills (SQL, Python)

• Strong understanding of coding standards and testing

• Experience in managing projects and leading teams

• Have a growth mindset and a willingness to learn new things or share knowledge with others

• Ability to self-learn

• Understand Kubernetes is a plus

• Understanding the data lineage concept is a plus

• Understand data observability concept is a plus

• Understand data mesh concept is a plus