Hiring: Specialist Data Engineer Company: Metropolitan Transportation Authority Location: New York, NY Job Posted Time: 2026-09-16 14:17:02 Target Skills & Keywords : AWS, Agile, Airflow, Ansible, Azure, CI/CD, Confluence, Data Lake, Data Pipeline, Data Warehouse, Databricks, ELT, ETL, GCP, Git, IAM, Integration Testing, Jira, Kanban, PySpark, Python, SQL, SQL Server, Scrum, Terraform, Unit Testing About the job Experience: •3 years of relevant experience. •Minimum 3 years of relevant experience. Required Skills: •Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making. •Extract, load and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files. •Write well designed, efficient code, that adheres to security standards. •Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner to minimize disruptions in data processing for on premise and cloud environments. •Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them. •Writes complex queries and scripts to efficiently manipulate, transform, and process raw data. •Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality. •Continuously optimize data pipelines for performance, scalability, and reliability. Qualifications: •Bachelor’s Degree in Arts/Sciences (BA/BS). An equivalent combination of education and experience may be considered in lieu of a degree. •Microsoft Certified: Azure Data Engineer Associate is preferred. •Microsoft Certified: Azure Data Fundamentals is preferred. •Strong knowledge of Big Data architectures, large data warehouse and Data Lake solutions. •Proficient in cloud services including Azure Databricks. Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview and Power BI. •Proficient in cloud deployments (MS Azure preferred) in an agile SDLC environment, leveraging modern programming languages, DevOps. •In-depth knowledge of data engineering concepts, ELT/ETL principles, and data modeling. •Solid programming skills in languages such as Python, Pyspark, SQL. Compensation: •$114,070 - $134,641 / year Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!