Hiring: Senior Data Engineer Company: Versant Media Location: New York, NY Job Posted Time: 2026-09-10 11:15:43 Employment Type: Hybrid Target Skills & Keywords : Airflow, CI/CD, Dagster, Data Pipeline, Databricks, Delta Lake, EMR, Event-Driven, Feature Store, Flink, Git, MLOps, MLflow, PySpark, RBAC, SQL, Spark About the job Experience: •5+ years of experience building production-grade data pipelines in cloud environments using Spark-based platforms (e.g., Databricks, EMR, Dataproc, open-source Spark) Required Skills: •Design and implement lakehouse architecture using Delta Lake, including medallion pipeline patterns (Bronze/Silver/Gold), schema enforcement, and time travel •Build and operate batch and real-time ingestion pipelines leveraging Databricks Auto Loader, Structured Streaming, and Change Data Capture patterns •Implement data governance and security using Unity Catalog, RBAC, and compliance-driven practices for sensitive environments •Optimize performance and manage costs through FinOps strategies, including cluster sizing, workload tagging, Spark tuning, and Photon acceleration •Design, implement, and maintain CI/CD pipelines and orchestration workflows using Databricks Workflows, Delta Live Tables, and tools such as Airflow •Partner cross-functionally with Data Science teams on ML workflows, including MLflow, feature store integration, and model lifecycle management •Ensure data quality, observability, and lineage across media-specific datasets such as streaming logs, ad impressions, and audience metrics •Provide technical mentorship through code reviews, pairing and knowledge sharing Qualifications: •Bachelor’s degree in Computer Science, Data Engineering, or equivalent practical experience •Expertise in PySpark, SQL, and Spark-based data processing, with experience operating pipelines at scale in production •Hands-on experience building batch or streaming production data pipelines using distributed processing frameworks (e.g., Spark, Flink) and query engines such as Presto •Proficiency with orchestration tools such as Apache Airflow or Dagster, with hands-on experience in CI/CD, monitoring, alerting, and data quality for production systems •Proficiency with Git and collaborative development workflows •Build and operate batch and real-time ingestion pipelines using Spark based batch and streaming patterns (e.g., Structured Streaming, CDC), with experience on Databricks or comparable platforms •Solid understanding of infrastructure, networking, and data security fundamentals •Operational familiarity with Unity Catalog, data governance, and compliance frameworks (e.g., PCI) •Applied hands-on capability in CI/CD pipelines, orchestration tools, and infrastructure-as-code •Background in media and entertainment data (e.g., video metadata, ad tech, audience analytics) Compensation: •$140,000 - $160,000 / year Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!