Hiring: Sr. Data Integration Engineer Company: Sparksoft Corporation Location: Columbia, MD Job Posted Time: 2026-09-16 13:21:43 Employment Type: Hybrid Target Skills & Keywords : AI, AWS, Accessibility, Agile, Avro, CI/CD, CloudWatch, Confluence, Data Pipeline, Event-Driven, Git, HIPAA, IAM, Iceberg, Infrastructure as Code, Jira, Kinesis, Lambda, Parquet, PySpark, Python, Redshift, S3, SAFe, SNS, SQS, Spark, Step Functions About the job Experience: •7+ years of relevant experience •Strong Python development skills, including modular design, testing, debugging, packaging, dependency management, and performance optimization. •Hands-on experience developing data pipeline solutions with AWS Glue and integrating Glue with Amazon S3 and the AWS Glue Data Catalog. •Practical experience with multiple AWS data, integration, security, and monitoring services used to deliver end-to-end data pipelines. •Demonstrated experience ingesting, parsing, validating, transforming, and troubleshooting JSON and NDJSON, including nested structures, malformed records, schema drift, and large-file processing. Required Skills: •Design and develop resilient batch and event-driven data pipelines using Python, AWS Glue, and appropriate AWS managed services. •Build AWS Glue jobs, crawlers, workflows, triggers, and Data Catalog integrations to discover, transform, govern, and publish datasets. •Ingest and process structured and semi-structured data from files, APIs, databases, and streaming or messaging sources, with particular expertise in JSON and NDJSON formats. •Develop efficient Python components for parsing, schema validation, normalization, enrichment, deduplication, aggregation, and data quality checks. •Use AWS services such as Amazon S3, AWS Lambda, Amazon EventBridge, AWS Step Functions, Amazon SQS, Amazon SNS, Amazon Kinesis, Amazon Athena, Amazon Redshift, AWS Lake Formation, AWS Secrets Manager, AWS KMS, Amazon CloudWatch, and AWS IAM as solution needs dictate. •Create automated unit, integration, regression, and data reconciliation tests; embed data quality controls throughout the pipeline lifecycle. •Implement operational monitoring, logging, alerting, traceability, restartability, error handling, and recovery mechanisms for production pipelines. •Automate infrastructure and deployment processes using infrastructure as code and CI/CD practices. Compensation: •Competitive benefits and rewards package •Competitive compensation and a 401(k) with employer contributions to help you plan for the future Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!