About the role
Senior Data Engineer- SQL Server to Databricks Migration
Job Type: Contract (6-12 Months)- Possibleextension
Location:Philadelphia, PA /Vienna, VA- Hybrid / Open to remote job
Job description:
üJob summary
üWe are seeking a senior data engineer with deep Microsoft SQL Server experience to assess and convert a legacy warehouse to Azure Databricks.
üYou will read complex T-SQL stored procedures and SSIS packages, document the business rules they contain, rebuild them as Databricks pipelines, and reconcile the results before the legacy platform is retired.
Key responsibilities
üAssess T-SQL stored procedures, SSIS packages and scheduled jobs for complexity, dependencies, and embedded business rules.
üRecommend, for each object, whether to retire, convert or redesign it.
üConvert legacy logic to Databricks SQL and PySpark pipelines using agreed patterns and conversion tooling.
üReview and correct automatically converted code.
üBuild and run reconciliation between SQL Server and Databricks and resolve differences with business analysts.
üMigrate job schedules and dependencies to Databricks workflows.
üSupport cutover and the evidence needed to decommission each legacy component.
üDocument data flows, mappings and design decisions in the client's repositories.
Required qualifications
ü8+ years in data engineering or database development, including 5+ years of hands-on T-SQL and SSIS.
ü2+ years building pipelines on Databricks, or an equivalent Spark-based lakehouse platform, with Delta Lake.
üAt least one completed migration from SQL Server or another warehouse to a cloud data platform.
üStrong SQL performance tuning and the ability to reverse-engineer undocumented code.
üExperience with data reconciliation and testing at scale.
üGit-based development with pull requests and automated deployment.
Preferred qualifications
üDatabricks Lakeflow pipelines and jobs, and Unity Catalog.
üCode-conversion tools such as Databricks Lakebridge.
üDatabricks Data Engineer Associate or Professional certification.
üFinancial services or wealth management data.
üExperience with enterprise schedulers such as SQL Agent or ActiveBatch.