Job description & Roles and responsibilities
Work closely with Deloitte IT to design analytics and data solutions that align with firm strategy and meet business goals. Work in Azure cloud and other cloud environments to Azure Databricks that enables an enterprise view of credible data for ingestion into systems and reporting. Define and own end-to-end Architecture from definition phase to go-live phase. Review all artifacts developed by team and assist release mgmt. to promote changes to various lifecycles. Ensure clarity on NFR and address these requirements during Architecture definition phase. Define Logical, Technical & Physical views of Architecture. Define reusable components/frameworks, common schemas, standards to be used. Help Technical Lead in identifying key technical risks and mitigation plan for the same and in estimation and planning. Mentor & guide Tech Leads & Engineers.
Key Responsibilities
-
Platform Architecture & Design
-
Define architecture patterns for ingestion, transformation, and serving layers
-
Establish best practices for Unity Catalog, Delta Lake, and orchestration
-
Support scalable multi-domain data product architecture
-
Data Engineering & Delivery Leadership
-
Guide pod delivery teams
-
Define development standards for pipelines, notebooks, and CI/CD
-
Drive transition to Databricks-native pipelines
-
Performance Optimization & Cost Management
-
Optimize pipelines for performance and cost
-
Monitor cost anomalies and runtime variability
-
Platform Integration & Ecosystem Alignment
-
Integrate Databricks with Azure ecosystem and Power BI
-
Support Fabric alignment and code portability
-
Data Governance & Security
-
Implement governance using Unity Catalog
-
Ensure secure and scalable access patterns
-
Engineering Standards & Capability Build
-
Define reusable frameworks and templates
-
Mentor teams and build internal capability
Stakeholder Engagement
-
Partner with architects, product managers, and domain modelers
-
Provide input into roadmap and hiring decisions
Qualifications
Required Skills & Experience
-
At least 3 years of strong experience with Databricks (Spark, Delta Lake, notebooks, pipelines)
-
Experience with Unity Catalog and governance models
-
Experience with Azure Data Lake and data transformation frameworks
-
Proven experience designing enterprise-scale data platforms
-
Experience optimizing large-scale pipelines and distributed systems
-
Leadership experience in global, matrixed environments
-
Databricks certified
Nice to Have
-
Experience with Microsoft Fabric ecosystem
-
Power BI integration experience
-
ADF or legacy ETL migration experience
Spirit Omega is committed to a diverse and inclusive workplace. We welcome applications from anyone, including members of Indigenous peoples, Women, visible minorities, persons with disabilities, persons of minority sexual orientations and gender identities, and others with the skills and knowledge to productively engage with diverse communities.
Looking for more opportunities? Check out our website at jobs.spiritomega.com
#INDSPO
