Principal Data Engineer
- Lead the design and implementation of advanced data architecture.
- Oversee production-grade MLOps and AI workloads.
- Architect secure cloud environments and optimize data pipelines.
Job Description
Lead AI & Data Platform Engineer
- Location: Melbourne (Hybrid Work Model)
- Salary: $180,000 – $210,000 + Super
- Industry: Financial Services
- Contract Type: Permanent / Full-Time
About the Role
We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.
You will be our internal Databricks Expert, driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.
Key Responsibilities
1. Databricks Platform Excellence
- Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
- Enforce robust data governance and secure discovery across the enterprise using Unity Catalog.
- Build, monitor, and scale production pipelines via Delta Live Tables.
2. Production AI & MLOps Infrastructure
- Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
- Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI, MLflow, Langchain, and Vector Databases.
- Manage corporate feature stores and optimize model serving endpoints.
3. AWS Cloud Architecture & Infrastructure as Code (IaC)
- Design secure, high-performance cloud environments using Terraform.
- Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
- Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.
4. Performance Tuning & Streaming
- Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
- Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
- Architect real-time streaming pipelines leveraging Kinesis or Kafka.
Your Profile
To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.
Technical Requirements:
- Core Languages: Mastery of PySpark, Python, SQL, and Scala.
- AI Frameworks: Strong familiarity with PyTorch, TensorFlow, or Hugging Face.
- DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
- Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy, compliance, and risk frameworks).
What’s on Offer
- Competitive Compensation: $180K – $210K base salary, depending on experience.
- Hybrid Flexibility: A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
- Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.
We vet all employer accounts and do our best to keep job ads safe, but scams can still occur. Be cautious when sharing personal information — never provide financial details or make payments during the application process. For extra security, use the Apply button on our site when proceeding.