Lead Software Engineer - Java, Spark, Kubernetes, AWS, Databricks
Role details
Job location
Tech stack
Job description
As a Vice President, Lead Software Engineer in our platform engineering team, you will architect, build, and scale data-intensive platforms and services. You will own the design and delivery of high-throughput, low-latency applications, mentor engineers, and drive best practices across cloud-native, batch, and streaming workloads. You will partner with cross-functional teams to translate requirements into secure, reusable APIs and services, ensuring compliance and operational excellence. Your leadership will help shape our technical direction and foster a culture of innovation., * Lead architecture and end-to-end delivery of high-throughput, low-latency applications and data pipelines
- Design and implement resilient, observable, and cost-efficient microservices and data processing frameworks
- Build scalable Spark jobs on Databricks, optimize performance, and enforce coding standards
- Define and evolve platform engineering practices including CI/CD, infrastructure-as-code, and automated testing
- Drive Kubernetes-based runtime standards and enforce SRE/operability guardrails
- Partner with product, data, and platform teams to translate requirements into secure APIs, datasets, and services
- Mentor, coach, and conduct code/design reviews for engineers; set technical direction and uphold engineering excellence
- Champion cloud architecture on AWS, including networking, compute, storage, security, and cost governance
- Instrument systems with robust observability and implement runbooks, playbooks, and on-call readiness
- Ensure compliance with data governance, security standards, and regulatory requirements
- Enforce RBAC, encryption, and auditability across platforms
Requirements
- Strong track record of delivering production systems in software engineering roles
- Expertise in Java (8+) with deep knowledge of concurrency, memory management, and performance tuning
- Advanced experience with Apache Spark and Databricks, including job optimization and Delta Lake patterns
- Hands-on experience with Kubernetes (EKS or equivalent), deploying microservices, and service-to-service networking
- Proficiency in AWS services including IAM, VPC, EC2/EKS, S3, Lambda, Step Functions, and CloudWatch
- Experience with CI/CD tools, infrastructure-as-code, and containerization
- Solid data engineering fundamentals including partitioning, schema evolution, and data quality frameworks
- Practical knowledge of observability stacks, distributed tracing, and production troubleshooting
- Excellent communication and leadership skills
- Ability to influence architecture and guide teams through complex technical decisions
- Commitment to compliance and security standards
Preferred Qualifications, Capabilities, and Skills:
- Experience with Databricks Workflows/Jobs, Unity Catalog, and Delta Live Tables
- Familiarity with API design (REST/gRPC), messaging (Kafka/Kinesis), and streaming topologies
- Security-first mindset with experience in secrets management and compliance controls
- Background in financial services or other highly regulated environments
- Exposure to Python/Scala for Spark and SQL performance tuning