Manager II

Please find the job description below for M02 , Must be strong in System Design and Architecture in any development process(Java/Big Data or Python).

Exp - 13 to 16years
Location - Hyderabad

A Manager II is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals.
The role balances:
• People management
• Technical oversight
• Delivery accountability
Typically owns a single team of engineers

Team Leadership & Delivery
• Lead a team of software and data engineers (typically 5–10 members).
• Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements.
• Drive execution of large-scale distributed processing initiatives and data platform modernization efforts.
• Foster a culture of engineering excellence, innovation, and accountability.
Technical Oversight
• Provide technical leadership for large-scale distributed data processing systems built on Apache Spark.
• Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.
• Establish best practices for Spark application development, optimization, observability, and operational excellence.
• Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls.
• Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.
• Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.
Data Platform & Spark Engineering
• Design and oversee development of batch and real-time data processing pipelines using Apache Spark.
• Collaborate on architecture involving Spark, Iceberg, Hive, Hadoop, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.
• Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows.
• Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.
• Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.
Stakeholder Collaboration
• Work closely with product managers, architects, data scientists, platform teams, and business stakeholders.
• Translate business and data processing requirements into executable engineering plans.
• Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.
• Partner with cross-functional teams to deliver data-driven products and platform capabilities.
People Management
• Conduct performance reviews, coaching, and regular feedback sessions.
• Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.
• Build technical depth within the team through knowledge sharing and career development.
• Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.
Process & Quality Management
• Ensure Agile, SDLC and engineering governance practices are followed.
• Track delivery, quality, reliability, and operational metrics.
• Improve engineering productivity through automation, standardization, and platform improvements.
• Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.

Required Knowledge and Experience

Technical Expertise
• Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).
• Proven experience building and operating large-scale distributed data processing systems.
• Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.
• Experience with one or more programming languages such as Scala, Java, or Python.
• Experience with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.
• Experience designing and maintaining batch and streaming data pipelines.
• Strong understanding of distributed systems, cloud-native architectures, and scalability principles.
• Hands-on experience deploying and managing Spark workloads on AWS and/or GCP.
•
Leadership Experience
• Proven experience leading engineering teams and delivering complex technical initiatives.
• Ability to balance hands-on technical leadership with people management responsibilities.
• Experience driving cross-functional collaboration and influencing technical direction.
• Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.
•
Scope & Positioning
• Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.
• Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.
• Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.
•
Preferred Qualifications
• Experience with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.
• Experience with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.
• Experience managing terabyte-to-petabyte scale data processing environments.
• Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.
• Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.