About the Role
At Think Right Technologies we are seeking a highly skilled and motivated Lead Data Engineer to spearhead our data initiatives, design scalable architectures, and manage data pipelines using Azure or AWS cloud platforms. The ideal candidate will have a strong technical background, proven leadership skills, and the ability to collaborate across teams to deliver impactful data solutions.
Key Responsibilities
Leadership and Strategy:
- Lead and mentor a team of data engineers in building and maintaining robust data infrastructure.
- Define and implement the data engineering roadmap aligned with organizational goals.
- Advocate for best practices in data management, security, and governance.
Data Architecture and Pipeline Development:
- Design and implement scalable, efficient, and reliable data pipelines and ETL/ELT processes.
- Optimize existing data architectures for performance and scalability in Azure or AWS environments.
- Manage large-scale data lakes, warehouses, and real-time streaming data.
Collaboration:
- Partner with data scientists, analysts, and other stakeholders to understand data requirements.
- Translate business needs into technical solutions that align with industry standards.
Cloud Expertise:
- Utilize Azure or AWS cloud services, including but not limited to Databricks, Synapse, Redshift, Glue, and S3.
- Monitor and manage cloud resources for cost and efficiency optimization.
Automation and Monitoring:
- Develop automation scripts for deploying and maintaining data workflows.
- Implement monitoring tools to ensure data pipeline reliability and system health.
Required Qualifications:
- Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Experience: 9 - 12 years of experience in data engineering, with 2 - 4 years leading technical teams.
Required Technical Skills:
- Proficiency in programming languages such as Python, Scala, or Java.
- Expertise in SQL and database systems (e.g., PostgreSQL, Snowflake, SQL Server).
- Hands-on experience with Azure (e.g., Data Factory, Synapse, Data Lake) or AWS (e.g., Glue, Redshift, S3).
- Strong understanding of data modelling, warehousing, and big data technologies (e.g., Spark, Kafka).
- Experience with CI/CD pipelines, Terraform, and version control systems like Git.
Required Leadership Qualities:
- Leadership: Ability to inspire and guide a team towards achieving goals.
- Ownership: Demonstrates a strong sense of responsibility for their tasks and outcomes. Takes initiative to identify and solve problems without waiting for direction.
- Problem-Solving: Creative and analytical thinking to overcome challenges.
- Decision-Making: Strong ability to make informed and timely decisions.
- Communication: Ability to convey ideas and instructions clearly and concisely. Capable of persuading and motivating others to achieve common goals. Provides constructive feedback in a way that encourages growth and improvement.
Preferred Qualifications
- Certifications in Azure or AWS (e.g., Azure Data Engineer Associate, AWS Certified Data Analytics).
- Experience with machine learning workflows and AI/ML tools.
- Familiarity with orchestration tools like Apache Airflow or Azure Data Factory pipelines.