This role is for one of Weekday’s clients Salary range: Rs 8000000 - Rs 20000000 (ie INR 80 - 200 LPA) Min Experience: 5+ years Location: Bengaluru JobType: full-time We are seeking an experienced Senior Data Engineer with 5–15 years of experience to design, build, and optimize scalable data systems and pipelines. The ideal candidate will have strong expertise in data mining, pattern identification, large-scale systems, data processing, and distributed architectures . You will work closely with engineering, analytics, product, and business teams to transform complex datasets into reliable, actionable data solutions. Key Responsibilities - Design, develop, and maintain scalable data pipelines, ETL/ELT workflows, and data processing systems . - Perform advanced data mining across large and complex datasets to identify trends, relationships, anomalies, and valuable insights. - Analyze datasets to identify meaningful patterns and correlations that can support business and product decisions. - Build and optimize data architectures capable of handling large-scale data volumes and high-throughput workloads . - Develop reliable batch and real-time data processing solutions using distributed computing technologies. - Work with structured and unstructured data from multiple sources and ensure data is accurately processed and transformed. - Optimize data pipelines, queries, storage, and processing frameworks for performance, scalability, and reliability. - Implement data quality, validation, monitoring, and error-handling mechanisms across data workflows. - Collaborate with Data Scientists, Analysts, Software Engineers, Product Managers, and other stakeholders to understand data requirements. - Design and maintain data models, schemas, and data integration frameworks. - Troubleshoot complex data and pipeline issues and drive improvements in system reliability and efficiency. - Contribute to technical architecture decisions and establish best practices for large-scale data engineering. - Mentor junior and mid-level engineers and contribute to improving engineering standards and processes. Must-Have Skills - Data Mining - Strong understanding of pattern identification, trend analysis, and data exploration - Experience working with Large Scale Systems - Strong knowledge of data structures, algorithms, and database concepts - Hands-on experience building ETL/ELT pipelines and data processing workflows - Experience with SQL and relational databases - Proficiency in at least one programming language such as Python, Java, or Scala - Experience with distributed data processing and large-scale data platforms - Strong understanding of data modeling, data warehousing, and data architecture - Experience optimizing data pipelines and processing workloads for performance and scalability - Strong analytical and problem-solving abilities Good-to-Have Skills - Experience with Apache Spark, Kafka, Hadoop, Flink, or similar distributed technologies - Exposure to cloud platforms such as AWS, Microsoft Azure, or Google Cloud - Experience with data lakes, lakehouses, and modern data warehouse technologies - Knowledge of real-time and streaming data architectures - Experience with data governance, observability, and data quality frameworks - Familiarity with machine learning or analytics workflows is a plus - Experience working with high-volume, high-velocity, or complex datasets

Senior Data Engineer

similar openings