Lead Data Engineer (AWS)

Overview

Being good neighbors – helping people, investing in our communities, and making the world a better place – is who we are at State Farm. It is at the core of how we operate and the reason for our success. Come join a #1 team and do some good!

HYBRID:

Qualified candidates must live or relocate within a 180-mile radius of a hub location listed below and should plan to spend time working from home and some time working in the office as part of our hybrid work environment.

HUB LOCATIONS:

Bloomington, IL ; Richardson, TX ; Tempe, AZ ; Dunwoody, GA

SPONSORSHIP:

Applicants for this position are required to be eligible to lawfully work in the U.S. immediately; employer will not sponsor applicants for U.S. work authorization (e.g. H-1B visa) for this opportunity.

Responsibilities

• Innovation Group is seeking a Data Engineer or Lead Data Engineer to join our product execution team within SAIL (State Farm Applied Innovation Lab) focused on Agency and Digital. You will work alongside software engineers, product teams, and business partners to design and build the data capabilities that enable experimentation, product development, and emerging technology solutions.
• Our teams operate in environments where the problem, data, and solution may not be fully defined at the outset. We are looking for engineers who are comfortable working through that ambiguity, quickly learning new technologies and domains, and turning complex data needs into practical, scalable solutions.
• Day to day, the work centers on three areas: writing code to move and transform data; partnering with software engineers and product teams to determine how data can be accessed, analyzed, and used; and designing and maintaining the platforms where structured and unstructured data lives. Strong engineering fundamentals are essential, but success in this role also requires flexibility, curiosity, innovative thinking, and the ability to navigate a complex organization.
• The ideal candidate understands the fundamentals of data engineering and how data solutions are designed, built, deployed, monitored, and maintained as production-quality applications rather than one-off scripts. They communicate clearly, collaborate effectively across technical and non-technical teams, seek to understand the business problem behind the request, and continuously develop their skills as technologies and needs evolve. Depending on the initiative, this role may also help lead data science, machine learning, and generative AI efforts, working across the lifecycle from data exploration and experimentation through engineering, deployment, and ongoing operation.
• At the Lead Data Engineer level, the role also includes providing technical leadership, mentoring engineers, influencing architecture and engineering practices, and helping shape the direction of the team's data capabilities and solutions.

Qualifications

• 3 plus years with python development for extracting, transforming, processing, and moving data
• Proficiency writing SQL queries and working with relational databases
• Experience designing, building, and maintaining ETL/ELT data pipelines
• Experience working with structured, semi-structured, and unstructured data, including formats such as CSV, JSON, and text
• Experience with public cloud platforms, preferably AWS, including services such as S3, Glue, and Lambda
• Experience using Git and modern version-control practices
• Familiarity with CI/CD practices and deploying code into shared or production environments
• Comfort working in Linux and command-line environments
• Understanding of software engineering principles such as testing, maintainability, observability, and reusable design
• Ability to troubleshoot data quality, pipeline, integration, and performance issues
• Additional Skills That Would Be Valuable
• Experience designing data architectures for new products, experiments, or emerging technology use cases
• Familiarity with machine learning concepts and the end-to-end ML lifecycle, including data preparation, feature engineering, model development, evaluation, deployment, monitoring, and iteration
• Experience partnering with data scientists or ML engineers to operationalize models and build the data pipelines and infrastructure that support machine learning solutions
• Familiarity with generative AI, large language models (LLMs), and the data considerations associated with building AI-enabled applications
• Experience designing or building LLM-based workflows, including agentic workflows that use tools, APIs, retrieval, orchestration, or multi-step reasoning to complete tasks
• Experience using AI-assisted development tools as part of the engineering workflow to accelerate coding, testing, debugging, documentation, and technical exploration
• Ability to evaluate AI-generated code and outputs critically and apply appropriate engineering, security, testing, and governance practices
• Experience with APIs, event-driven architectures, or streaming data
• Familiarity with object storage and modern data platforms
• Experience working with large or complex datasets
• Understanding of data governance, security, privacy, lineage, and access-management considerations
• Experience working in Agile or product-oriented engineering environments
• Ability to translate business needs into technical data requirements and solutions
• Experience working across teams and managing technical dependencies
• Comfort experimenting with new technologies and solving problems where the path forward is not yet fully defined

Responsibilities

• Innovation Group is seeking a Data Engineer or Lead Data Engineer to join our product execution team within SAIL (State Farm Applied Innovation Lab) focused on Agency and Digital. You will work alongside software engineers, product teams, and business partners to design and build the data capabilities that enable experimentation, product development, and emerging technology solutions.
• Our teams operate in environments where the problem, data, and solution may not be fully defined at the outset. We are looking for engineers who are comfortable working through that ambiguity, quickly learning new technologies and domains, and turning complex data needs into practical, scalable solutions.
• Day to day, the work centers on three areas: writing code to move and transform data; partnering with software engineers and product teams to determine how data can be accessed, analyzed, and used; and designing and maintaining the platforms where structured and unstructured data lives. Strong engineering fundamentals are essential, but success in this role also requires flexibility, curiosity, innovative thinking, and the ability to navigate a complex organization.
• The ideal candidate understands the fundamentals of data engineering and how data solutions are designed, built, deployed, monitored, and maintained as production-quality applications rather than one-off scripts. They communicate clearly, collaborate effectively across technical and non-technical teams, seek to understand the business problem behind the request, and continuously develop their skills as technologies and needs evolve. Depending on the initiative, this role may also help lead data science, machine learning, and generative AI efforts, working across the lifecycle from data exploration and experimentation through engineering, deployment, and ongoing operation.
• At the Lead Data Engineer level, the role also includes providing technical leadership, mentoring engineers, influencing architecture and engineering practices, and helping shape the direction of the team's data capabilities and solutions.

Qualifications

• 3 plus years with python development for extracting, transforming, processing, and moving data
• Proficiency writing SQL queries and working with relational databases
• Experience designing, building, and maintaining ETL/ELT data pipelines
• Experience working with structured, semi-structured, and unstructured data, including formats such as CSV, JSON, and text
• Experience with public cloud platforms, preferably AWS, including services such as S3, Glue, and Lambda
• Experience using Git and modern version-control practices
• Familiarity with CI/CD practices and deploying code into shared or production environments
• Comfort working in Linux and command-line environments
• Understanding of software engineering principles such as testing, maintainability, observability, and reusable design
• Ability to troubleshoot data quality, pipeline, integration, and performance issues
• Additional Skills That Would Be Valuable
• Experience designing data architectures for new products, experiments, or emerging technology use cases
• Familiarity with machine learning concepts and the end-to-end ML lifecycle, including data preparation, feature engineering, model development, evaluation, deployment, monitoring, and iteration
• Experience partnering with data scientists or ML engineers to operationalize models and build the data pipelines and infrastructure that support machine learning solutions
• Familiarity with generative AI, large language models (LLMs), and the data considerations associated with building AI-enabled applications
• Experience designing or building LLM-based workflows, including agentic workflows that use tools, APIs, retrieval, orchestration, or multi-step reasoning to complete tasks
• Experience using AI-assisted development tools as part of the engineering workflow to accelerate coding, testing, debugging, documentation, and technical exploration
• Ability to evaluate AI-generated code and outputs critically and apply appropriate engineering, security, testing, and governance practices
• Experience with APIs, event-driven architectures, or streaming data
• Familiarity with object storage and modern data platforms
• Experience working with large or complex datasets
• Understanding of data governance, security, privacy, lineage, and access-management considerations
• Experience working in Agile or product-oriented engineering environments
• Ability to translate business needs into technical data requirements and solutions
• Experience working across teams and managing technical dependencies
• Comfort experimenting with new technologies and solving problems where the path forward is not yet fully defined