Lead Data Engineer (Remote)
Earnest · Remote (USA)
Earnest empowers people with the financial capital they need to live better lives.
We're an accomplished team of technology, finance, and design geeks who believe consumer lending can be radically improved and are doing something about it. We are disrupting the trillion dollar student loan industry by redefining what it means to be creditworthy. We created a company that combines data science, streamlined design, and technology to:
- Build products that simplify the lending process
- Personalize loans to suit the needs of our customers
- Engage with our customers through more human experiences
Our culture is one that values transparency and blameless problem solving. Earnest has a strong track record of employee growth and career progression. Earnies are empathetic, product-focused, proactive, and curious.
As a Lead Data Engineer, you will report to the Head of Data and work closely with other members of the Data team to deploy predictive models as a service, build our data infrastructure, and maintain and improve data pipelines. We believe that data engineers should not write ETL, unless data pipelines are necessary to support the services that we build and maintain. Instead, your time is better spent writing tooling and creating abstractions to support the work of analysts and data scientists on the team, so that other users in the company can write and maintain their own ETL
What you’ll do:
- A lead data engineer will set up backend services in Python that support predictive models or third party integrations. This will involve the following:
- Setting up logging, monitoring and alerting.
- Writing the ETL to expose the data from predictive services
- Writing tests (unit, integration, regression, property, smoke, and e2e tests).
- Writing the business execution logic for the service
- Setting up a CI/CD pipelines
- Troubleshoot, identify, and fix defects through standard techniques such as debugging, profiling, logging, or log analysis via Splunk.
- Writing Database migrations and setting up database models.
- Writing case classes using existing REST API frameworks.
- Document technical designs in Confluence or Github Readmes
- A lead data engineer will be involved in writing ETL, ETL tooling, and Data Infrastructure for the following purposes:
- Automatic ingestion of data from production and other external services into the Data Lake (S3) and Data Warehouse (Snowflake).
- Setting up tooling and the abstraction patterns for orchestrating ETLs (Airflow, DBT) and other batch jobs that move data into the data-warehouse or from the data-warehouse.
- Setting up ELT monitoring and alerting
- Set up data infrastructure through Ansible and Terraform
- A lead data engineer will manage the data engineering backlog and put data product requirements together.
- Support cross-functional requirements gathering from engineering/product management, analytics and data science
Software Architecture and Leadership
- A lead data engineer will be involved setting up the technical direction for the team by proposing technologies, tools, languages that can be used to solve technical and business problems.
- Document, decompose, and size project planning in JIRA. Provide estimates around project completion and communicate acceptance criteria for when a project could be considered complete.
Mentoring and Coaching
- Trains other data engineers (both senior and junior)
- Coaches other engineers on best practices for exposing and storing data in production.
- Provide coding and architecture feedback using Github or through pair programming.
Ideal background and expertise:
- 6+ years of professional development experience with server-side concepts such as microservices, databases, caching, monitoring, and scalability.
- 4+ years in Python
- 3+ years with OLTP databases such as Postgresql, MySQL
- 3+ years with OLAP databases such as Snowflake, BigQuery, Redshift
- 2+ years working with container technologies such as Docker, Kubernetes
- 2+ years with AWS cloud technologies such as S3, EC2, VPC, IAM, RDS, SNS, Cloudfront, EMR, ELB.
- 2+ years working with workflow scheduling technologies, such as Airflow, Luigi.
Earnest Perks & Benefits:
- Great culture and an awesome team
- Health, Dental, & Vision benefits plus savings plans
- Employee Stock Purchase Plan
- 401(k) plan to help you save for retirement plus a company match
- Tuition reimbursement program
- $1000 flight on each Earnie-versary to anywhere in the world and 25 days of annual PTO
Earnest provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability or genetics. Qualified applicants with criminal histories will be considered for the position in a manner consistent with the Fair Chance Ordinance.