Research Programmer/Analyst - Machine Learning Department - School of Computer Science - CMU
Carnegie Mellon University, School of Computer Science, Delphi Group · Pittsburgh, PA (Hybrid/Remote)
The Delphi group in Carnegie Mellon's Machine Learning Department is looking for a talented engineer to prevent, identify, diagnose, and rectify problems or outages that occur in the data processing pipelines for our indicators. This person will ideally have competency in linux systems administration, basic database & script programming, data analysis, and software engineering in a team setting.
Carnegie Mellon University’s mission is to build a transformative educational experience for students focused on deep disciplinary knowledge; problem solving; leadership, communication, and interpersonal skills; and personal health and well-being. Carnegie Mellon’s School of Computer Science (SCS) is widely recognized as one of the first and the best computer science programs in the world. Our programs train the next generation of innovators to take real-world problems and improve the way people live and work.
Headquartered at Carnegie Mellon University, the Delphi research group was founded in 2012 to advance the theory and practice of epidemic forecasting. Pre-pandemic our focus was seasonal influenza; now it's COVID. Currently our focus is on developing and maintaining COVIDcast, which provides the nation's largest repository of diverse, geographically-detailed, real-time indicators of COVID activity. We work on the full data pipeline: several of the underlying data sources (on which these indicators are built) would not exist or be publicly available without our efforts; and on the other end, we use these indicators to power our nowcasting and forecasting models. COVIDcast is used regularly by public health officials, government agencies, data journalists, healthcare companies, financial firms, and fellow modelers. We work closely with the CDC, and are currently a CDC Center of Excellence for Forecasting (one of two nationally).
We are looking for a talented engineer to prevent, identify, diagnose, and rectify problems or outages that occur in the data processing pipelines for our indicators. This person will ideally have competency in linux systems administration, basic database & script programming, data analysis, and software engineering in a team setting. This is a combination of site reliability, operations, system administration, and light project management work. We currently run 12 individual indicator pipelines for tracking COVID, producing nearly 200 distinct time series. We're on schedule to exceed 2B data points over the summer, and plan to massively expand our coverage to include influenza in the coming months. Our job scheduling system was developed in-house, but for monitoring we use Elastic/Kibana and NewRelic. You do not need to have specific experience with either to apply.
Core responsibilities will include:
- Maintain indicator uptime and correctness
- Monitor system output
- Identify the cause of any runtime errors, and either make repairs directly or recommend alternate solutions
- Respond to bug reports from data users
- Monitor our data sources for any changes in formatting or data collection policy, and either make the necessary adjustments to our code, or flag for review
- Monitor system performance, and flag any trends that may jeopardize future uptime
- Design, implement, and deploy changes or systems that will scale with dramatic (10x) increases to the data we process daily
- Reduce maintenance effort
- Improve fault response time
- Decrease opportunities for human error
Flexibility, excellence, and passion are vital qualities within the School of Computer Science. Inclusion, collaboration and cultural sensitivity are valued competencies at CMU. Therefore, we are in search of a team member who is able to effectively interact with a varied population of internal and external partners at a high level of integrity. We are looking for someone who shares our values and who will support the mission of the university through their work.
Experience and qualifications:
- Bachelor’s Degree in Computer Science, Information Systems or related field required.
- Minimum of 1 year of related experience required.
- Basic linux system administration: system layout, file permissions, utilities (syslog, cron), diagnostic tools (ps, htop, grep)
- Basic database use: mysql, mysqldump
- Rough script programming (Python, bash)
- Software development in a team setting (Python certainly; R if you have it)
- Data analysis
- Diagnosing and fixing runtime errors and logic bugs
- Performing basic growth projections to predict future problems
- Communicating results
Successful background check