Data Engineer

Data Engineer

Oakland, CA 94612

12 Months

Responsibilities

• Work closely with Subject Matter Experts (SMEs) to design and develop the ontology (data model) and data pipelines.
• Design, implement, and maintain robust data pipelines and ETL processes.
• Develop and optimize cloud-based data storage and processing solutions.
• Collaborate with data analysts and data scientists to understand data requirements and deliver high-quality data solutions.
• Ensure data integrity and security across all data workflows and storage solutions.
• Monitor and troubleshoot data pipelines, addressing any issues promptly to ensure the smooth flow of data.
• End to End data pipeline development – responsible for using Python and Pyspark to perform ETL and derive new datasets necessary for business applications.

Minimum Qualifications

• Bachelor's Degree: in Computer Science, Engineering, or related field or related experience.
• 5+ years of experience: as a data engineer.
• Hands-on experience: with building data pipelines.
• Strong SQL skills: and experience working with large datasets and complex data structures.
• Proficient in: Python, and Pyspark.
• Experience in: Typescript (preferred) or Javascript.
• Excellent problem-solving: and analytical skills with a strong attention to detail.

Preferred Qualifications

• Experience with: Palantir Foundry application development and data visualization tools.
• Knowledge with: commercial visualization tools such as Tableau or Power BI.
• Experience in: Typescript (preferred) or Javascript.
• Databases: familiarity with common relational database models and proprietary instantiations, such as SAP, Salesforce etc.
• Git: knowledge of version control / collaboration workflows and best practices.
• Agile: familiarity with agile and iterative working methodology and rapid user feedback gathering concepts.
• UX design: knowledge of best practices and applications.