Job Duties / Job Description
Role: Data Engineer
As a Data Engineer, you play a critical role in building and maintaining the data infrastructure of our client. You are responsible for the design, construction, installation, and maintenance of large-scale processing systems. Your work ensures that data is accessible and readily available for use in analytics and reporting.
Key Responsibilities:
- Data Pipeline Design and Development:
- Design, implement, and maintain scalable data pipelines to ingest, process, and store data from various sources, including databases, data lakes, APIs, and logs.
- Develop and manage ETL processes for efficient data movement, transformation, and loading.
- Develop and implement data transformation processes to cleanse, enrich, and aggregate raw data into usable formats for analytics and reporting.
- Participate in developing and maintaining information sources and contacts as directed; may attend meetings to collect information related to projects.
- Create information models using ontologies and taxonomies and other semantic models.
- Implement information models in relational or graph databases as appropriate.
- Employ programming languages like SQL, NoSQL, OWL, RDF, SPARQL, R, or Python to create data pipelines, linking data sources and ensuring that data pipelines are reliable and efficient.
- Manipulate, transform, and organize data to prepare for analysis and visualization. – 50% OF THE WORK.
- Data Modeling:
- Create and maintain data models and schemas to support business requirements while ensuring data accuracy, integrity, and performance.
- Administer and optimize databases and data warehouses to ensure high availability, reliability, and efficient query performance. Implement robust data integration solutions, allowing different systems to share and sync data efficiently. – 15% OF THE WORK.
- Data Quality Assurance:
- Establish and maintain data quality checks and data monitoring processes to detect and resolve data inconsistencies and anomalies.
- Optimize data pipelines, databases, and queries to enhance system performance, reduce latency, and lower operational costs. – 10% OF THE WORK.
- Data Security:
- Implement data security measures, access controls, and encryption to protect sensitive information.
- Identify improvements to processes and workflows, related to physical and logical data protection.
- Collaborate with Information Security, Network Security and compliance teams to validate that Data security design, processes and procedures align with broader agency policies, standards and requirements. – 10% OF THE WORK.
- Collaboration:
- Work closely with data scientists, data analysts, and business stakeholders to understand their data requirements, provide support, and deliver solutions to meet their needs.
- Collaborate with senior study leaders to develop, construct, test, and maintain the data architecture for research projects or portfolios. – 10% OF THE WORK.
- Documentation:
- Document data engineering processes, configurations, and best practices for knowledge sharing and to maintain an accessible knowledge base. – 5% OF THE WORK.