Mastering Data Integration
Developing ETL Pipelines for Data Ingestion and Transformation
Data Engineering is a wide subject that spans a variety of titles, with a core focus on building dependable infrastructures to enable constant data flow in a data-driven environment. These people function as facilitators of clean and raw data from multiple sources so that people may utilize it to make data-driven choices inside the company.
Data engineering is the process of developing and constructing large-scale data collection, storage, and analysis systems. It's a wide-ranging field with applications in almost every sector. Organizations can gather large volumes of data, but they need the proper people and technology to guarantee that the data is in a useful form by the time it reaches data scientists and analysts.
Contact Us
Job responsibilities
Design database and data pipeline/ETL using emerging technologies and tools
Drive the team to develop operationally efficient analytic solutions
Define standards, methodologies for the Data Warehousing environment
Design and build highly scalable data pipelines using new generation tools and technologies like AWS, Snowflake, Spark, Kafka to induct data from various systems
Translate complex business requirements into scalable technical solutions meeting data warehousing design standards
Create scalable data pipelines and ETL applications that support business operations in advertising, content, and finance/accounting
Assist with deciphering data migration issues and improving system performance
Collaborate efficiently with product management, technical program management, operations, and other engineers
Minimum Requirements
BS, MS, or Ph.D. in Computer Science or a relevant technical field
Extensive experience building scalable data systems and data-driven products working with cross-functional teams
2+ years of related software/data engineering experience with a proficiency in Python
Ability to create data pipelines and ETL applications with large data sets
Proficiency in building REST APIs for back-end services
Exposure to implementing, testing, debugging, and deploying data pipelines using any of the following tools: Prefect, Airflow, Glue, Kafka, Serverless(Lambda, Kinesis, SQS, SNS), Fivetran, or Stitch Data/Singer
Experience with any of the Cloud data warehousing technologies: Redshift, BigQuery, Spark, Snowflake, Presto, Athena, S3
Experience with SQL DB administration (PostgreSQL, MS SQL, etc.)
Preffered Skills
Understanding of complex, distributed, microservice web architectures
Experience with Python back-end, ETL to move from one database to another
Solid understanding of analytics needs and proactive-ness to build generic solutions to improve the efficiency
© 2024. All Rights Reserved to Fly High Staffing Solutions
