We are looking for a GCP Data Engineering Specialist for the following responsibilities:
· Design, develop and operate production-grade data pipelines for data ingestion and processing, enabling downstream pipelines and analytics products, considering both technology and business requirements
· Design data compliance and access management of personally identifiable sensitive datasets in bigquery
· Work with and contribute to a DevOps setup (continuous integration, and deployment) on GCP
· Set up monitoring and alerting for data ops, quality and availability
· Work closely with Product Owner and business stakeholders to ensure business value realization as part of a cross-functional agile team with Product Owner, Data Scientists/Analysts/Stewards and Data/ML/Software/DevOps Engineers.
Technical architectExtract data from external source systems
Perform data profiling to validate data quality
ETL development using Googles Big Query and supporting tools
Prepare data and access management for advanced analytics and self-service
Create ingestions patterns that allow integration with external data sources
Mapping tables for additional dimensions
Flexible with agile mindset, be prepared to handle ad hoc requests from stakeholders, due to some things are implemented on the fly Familiar with Agile, Scrum
Self-driven, action- and goal-oriented, good communication to technical and non-technical stakeholders
Tools and techniques:
Google Cloud Platform.
Data Engineering: developing data pipelines/ETL for data lakes, data warehouses and data marts.
Complex for processing and analysis using Bigquery
DevOps, DataOps, CI/CD, Infrastructure as code/config
Apache Spark / Beam / Dataflow
Azure Database (SSMS) as data source
Azure portal (assigning roles and resources for Data scientists or other consumers, setting up Logic Apps, creating/managing data lakes)
Power Platform (Power BI, PowerApps, Power Automate)
What 3 things from the box above are most important?
· GCP Data Engineering/Warehousing