Duties and Responsibilities :
Build Data Pipelines : Managed the design, creation, maintenance, training, and optimization of data pipelines as workloads move from development to production for specific use cases.
Architecting, creating and maintaining data pipelines will be the primary responsibility of the Data Engineer.
Drive Automation through effective metadata management : The Data Engineer will be responsible for using innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity.
Skills and Experience :
B.A. or B.S. in Information Technology, Data Science, or related field.
The ideal candidate will have a combination of IT skills, data governance skills, data movement skills, IoT skills and business understanding.
At least 2 to 6 years or more of work experience in data management disciplines including data integration, modeling, optimization, and data quality with specific focus on IoT related use cases and applications.
Ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, schemas, metadata, and workload management.
js, RESTful services, GraphQL, message brokers (MQTT), JSON, HTML / HTML5, CSS, etc.)
Experienced in designing solutions based on cloud architectures using technical components such as : Azure IoT Hub, serverless computing (e.
g. AWS Lambda, Azure Functions), compute (e.g. Kubernetes containers), Cloud-based databases (e.g. Azure database), and raw storage (e.
g. storage blobs, S3, etc.).
Experience with Microsoft Azure and the various data services like Synopsis.