Enterprise Solutions, Inc.
https://www.enterprisesolutioninc.com/wp-content/uploads/2020/07/Enterprise-Logo-web.png
https://www.enterprisesolutioninc.com
https://www.enterprisesolutioninc.com
USD
60
70
HOUR
true
Must have skills:
Databricks, MLFlow, PySpark, SQL Adavance, Medallion Architecture
Job Description:
Senior ML Data Engineer Feature Engineering ETL
We seek a talented Senior Data Engineer with ML feature engineering expertise to join our Consumer ML team This role involves designing and implementing advanced feature engineering and ETL pipelines to enable robust machine learning applications The ideal candidate has hands-on experience with Databricks a deep understanding of the medallion architecture and a proven track record of supporting the end-to-end ML lifecycle Experience with MLflow and an aptitude for creating data driven insights are highly desirable.
Key Responsibilities
Feature Engineering Data Integration Develop and maintain feature engineering pipelines using Databricks to support ML models effectively
Data Pipeline Development Integrate diverse data sources eg clickstreams user behaviour demographic data to create user behaviour features profiles for complex ML tasks
Medallion Architecture Design and implement ETL, ELT pipelines aligned with the bronze silver and gold layers of the medallion architecture
Model Support Build data pipelines to support ML model training calibration and deployment leveraging MLflow for experiment tracking and performance monitoring
Query Optimization Low Latency Pipelines Design low latency production ready data pipelines to support real-time and batch model inference
CICD Practices Apply CICD principles for seamless pipeline deployment
Data Governance Ensure pipelines comply with security and regulatory standards particularly for handling PII and maintain metadata and master data across the data catalogue
Collaboration Work closely with ml scientist’s ml engineers and other stakeholders to align data transformation with business objectives
Qualifications
7 years in data engineering and at least 4 years focusing on ML feature engineering ETL pipeline development and data preparation for ML
Proven experience managing pipelines on Databricks using Apache Spark with a strong understanding of the medallion architecture
Familiarity with ML lifecycle management with MLflow experience as a strong plus and advanced skills in Apache Spark PySpark for big data processing and analytics
Proficient in Python for data manipulation and SQL for query optimization
Experience building pipelines for real-time and batch model serving in production environments and knowledge of CICD practices for ETLELT pipeline development
Expertise in metadata and master data management within technical data catalogues
Understanding of data security and compliance especially with sensitive data like PII
Senior ML Data Engineer
Mississauga, ON L4T 0A1 Canada
Posted: 12/11/2024
2024-12-11
2025-02-19
Job Number: z5G7h3l6a1kMvyS65NP3c9hHyOe_UAKDa3Na0919GKM=
Pay Rate: 60-70 Hourly CAD
Job Description
Must have skills:
Databricks, MLFlow, PySpark, SQL Adavance, Medallion Architecture
Job Description:
Senior ML Data Engineer Feature Engineering ETL
We seek a talented Senior Data Engineer with ML feature engineering expertise to join our Consumer ML team This role involves designing and implementing advanced feature engineering and ETL pipelines to enable robust machine learning applications The ideal candidate has hands-on experience with Databricks a deep understanding of the medallion architecture and a proven track record of supporting the end-to-end ML lifecycle Experience with MLflow and an aptitude for creating data driven insights are highly desirable.
Key Responsibilities
Feature Engineering Data Integration Develop and maintain feature engineering pipelines using Databricks to support ML models effectively
Data Pipeline Development Integrate diverse data sources eg clickstreams user behaviour demographic data to create user behaviour features profiles for complex ML tasks
Medallion Architecture Design and implement ETL, ELT pipelines aligned with the bronze silver and gold layers of the medallion architecture
Model Support Build data pipelines to support ML model training calibration and deployment leveraging MLflow for experiment tracking and performance monitoring
Query Optimization Low Latency Pipelines Design low latency production ready data pipelines to support real-time and batch model inference
CICD Practices Apply CICD principles for seamless pipeline deployment
Data Governance Ensure pipelines comply with security and regulatory standards particularly for handling PII and maintain metadata and master data across the data catalogue
Collaboration Work closely with ml scientist’s ml engineers and other stakeholders to align data transformation with business objectives
Qualifications
7 years in data engineering and at least 4 years focusing on ML feature engineering ETL pipeline development and data preparation for ML
Proven experience managing pipelines on Databricks using Apache Spark with a strong understanding of the medallion architecture
Familiarity with ML lifecycle management with MLflow experience as a strong plus and advanced skills in Apache Spark PySpark for big data processing and analytics
Proficient in Python for data manipulation and SQL for query optimization
Experience building pipelines for real-time and batch model serving in production environments and knowledge of CICD practices for ETLELT pipeline development
Expertise in metadata and master data management within technical data catalogues
Understanding of data security and compliance especially with sensitive data like PII
Share This Job:
Related Jobs:
Login to save this search and get notified of similar positions.About Mississauga, ON
Discover your next career opportunity in the vibrant area around Mississauga, Ontario! With its proximity to Toronto and the picturesque shores of Lake Ontario, this region offers a perfect balance of urban excitement and natural beauty. From the iconic CN Tower to the multicultural culinary scene at Celebration Square, there is something for everyone here. Explore job openings in industries spanning from finance to tech, and consider joining the thriving workforce of this dynamic area. Embrace the charm of Mississauga with its renowned art galleries like the Art Gallery of Mississauga, catch a show at the Living Arts Centre, or enjoy a game at the Paramount Fine Foods Centre. Start your journey towards a fulfilling career in Mississauga today!