This job is OPEN to APPLY for ALL Nationalities, unless otherwise specified.
Job Summary
We are seeking a Senior Data Engineer to architect and develop the core data infrastructure that will underpin our AI-driven transformation. In this pivotal role, you will move beyond traditional ETL processes to design and deploy high-security Data Lakehouse environments, establishing a single, authoritative source of truth for our AI systems. You will lead the technical implementation of 'Workstream 1: Data & Platform Foundations,' a critical initiative for our flagship projects. Your responsibilities will include integrating complex enterprise systems such as SAP S/4HANA and Ariba, as well as processing unstructured data like technical drawings and regulatory documents. By collaborating across teams, you will build high-performance pipelines and systems that power Intelligent Supply Chain forecasting and generative AI tools. Operating within a structured 'Sprint Zero' framework, you will ensure robust data lineage, security, and compliance with defense-grade standards. This role demands expertise in both structured and unstructured data engineering, with a focus on creating scalable, secure, and high-performance data architectures. Your work will directly enable AI agents to optimize procurement processes and support engineers in developing next-generation systems, making you instrumental in transforming raw data into strategic advantage for national defense capabilities.
Key Responsibilities
Design and deploy defense-grade Data Lakehouse architectures to serve as the Single Source of Truth for AI-driven Supply Chain Intelligence, ensuring high-performance pipelines and systems for Intelligent Supply Chain forecasting and generative AI tools.
Lead the technical execution of Workstream 1: Data & Platform Foundations, mapping rigid enterprise systems (e.g., SAP S/4HANA, Ariba) and collaborating with cross-functional teams to integrate and process complex unstructured data (technical drawings, regulatory text) for AI applications.
Architect and deploy ingestion pipelines to extract high-volume transactional data from ERP systems like SAP S/4HANA, Ariba, and PLM, ensuring near real-time availability for forecasting models and AI agents.
Build connectors for external market intelligence feeds (e.g., S&P Global, Orbis, EcoVadis) to enrich internal procurement data with macroeconomic and geopolitical signals for enhanced decision-making.
Design and implement a standardized procurement data model and taxonomy across multiple entities, harmonizing fragmented datasets into a cohesive analytics layer.
Engineer pipelines to ingest, process, and transform unstructured technical data (PDF tender documents, CAD metadata, historical CONOPS) into vector-ready formats for Retrieval-Augmented Generation (RAG) applications.
Manage and optimize Vector Databases (e.g., Weaviate) to store embeddings of archival proposals and engineering snippets, ensuring high-speed retrieval for AI drafting assistants and generative tools.
Establish data lineage and traceability protocols to link requirements to physical components, supporting Model-Based Systems Engineering (MBSE) and the Digital Thread implementation.
Implement Role-Based Access Control (RBAC), audit logging, and data redaction policies to ensure compliance with export controls and strict on-premise security requirements.
Deploy automated data quality frameworks to validate Bill of Materials (BOM) completeness and cost data accuracy before ingestion into AI models.
Optimize data pipelines for on-premise GPU clusters and air-gapped environments, ensuring efficiency and performance within existing infrastructure constraints.
Operate within a structured Sprint Zero environment to ensure data lineage, security, and governance meet defense-grade standards.
Qualifications and Experience
5+ years of experience in Data Engineering, with at least 2 years focused on building pipelines for Machine Learning or Generative AI applications in an enterprise setting.
Expert proficiency in Python, SQL, and modern data engineering frameworks including Apache Spark, Kafka, and Airflow.
Strong experience extracting data from complex ERP environments, specifically SAP S/4HANA and SAP Ariba, with familiarity in SAP BTP as a plus.
Deep understanding of Data Lakehouse architectures (Databricks/Delta Lake), Relational Databases (PostgreSQL), and Vector Databases (Weaviate/Milvus).
Experience building pipelines for RAG solutions, Conversational AI agents, and classical ML models using tools such as dbt, dagster, or prefect.
Proficiency with containerization (Docker, Kubernetes) and CI/CD pipelines for deploying data workflows in secure environments.
Experience in Supply Chain, Manufacturing, or Defense sectors, with the ability to understand 'Bill of Materials' (BOM) structures and procurement lifecycles.
Ability to navigate the governance challenges between agile data work and rigid systems engineering requirements, ensuring data deliverables meet formal Stage Gate reviews.
Proven ability to collaborate with Data Scientists and Backend Engineers to define data schemas that support predictive modeling and AI agents.
Technical Skills
Expert proficiency in Python and SQL for data engineering tasks
Strong experience with modern data engineering frameworks, including Apache Spark, Kafka, and Airflow
Deep expertise in extracting and processing data from complex ERP environments, specifically SAP S/4HANA and SAP Ariba, with familiarity in SAP BTP as a plus
Comprehensive understanding of Data Lakehouse architectures, such as Databricks and Delta Lake, for scalable and secure data storage
Experience with relational databases, including PostgreSQL, and vector databases like Weaviate and Milvus for AI-driven applications
Proven ability to develop data pipelines for Retrieval-Augmented Generation (RAG) solutions, conversational agents, and classical machine learning models using tools like dbt, Dagster, or Prefect
Proficiency in containerization technologies, including Docker and Kubernetes, and experience implementing CI/CD pipelines for secure data workflow deployment
Knowledge of defense-grade security protocols, including Role-Based Access Control (RBAC), audit logging, and data redaction policies for compliance with export controls and on-premise security requirements
Experience building and optimizing pipelines for high-performance GPU clusters and air-gapped environments
Familiarity with automated data quality frameworks to validate critical datasets such as Bill of Materials (BOM) and cost data for AI model accuracy
Ability to design and implement standardized procurement data models and taxonomies to harmonize fragmented datasets across multiple entities
Experience engineering pipelines for unstructured data ingestion, including PDF tender documents, CAD metadata, and historical CONOPS, transforming them into vector-ready formats for AI applications
Knowledge of Model-Based Systems Engineering (MBSE) principles and the ability to establish data lineage and traceability protocols for the Digital Thread
DETAILS TO REGISTER FOR THIS JOB:
Make Sure Your CV is ATS-friendly.
https://candergroup.uae.stafio.ai/jd-short-view/senior-data-engineer-32834236-1518-40d9-964d-03d7e2b8f2fc/?jobId=32834236-1518-40d9-964d-03d7e2b8f2fc&src=LinkedIn
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
What is an ATS CV?
Applicant Tracking System or "ATS", is the software that 'READS' your CV and stores this information in a database. Like this:
If the CV is not made as per the RULES of the ATS, the information will get messed up in the database. In the image below, 'PMP' has gone into the 'Mobile Number' column. The Mobile Number '052 1234567' has gone into the 'Main Skill' column.
So a recruiter searching for a candidate with PMP certification, will never find this person, in spite of his 'CV information' existing in the database...
...And, he will wonder why he gets rejected in spite of being a PERFECT MATCH for the Job.
Our leading services are as follows. Click the link as per your experience level:
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
Join the Fastest Growing Job Group on LinkedIn:
Join the Middle East's Best Job Groups on Whatsapp:
Join our Telegram Group:
Submit your CV here, as well:
---------PAID ADVERTISEMENT------------
Watch, Like, and Subscribe to the YouTube Channels:
House of Focus creates original cinematic Deep Focus Music and immersive Flow State Soundtracks™ designed for studying, coding, reading, writing, creative work, and distraction-free productivity.
https://www.youtube.com/@HouseofFocus-28
Hedgehog Haven - Discover beautifully crafted, photorealistic hedgehog videos set in tranquil forests, cozy gardens, misty mornings, gentle rain, autumn leaves, snowy landscapes, and magical natural habitats from around the world.
https://www.youtube.com/@HedgehogHaven-6
If you want to promote your services/products as well, get in touch with us.
SOCIAL INITIATIVE:
We publish job vacancies on this Job Portal (https://www.dubai-jobs.me) and our Whatsapp groups (www.dubai-forever.com/whatsapp-jobs.html) for the benefit of job-seekers.
It is to help people who are searching for jobs from across the world.
This is a Social Initiative from our team @ dubai-forever.com, so please help in this Noble Task by Forwarding these jobs within your Network.
Do this GOOD DEED.
You never know who will benefit from it.
You've heard about KARMA right?
It's waiting to give you 10X of whatever you do... Do the Good Deed. FORWARD NOW!
π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π² π§π π°πΌ π¦πͺ πΈπ¦ πΆπ¦ π΄π²
