Senior Data Engineer
- Salary
- ₹23–39 LPA
- Location
- Bangalore, India
- Level
- Senior
- Experience
- Senior
About the company & role
Senior Data Engineer Multimodal Pipelines AWS Data Lake GPU Validation AI Training Data ABOUT HUMYN LABS Humyn Labs builds the intelligence layer for physical-world AI. We convert human action — sight, sound, movement, touch — into training-grade multimodal data for embodied AI, captured across 20+ countries in the real environments where physical AI actually deploys, not the labs where it is built. Our egocentric corpus ships to robotics and world-model teams as labeled data: 6-DoF head pose, metric 3D hand keypoints, depth, object tracks. The labels are the product. Their accuracy is the company. Role Overview We are looking for a Senior Data Engineer to own, extend, and harden the multimodal data infrastructure that powers our AI training data supply chain. You will work directly with the Head of Data, building and maintaining production-grade pipelines that move real-world audio, video, and egocentric capture data from ingestion through AWS-scale storage, into GPU-backed technical validation, and finally to delivery-ready datasets for frontier AI lab customers. This role sits at the intersection of data engineering and ML infrastructure — you will own the full pipeline: ingestion, lake management, quality validation, and compute handoff. What You Will Work On Multimodal Ingestion & Preprocessing Build and maintain pipelines that ingest real-world audio, egocentric video, and image data at scale — covering format normalization, chunking, metadata extraction, and landing into S3-based storage with consistent schema and partitioning. AWS Data Lake Manage and extend the KGen data lake: Athena query optimization, Glue crawlers and cataloguing, Apache Hudi table management, Lake Formation column-level permissions, and S3 lifecycle policies at TB–PB scale. GPU Validation Pipeline Handoff Design and maintain the data layer that feeds GPU-backed technical validation workers — own sharding strategies, manifest generation, throughput optimization, and I/O design so validation compute is never the bottleneck. Understand how data format choices (Parquet, WebDataset, sharded archives) affect GPU-side loading performance. Airflow DAG Management Author, debug, and monitor Airflow DAGs for scheduled processing, GPU job orchestration, and pipeline coordination across ingestion, validation, and delivery stages. QC and Annotation Tooling Support the FastAPI-backed audio QC portal used by annotation workers; extend data validation and quality-check scripts across egocentric video and audio datasets. Universal Data Schema (UDS) Contribute to and enforce the Universal Data Schema for audio, image, and code modalities in the Humyn Labs dataset marketplace — covering schema evolution, versioning, and partition strategies. Infrastructure and Access Management Maintain AWS IAM, Lake Formation, and S3 bucket policies; manage data engineer access controls; handle cross-region data movement and vendor data sharing infrastructure. ETL and Third-Party Integrations Build and maintain ingestion pipelines from APIs (Twitch, gaming analytics, Google Forms) into DynamoDB and PostgreSQL where required by business context. You Should Have 4+ years in a data engineering role with end-to-end pipeline ownership Strong Python — async patterns, subprocess management, API clients, data processing at scale Hands-on AWS — Athena, Glue, S3, DynamoDB, Lake Formation; production-grade, not just familiarity Apache Hudi or Delta Lake — schema evolution, partition strategies, upsert patterns SQL proficiency — able to write and optimise complex analytical queries Experience with Airflow or an equivalent workflow orchestrator Demonstrated experience with large-scale media data pipelines — audio/video format conversion, metadata extraction, chunking, egocentric or multimodal datasets (hard requirement, not a plus) Understanding of how data format and I/O design affects downstream GPU compute workloads — WebDataset, sharded Parquet, tfrecord, or equivalent Nice to Have Direct experience designing data handoff for GPU clusters (AWS Batch GPU instances, Ray, or SLURM) Familiarity with ML training data formats and dataset standards used by AI labs (Hugging Face datasets, WebDataset, dataset cards) Experience with rclone, large-scale file transfer, or cloud-to-cloud sync pipelines Exposure to data lineage or provenance tooling (OpenLineage, DataHub, or custom metadata schemas)
About Humyn Labs
Supplies robotics training data. The round is $26M equity and $21M debt.