Jumia is a expertise firm born in 2012, current in 14 African nations, with its fundamental headquarters in Lagos, Nigeria. Jumia is constructed round a market, a logistics service, and a fee service. The logistics service allows the supply of packages by a community of native companions, and the fee service facilitates the funds of on-line transactions inside Jumia’s ecosystem. Jumia is current in NYSE and has a market cap of $554 million.
On this publish, we share a part of the journey that Jumia took with AWS Skilled Companies to modernize its information platform that ran underneath a Hadoop distribution to AWS serverless primarily based options. A few of the challenges that motivated the modernization have been the excessive price of upkeep, lack of agility to scale computing at particular occasions, job queuing, lack of innovation when it got here to buying extra trendy applied sciences, advanced automation of the infrastructure and functions, and the lack to develop domestically.
Answer overview
The fundamental idea of the modernization challenge is to create metadata-driven frameworks, that are reusable, scalable, and ready to reply to the totally different phases of the modernization course of. These phases are: information orchestration, information migration, information ingestion, information processing, and information upkeep.
This standardization for every part was thought-about as a method to streamline the event workflows and decrease the danger of errors that may come up from utilizing disparate strategies. This additionally enabled migration of various sorts of knowledge following an identical method whatever the use case. By adopting this method, the info dealing with is constant, extra environment friendly, and extra simple to handle throughout totally different initiatives and groups. As well as, though the use circumstances have autonomy of their area from a governance perspective, on high of them is a centralized governance mannequin that defines the entry management within the shared architectural parts. Importantly, this implementation emphasizes information safety by imposing encryption throughout all companies, together with Amazon Easy Storage Service (Amazon S3) and Amazon DynamoDB. Moreover, it adheres to the precept of least privilege, thereby enhancing total system safety and lowering potential vulnerabilities.
The next diagram describes the frameworks that have been created. On this design, the workloads within the new information platform are divided by use case. Every use case requires the creation of a set of YAML recordsdata for every part, from information migration to information move orchestration, and they’re principally the enter of the system. The output is a set of DAGs that run the precise duties.
Within the following sections, we talk about the aims, implementation, and learnings of every part in additional element.
Information orchestration
The target of this part is to construct a metadata-driven framework to orchestrate the info flows alongside the entire modernization course of. The orchestration framework supplies a sturdy and scalable answer that has the next capacities: dynamically create DAGs, combine natively with non-AWS companies, enable the creation of dependencies primarily based on previous executions, and add an accessible metadata technology per every execution. Subsequently, it was determined to make use of Amazon Managed Workflows for Apache Airflow (Amazon MWAA), which, by the Apache Airflow engine, supplies these functionalities whereas abstracting customers from the administration operation.
The next is the outline of the metadata recordsdata which can be supplied as a part of the info orchestration part for a given use case that performs the info processing utilizing Spark on Amazon EMR Serverless:
proprietor: # Use case proprietor
dags: # Record of DAGs to be created for this use case
- identify: # Use case identify
sort: # Sort of DAG (might be migration, ingestion, transformation or upkeep)
tags: # Record of TAGs
notification: # Defines notificacions for this DAGs
on_success_callback: true
on_failure_callback: true
spark: # Spark job data
entrypoint: # Spark script
arguments: # Arguments required by the Spark script
spark_submit_parameters: # Spark submit parameters.
The concept behind all of the frameworks is to construct reusable artifacts that allow the event groups to speed up their work whereas offering reliability. On this case, the framework supplies the capabilities to create DAG objects inside Amazon MWAA primarily based on configuration recordsdata (YAML recordsdata).
This explicit framework is constructed on layers that add totally different functionalities to the ultimate DAG:
- DAGs – The DAGs are constructed primarily based on the metadata data supplied to the framework. The info engineers don’t have to put in writing Python code with a purpose to create the DAGs, they’re routinely created and this module is in control of performing this dynamic creation of DAGs.
- Validations – This layer handles YAML file validation with a purpose to stop corrupted recordsdata from affecting the creation of different DAGs.
- Dependencies – This layer handles dependencies amongst totally different DAGs with a purpose to deal with advanced interconnections.
- Notifications – This layer handles the kind of notifications and alerts which can be a part of the workflows.
One side to contemplate when utilizing Amazon MWAA is that, being a managed service, it requires some upkeep from the customers, and it’s necessary to have a superb understanding of the variety of DAGs and processes that you just’re anticipated to have with a purpose to fine-tune the occasion and acquire the specified efficiency. A few of the parameters that have been fine-tuned through the engagement have been core.dagbag_import_timeout, core.dag_file_processor_timeout, core.min_serialized_dag_update_interval, core.min_serialized_dag_fetch_interval, scheduler.min_file_process_interval, scheduler.max_dagruns_to_create_per_loop, scheduler.processor_poll_interval, scheduler.dag_dir_list_interval, and celery.worker_autoscale.
One of many layers described within the previous diagram corresponds to validation. This was an necessary element for the creation of dynamic DAGs. As a result of the enter to the framework consists of YML recordsdata, it was determined to filter out corrupted recordsdata earlier than making an attempt to create the DAG objects. Following this method, Jumia may keep away from undesired interruptions of the entire course of. The module that really builds DAGs solely receives configuration recordsdata that observe the required specs to efficiently create them. In case of corrupted recordsdata, data relating to the precise points is logged into Amazon CloudWatch in order that builders can repair them.
Information migration
The target of this part is to construct a metadata-driven framework for migrating information from HDFS to Amazon S3 with Apache Iceberg storage format, which entails the least operational overhead, supplies scalability capability throughout peak hours, and ensures information integrity and confidentiality.
The next diagram illustrates the structure.
Throughout this part, a metadata-driven framework in-built PySpark receives a configuration file as enter in order that some migration duties can run in an Amazon EMR Serverless job. This job makes use of the PySpark framework because the script location. Then the orchestration framework described beforehand is used to create a migration DAG that runs the next duties:
- The primary job creates the DDLs in Iceberg format within the AWS Glue Information Catalog utilizing the migration framework inside an Amazon EMR Serverless job.
- After the tables are created, the second job transfers HDFS information to a touchdown bucket in Amazon S3 utilizing AWS DataSync to sync buyer information. This course of brings information from all of the totally different layers of the info lake.
- When this course of is full, a 3rd job converts information to Iceberg format from the touchdown bucket to the vacation spot bucket (uncooked, course of, or analytics) utilizing once more an alternative choice of the migration framework embedded in an Amazon EMR Serverless job.
Information switch efficiency is healthier when the scale of the recordsdata to be transferred is round 128–256 MB, so it’s advisable to compress the recordsdata on the supply. By lowering the variety of recordsdata, metadata evaluation and integrity phases are diminished, dashing up the migration part.
Information ingestion
The target of this part is to implement one other framework primarily based on metadata that responds to the 2 information ingestion fashions. A batch mode is answerable for extracting information from totally different information sources (similar to Oracle or PostgreSQL) and a micro-batch-based mode extracts information from a Kafka cluster that, primarily based on configuration parameters, has the capability to run native streams in streaming.
The next diagram illustrates the structure for the batch and micro-batch and streaming method.
Throughout this part, a metadata-driven framework builds the logic to convey information from Kafka, databases, or exterior companies, that will likely be run utilizing an ingestion DAG deployed in Amazon MWAA.
Spark Structured Streaming was used to ingest information from Kafka subjects. The framework receives configuration recordsdata in YAML format that point out which subjects to learn, what extraction processes needs to be carried out, whether or not it needs to be learn in streaming or micro-batch, and by which vacation spot desk the data needs to be saved, amongst different configurations.
For batch ingestion, a metadata-driven framework written in Pyspark was carried out. In the identical means because the earlier one, the framework obtained a configuration in YAML format with the tables to be migrated and their vacation spot.
One of many elements to contemplate in this sort of migration is the synchronization of knowledge from the ingestion part and the migration part, in order that there isn’t a lack of information and that information just isn’t reprocessed unnecessarily. To this finish, an answer has been carried out that saves the timestamps of the final historic information (per desk) migrated in a DynamoDB desk. Each varieties of frameworks are programmed to make use of this information the primary time they’re run. For micro-batching use circumstances, which use Spark Structured Streaming, Kafka information is learn by assigning the worth saved in DynamoDB to the startingTimeStamp parameter. For all different executions, precedence will likely be given to the metadata within the checkpoint folder. This fashion, you can also make positive ingestion is synchronized with the info migration.
Information processing
The target on this part was to have the ability to deal with updates and deletions of knowledge in an object-oriented file system, so Iceberg is a key answer that was adopted all through the challenge as delta lake recordsdata due to its ACID capabilities. Though all phases use Iceberg as delta recordsdata, the processing part makes in depth use of Iceberg’s capabilities to do incremental processing of knowledge, creating the processing layer utilizing UPSERT utilizing Iceberg’s means to run MERGE INTO instructions.
The next diagram illustrates the structure.
The structure is much like the ingestion part, with simply modifications to the info supply to be Amazon S3. This method hastens the supply part and maintains high quality with a production-ready answer.
By default, Amazon EMR Serverless has the spark.dynamicAllocation.enabled parameter set to True. This selection scales up or down the variety of executors registered throughout the utility, primarily based on the workload. This brings numerous benefits when coping with several types of workloads, nevertheless it additionally brings concerns when utilizing Iceberg tables. As an example, whereas writing information into an Iceberg desk, the Amazon EMR Serverless utility can use numerous executors with a purpose to velocity up the duty. This can lead to reaching Amazon S3 limits, particularly the variety of requests per second per prefix. Because of this, it’s necessary to use good information partitioning practices.
One other necessary side to contemplate in these circumstances is the item storage file format. By default, Iceberg makes use of the Hive storage format, however it may be set to make use of ObjectStoreLocationProvider. By setting this property, a deterministic hash is generated for every file, with a hash appended straight after write.information.path. This will significantly decrease throttle requests primarily based on object prefix, in addition to maximize throughput for Amazon S3 associated I/O operations, as a result of the recordsdata written are equally distributed throughout a number of prefixes.
Information upkeep
When working with information lake desk codecs similar to Iceberg, it’s important to have interaction in routine upkeep duties to optimize desk metadata file administration, stopping numerous pointless recordsdata from accumulating and promptly eradicating any unused recordsdata. The target of this part was to construct one other framework that may carry out some of these duties on the tables throughout the information lake.
The next diagram illustrates the structure.
The framework, in addition to the opposite ones, receives a configuration file (YAML recordsdata) indicating the tables and the record of upkeep duties with their respective parameters. It was constructed on PySpark in order that it may run as an Amazon EMR Serverless job and might be orchestrated utilizing the orchestration framework similar to the opposite frameworks constructed as a part of this answer.
The next upkeep duties are supported by the framework:
- Expire snapshots – Snapshots can be utilized for rollback operations in addition to time touring queries. Nonetheless, they’ll accumulate over time and might result in efficiency degradation. It’s extremely advisable to frequently expire snapshots which can be not wanted.
- Take away previous metadata recordsdata – Metadata recordsdata can accumulate over time similar to snapshots. Eradicating them frequently can be advisable, particularly when coping with streaming or micro-batching operations, which was one of many circumstances of the general answer.
- Compact recordsdata – Because the variety of information recordsdata will increase, the variety of metadata saved within the manifest recordsdata additionally will increase, and small information recordsdata can result in much less environment friendly queries. As a result of this answer makes use of a streaming and micro-batching utility writing into Iceberg tables, the scale of the recordsdata tends to be small. Because of this, a way to compact recordsdata was crucial to reinforce the general efficiency.
- Exhausting delete information – One of many necessities was to have the ability to carry out arduous deletes within the information older than a sure time frame. This means eradicating expiring snapshots and eradicating metadata recordsdata.
The upkeep duties have been scheduled with totally different frequencies relying on the use case and the precise job. Because of this, the schedule data for this duties is outlined in every of the YAML recordsdata of the precise use case.
On the time this framework was carried out, there was no any automated upkeep answer on high of Iceberg tables. At AWS re:Invent 2024, Amazon S3 Tables performance has been launched to automatize the upkeep of Iceberg Tables . This performance automates file compaction, snapshot administration, and unreferenced file removing.
Conclusion
Constructing a knowledge platform on high of standarized frameworks that use metadata for various elements of the info dealing with course of, from information migration and ingestion to orchestration, enhances the visibility and management over every of the phases and considerably hastens implementation and growth processes. Moreover, by utilizing companies similar to Amazon EMR Serverless and DynamoDB, you possibly can convey all the advantages of serverless architectures, together with scalability, simplicity, versatile integration, improved reliability, and cost-efficiency.
With this structure, Jumia was in a position to scale back their information lake price by 50%. Moreover, with this method, information and DevOps groups have been in a position to deploy full infrastructures and information processing capabilities by creating metadata recordsdata together with Spark SQL recordsdata. This method has diminished turnaround time to manufacturing and diminished failure charges. Moreover, AWS Lake Formation supplied the capabilities to collaborate and govern datasets on varied storage layers on the AWS platform and externally.
Leveraging AWS for our information platform has not solely optimized and diminished our infrastructure prices but in addition standardized our workflows and methods of working throughout information groups and established a extra reliable single supply of reality for our information belongings. This transformation has boosted our effectivity and agility, enabling quicker insights and enhancing the general worth of our information platform.
– Hélder Russa, Head of Information Engineering at Jumia Group.
Take step one in the direction of streamlining the info migration course of now, with AWS.
Concerning the Authors
Ramón Díez is a Senior Buyer Supply Architect at Amazon Net Companies. He led the challenge with the agency conviction of utilizing expertise in service of the enterprise.
Paula Marenco is a Information Architect at Amazon Net Companies, she enjoys designing analytical options that convey gentle into complexity, turning intricate information processes into clear and actionable insights. Her work focuses on making information extra accessible and impactful for decision-making.
Hélder Russa is the Head of Information Engineering at Jumia Group, contributing to the technique definition, design, and implementation of a number of Jumia information platforms that help the general decision-making course of, in addition to operational options, information science initiatives, and real-time analytics.
Pedro Gonçalves is a Principal Information Engineer at Jumia Group, answerable for designing and overseeing the info structure, emphasizing on AWS Platform and datalakehouse applied sciences to make sure sturdy and agile information options and analytics capabilities.