Extract data from multiple XML files and load it to a data warehouse. The data comes from multiple dealers, and it contains vehicle sales and vehicle repair data (VW, Lexus, etc.).
Generate Data For DMS (Dealership Management System)
- Daily XML files (source) are relatively large in size.
- The data inside the files contain multiple child-parent relationship, some of which are not properly structured.
- Loading a large historical data with over 100 GB in size.
- Using ETL component that would flatten the XML files and map it against the table schema before the load process.
- Creating optimized queries for faster transfer and load (indexes added).
- Store the in-process data at the memory location instead of a physical location before loading it to the data warehouse.
- Error handling with log registration.
- Notification alert for successful or failed run.
- Automating the ETL to run periodically.
Subscribe For Awesome Articles