“knowledge is power” -Little finger (Game Of Thrones) Introduction: Last time, we left Bob in quite a situation. He was struggling with his store’s ordering. In particular, his issue was to determine when to Z-order books again to maintain the effectiveness of his shelf skipping technique. Not to forget, ordering the…

“With great power comes great responsibility” -Spiderman Introduction: The world generates 2.5 quintillion bytes per day. That’s 1,000 petabytes! So in line with this huge amount of data volumes generated globally, adaptation features and techniques are becoming more and more crucial in the struggle to harness it. Extracting insights and business decisions from…

After the announcement of Delta 2.0 during the Data + AI Summit in which Databricks fully open sourced Delta Lake, and the release of Iceberg 0.14.0 which added performance improvements for scan planning and spark queries, the community’s interest in the impact of these releases on their respective performance has…

Introduction In our previous blog, we compared Delta 1.2.0, Iceberg 0.13.1 and Hudi 011.1 and we published our findings only to find out that Onehouse saw a misrepresentation of the true power of Apache Hudi. Although we disagree, we took this seriously and we decided to run the benchmark again using…

Introduction: After comparing delta vs iceberg in our previous blog, a lot of people asked for benchmarking their latest versions and for Apache Hudi to be thrown into the mix. So, by popular demand, we did exactly that and we performed TPC-DS on Delta 1.2.0 , Iceberg 0.13.1 and Hudi 0.11.1…

Introduction : A data Lakehouse is an open data architecture that brings together the scalability and cost-effectiveness of data lakes with the reliability and performance of data warehouses on a single data platform. Simply said, the data lakehouse is the only data architecture that allows you to store all types of data…

Introduction Last time we left Bob in a very Complicated situation, he was facing too many problems with too little time to solve them (Part-1). Bob decided to take a different approach to overcome those challenges. In order to work smarter, Bob needs to go through all the challenges in the…

Introduction Open sourced in April 2019, Delta Lake is a Databricks project that brings reliability, performance and lifecycle management to data lakes. In October 2019, the Delta Lake Project was hosted by the Linux Foundation to Become the Open Standard for Data Lakes. In this article we will have two parts…

Rise of the streaming use cases In 2019, a study by Lightbend and The New Stack [1] revealed that The use of stream processing for AI/ML applications increased four-fold in two years and they expect that it will keep increasing in the coming years. This rise of streaming use cases is related to the fact that…