When talking about analytics systems, people often will first think about OLAP or data warehouses. There are two main camps that leverage different strategies to solve this problem, OLAP and Streaming Processing. Costs are minimized while still achieving ultra-low latency analytics. In a streaming-based real-time system, the event is processed in an incremental way and the query is kept running. But usually an OLAP query is heavy, and running such a query frequently is usually complicated and expensive. In case the user wants to lower latency, what he/she can do is decrease that scheduled interval. Assuming an anomaly event is randomly distributed in all event, a user can expected that the latency will be about 2.5 minutes. For a traditional OLAP system, if the user wants to make the analytic result be ready as soon as the event happens, the user has to run the query continuously, usually a scheduled query, say run a query in a interval of every 5 minutes. So this latency can be used to evaluate how long it takes to make a decision when events happen.įor a real-time system, it is important to enable immediate action when a specific event happens, so keeping the event latency low is important. The event latency here is defined as the time difference between when the event happens and when the analytic result is available (t1-t0). The latency is defined as the delay between the generation of the event and the event getting processed and emitted as an analytic result. Since the query is running in a streaming mode, and there is no bounded data start and data end, the response time makes no sense here. In a real-time streaming analytics system, the latency is different. This lag risks dramatically reducing the value of data for real-time systems. The query actually handles ‘old’ data that happened in the past. which end up with longer end-to-end latency.
Furthermore, some systems even make this ‘pre-query’ process very heavy or longer - with ETL, pre-computing, multiple-indexes, etc. However, this kind of ‘latency’ has nothing to do with the ‘pre-query’ process between the time when the event happens (t0) and the earliest time when data is available for query. The shorter response time might be helpful for dashboards or ad-hoc query purposes. The response time here is defined as the time difference between the start time of analytics and the result of analytics. But what’s the best way to achieve speed in your own analytics system? As Nathan Rothschild’s story shows us, speed wins. This additional time lag is expensive for enterprises. Depending on the query schedule, it could be minutes, hours, or even days later. As such, this definition doesn’t account for the time between initial data generation and actual query time. Our definition contrasts with traditional latency measurement standards used for query response time, which measure the amount of time it takes to process historical data. A real-time analytics system’s value is to make such processes happen with minimal delay. So, we see the definition of latency as the delay between data being generated and collected to data being processed and ready for decision making. In the age of the big data, the value of BI or analytics systems is to turn data into insight or wisdom. A win for the Rothschild family, and lovers of low-latency analytics, alike! So in a very real sense, the Rothschild family dynastic wealth was created in no small part from a low-latency data system, enabling Nathan to make bold decisions faster than other financiers. Rothschild leveraged this time/information advantage to buy up the government bond market, selling two years later for an enormous profit. The Rothschilds had set up an efficient information system, and knew that Napoleon had lost at Waterloo a day before the government did. Ever the savvy financier, Rothschild saw an opportunity and used speed to exploit it. In 1815, Napoleon was famously defeated at the Battle of Waterloo by the Duke of Wellington, marking the end of the Napoleonic Wars. One of my favorite examples of how “speed wins” comes from Nathan Rothschild, the renowned English-German financier. Those who can’t keep up? Relegated to the dustbins of history. Throughout history, the winning bet is usually on the faster beast or organization. Need for Speed: Evaluating Real-time Analytics Systems
0 kommentar(er)
