非同期処理がレイテンシを隠蔽し、応答性を向上させる方法

Editor’s Note: This article contains an exclusive excerpt from Latency by Pekka Engberg, which helps readers diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now. You can download three chapters for free from ScyllaDB. And if you really want to geek out on latency, join the community at P99 CONF (free and virtual). 

非同期処理は、さらなるレイテンシ削減が非現実的または不可能なシステムにおいて、システムの同時実行性を向上させ、レイテンシを隠蔽するための強力な手法です。I/O操作を非同期で実行し、非クリティカルな作業を延期することで、エンドユーザーがレイテンシを認識しないため、知覚される応答性を大幅に向上させることができます。パーティショニングやキャッシング、その他のレイテンシ最適化のような手法が絶対的なレイテンシを削減できるのに対し、非同期処理は補完的な戦略を提供します。これは、一部の操作に時間がかかっても、システムが応答性を維持できるようにすることでレイテンシを隠蔽します。

“Asynchronous processing offers a complementary strategy—it hides latency by allowing the system to remain responsive even when some operations take time.”

この記事では、同期処理と非同期処理の基本的な違いを探り、イベントループが効率的な非同期実行を可能にする方法を検証し、非同期システムが対処しなければならない主要な課題とトレードオフを調査します。

Asynchronous vs. synchronous processing

Asynchronous processing enables tasks to execute independently and in overlapping periods, unlike synchronous processing, where tasks run in sequence. In other words, in a synchronous system, a task is executed to completion before the next one can begin. For example, suppose a single-threaded synchronous server is reading and processing messages. It first reads a message from a socket, which may require the thread to block until the message fully arrives. The server then processes the message, and it finally sends a response before starting to process the next message. If request processing takes a long time or blocks, the system waits synchronously.

非同期処理はこの制約を取り除き、複数のタスクが同時に進行できるようにします。たとえば、非同期処理を使用するサーバーは、オペレーティングシステム（OS）のインターフェースであるI/O多重化を使用して、さまざまな接続の状態をポーリングすることで、複数の独立したリクエストを同時に処理できます。その後、サーバーはソケットが読み取り可能または書き込み可能になるなどのイベントに反応してリクエストを処理できます。同様に、非同期サーバーはネットワーク経由で応答の送信を開始し、応答を待たずに他のタスクに取り組むことができます。

非同期処理はconcurrent programmingに似ています。ただし、非同期処理は明示的なインターフェースを持つため、同時プログラミングとは異なります。たとえば、スレッドを使用した同時プログラミングでは、サーバーはスレッド間のコンテキストスイッチングによって同時実行性を維持しながら、リクエストを同期的に処理できます。サーバーはsend()およびrecv()システムコールを実行し、ソケットから読み取るものが何もない場合、またはソケットが書き込み可能でない場合はブロックします。サーバーがブロックすると、OSは同時実行のために別のスレッドに切り替えます。対照的に、非同期処理では、サーバーはI/O多重化インターフェースを使用してソケットの状態をポーリングします。I/O多重化インターフェースは、サーバーにどのソケットが読み取り可能かを伝え、サーバーはスレッドをブロックせずにそこから読み取ることができます。同様に、サーバーが応答を送信する場合、非同期インターフェースを使用して応答を送信しますが、その後、すぐにブロックせずに作業を継続でき、OSがバックグラウンドで応答を送信できます。

Figure 1 illustrates the difference between synchronous and asynchronous processing (which are also summarized in the sidebar titled “Differences between asynchronous, concurrent, and parallel processing”). In this example, we have two tasks, A and B, that must run to finish our work in full. Suppose a backend system needs to communicate with external systems A and B to complete a request it has received. In synchronous processing, each task must finish before the next one starts. We run task A until it is complete, including the I/O it submits, and then run task B. The total time needed is the time of all tasks added together. If a backend service needs to do all these tasks, users must wait for this total time to get their response.

しかし、非同期処理では、タスクAのI/OをタスクBと同時に実行でき、両方が完了した時点で終了します。I/Oが同時に実行される場合、各タスクは依然として同じ時間がかかりますが、ユーザーの待機時間ははるかに短くなります。これは、データベース呼び出しや他のサービスの呼び出しを行うバックエンドサービスでうまく機能します。これらのタスクは互いにブロックせずに独立して実行できるためです。ただし、注意点があります。I/Oを並列に実行できない場合、非同期処理を使用しても役に立たず、代わりに非同期タスクの管理がシステムに余分な作業を追加するため、実行が遅くなる可能性があります。

非同期処理は、レイテンシを隠蔽するための重要な手法でもあります。一部の操作は、レイテンシを削減するための最善の努力にもかかわらず、完了するのに長い時間がかかるため、すべての人が完了を待たずに操作を実行することが不可欠です。たとえば、バックエンドシステムは通常、サードパーティサービス、データベースサーバー、メッセージキューなどの外部システムとやり取りします。各やり取りにはある程度のレイテンシが追加されます。同期処理では、固有の並列性を活用せず、システムが作業を完了するのを待つアイドル時間が発生するシステムを構築することがよくあります。対照的に、非同期処理では、操作を非同期で開始し、完了したときに反応することで、待機時間を最小限に抑えることができます。

In synchronous processing, you structure your code as a sequence of operations that depend on each other. For example, a request processing function for a synchronous server might look something like the following.

Listing 1 A simple example of a synchronous system

fn process_requests(socket: &Socket) {
  loop {
    process_request(socket);
  }
}

fn process_request(socket: &Socket) {
  let msg = socket.recv();
  let request = parse_message(msg);
  let resp = match request {
    Request::GetUserInfo(id) => get_user_info(id);    
  };
  let resp = format_response(resp);
  socket.send(resp);
}

At a high-level, we have the process_requests function, which processes any incoming requests from a socket. In the process_request function, each step is run to completion before we start another step. We read a message from the socket, we parse the message to determine what the request is, we process the request, and we finally send a response over the socket. More importantly, we don’t start another process_request until we’ve sent out a response, and we don’t allow requests to be processed from multiple sockets either.

While concurrency primitives like coroutines and futures enable parallel execution they’re insufficient for efficient asynchronous processing, particularly for I/O. You must structure the application differently if a server processes thousands of concurrent connections. The event loop is the foundation for efficiently multiplexing I/O operations across many connections.

The event loop

The event loop is the central coordinator for all input and output operations—it’s at the heart of an asynchronous system. While traditional synchronous programs handle one connection at a time—like a single worker processing tasks in sequence—an event loop operates as a dispatcher, simultaneously managing thousands of I/O operations. This architectural pattern, sometimes called an I/O loop or I/O dispatcher, is how asynchronous processing handles concurrent operations efficiently. Instead of dedicating separate resources to each connection, the event loop multiplexes various I/O sources—network connections, file operations, timers, and more—by tracking their states and processing them when they’re ready.

“The event loop is the central coordinator for all input and output operations—it’s at the heart of an asynchronous system.”

The event loop follows a simple yet powerful pattern:

1. Poll for events.
2. Process events.
3. Run scheduled tasks.
4. Repeat.

The event loop polls for events such as incoming data from a socket, an expired timer, or I/O completion by using OS-specific I/O multiplexing interfaces such as io_uring and epoll on Linux, kqueue on macOS, and IOCP on Windows. These interfaces let you register interest in an event source and get a notification when an event happens. For example, instead of reading data from a socket, the application expresses interest in a socket becoming readable. When data arrives from the network to the socket, the OS notifies the application, via the I/O multiplexing interface, that the socket is now readable. The event loop discovers this via polling and calls into the application’s event handling logic to process the newly arrived data from the socket.

Let’s implement a basic event loop in Rust to understand its structure better:

struct EventLoop {
    // Holds registered event sources like sockets, files, timers
    sources: Vec<EventSource>,    
}

impl EventLoop {
    fn run(&mut self) {
        loop {
            // Create a new collection to store events
            let mut events = Events::new();

            // Poll for new events with a timeout
            self.poll(&mut events, Duration::from_millis(100));

            // Process each event that was found
            for event in events.iter() {
                self.process_event(&event);
            }

            // Run any scheduled tasks
            self.run_scheduled_tasks();
        }
    }
}

The EventLoop::run() method demonstrates the core functionality of event-driven programming: continuously polling for and processing events. The poll() method uses an OS-specific I/O multiplexing interface, such as io_uring, for events on event sources. As you can see in the example code, we also specify a timeout for event polling. A timeout is needed because I/O polling in the event loop is often the only synchronous code that blocks the thread until an event happens. Polling can block if the system is idle and no events occur, and this blocking can reduce the wasted CPU cycles when there’s nothing to do. However, to ensure that the event loop does not block forever, the timeout ensures that we return from poll(). This allows the event loop to also perform work that is not conditional to an event, such as executing background work. However, in some cases, you might use busy-polling to avoid the sleep/wakeup cycle latency for some latency-sensitive event loops.

The process_event function is responsible for processing any events discovered during polling. For example, if the application registered interest in data arriving from the network (such as a socket becoming readable), the process_event function reads from the socket and forwards the data for the application to process. A simple process_event function might look something like this:

struct EventLoop {
    // Holds registered event sources like sockets, files, timers
    sources: Vec<EventSource>,    
}

impl EventLoop {
    fn run(&mut self) {
        loop {
            // Create a new collection to store events
            let mut events = Events::new();

            // Poll for new events with a timeout
            self.poll(&mut events, Duration::from_millis(100));

            // Process each event that was found
            for event in events.iter() {
                self.process_event(&event);
            }

            // Run any scheduled tasks
            self.run_scheduled_tasks();
        }
    }
}

As you can see, each event is represented by an Event enumeration with variants for different events. The event-processing logic is specific to how the event loop is structured. For example, if the event loop uses callbacks for event handling, it calls them, delegating work to the application. The application may then perform the work in the callback or submit the work to another thread for processing.

Figure 10.2 visualizes how the event loop performs work. In this example, work is split into three separate tasks:

1. Accept connection
2. Process request
3. Send response

The first task runs when the I/O multiplexer notifies the event loop that there is an incoming connection. The application reacts to the event by accepting the connection and then registering interest about when the accepted socket becomes readable. When data arrives from the network, the OS notifies the event loop that the socket is readable. The application reacts to this by reading from the socket and processing the incoming request. Finally, the application registers interest in the socket becoming writable. When the OS has enough buffer memory for an outgoing response, it notifies the application, which writes the response to the socket.

“While the event loop is the low-level infrastructure for asynchronous processing, you’ll also need some concurrency primitives to specify dependencies between individual tasks.”

If you contrast the event loop to a synchronous server, which you saw in listing 1, you’ll see two key differences between these approaches:

• Non-blocking operations—The event loop does not block the thread, but instead registers interest in events such as a socket becoming readable, and it defers reading from the socket until that condition is true, handling other events meanwhile.
• Resource efficiency—A single thread running an event loop can handle thousands of concurrent connections because it does not need to wait for I/O operations to complete. Instead, the I/O multiplexing OS interface allows the event loop to poll for the status of multiple event sources, such as sockets, at the same time, performing event-based processing.

While the event loop is the low-level infrastructure for asynchronous processing, you’ll also need some concurrency primitives, like callbacks or futures to specify dependencies between individual tasks.

Challenges

While asynchronous processing can significantly improve application performance, it comes with several important pitfalls to consider:

• Complexity—Asynchronous code is generally more complex than synchronous code. You need to carefully manage task dependencies, handle errors across multiple operations, and deal with race conditions.
• Resource management—Running many tasks simultaneously can consume significant memory and system resources. You need to implement proper throttling and resource management.
• Debuggability—When something goes wrong in asynchronous code, it can be harder to track down the issue because the execution order isn’t always obvious, and stack traces might not tell the whole story.
• Error handling—With multiple operations running independently, error handling becomes more complex. You need to decide how failures in one task should affect other running tasks.

YOUTUBE.COM/THENEWSTACK

Tech moves fast, don't miss an episode. Subscribe to our YouTube channel to stream all our podcasts, interviews, demos, and more.

Group Created with Sketch.