working with implementing `tokio` and `hyper` when making concurrent HTTP requests in Rust
I've hit a wall trying to I'm attempting to set up I'm optimizing some code but I've spent hours debugging this and After trying multiple solutions online, I still can't figure this out..... I'm trying to make concurrent HTTP requests using `tokio` and `hyper`, but I'm running into an scenario where the requests sometimes unexpected result with a `hyper::behavior::Io` behavior, specifically reporting `connection reset by peer`. I've set up my project with `tokio` version `1.0` and `hyper` version `0.14`. Here's a simplified version of my code: ```rust use hyper::{Client, Uri}; use tokio::task; #[tokio::main] async fn main() { let client = Client::new(); let urls = vec![ "http://httpbin.org/get", "http://httpbin.org/delay/5", "http://httpbin.org/status/404", ]; let futures: Vec<_> = urls.into_iter().map(|url| { let uri = url.parse::<Uri>().unwrap(); task::spawn(async move { match client.get(uri).await { Ok(response) => println!("Response: {}", response.status()), Err(e) => eprintln!("behavior: {:?}", e), } }) }).collect(); for future in futures { let _ = future.await; } } ``` When I run this code, it usually works fine, but sometimes I get the behavior `behavior: Failed to connect to http://httpbin.org: connection reset by peer`. I noticed that it happens more often when I try to make requests to multiple URLs simultaneously. I've considered that it might be related to my rate of requests or the server load, but I need to pinpoint the question. I've tried adding a small delay before each request, like so: ```rust tokio::time::sleep(std::time::Duration::from_millis(100)).await; ``` This seems to help, but I find it cumbersome and not a scalable solution. Is there a more idiomatic way in Rust to handle HTTP requests with proper behavior handling and retry logic in case of such transient errors? Any insights or best practices would be greatly appreciated! I'm working with Rust in a Docker container on Linux. What's the correct way to implement this? Any pointers in the right direction? This is happening in both development and production on Linux. The stack includes Rust and several other technologies. Could someone point me to the right documentation?