Performance implementing np.where on large boolean arrays in NumPy 1.24.0
Hey everyone, I'm running into an issue that's driving me crazy. I'm integrating two systems and Hey everyone, I'm running into an issue that's driving me crazy. I'm working with important performance optimization when using `np.where` on large boolean arrays in NumPy 1.24.0. I have a boolean mask that is about 10 million elements long, and I attempt to extract the indices of `True` values for further processing. However, the execution time seems to increase disproportionately with the size of the input array. Below is a simplified version of my code: ```python import numpy as np # Creating a large boolean array mask = np.random.choice([False, True], size=10_000_000, p=[0.7, 0.3]) # Attempting to find indices of True values indices = np.where(mask)[0] ``` While this works fine for smaller arrays, I find that it takes a noticeable amount of time as the array grows. I've tried using `np.flatnonzero` as an alternative, but the performance doesn't seem to improve significantly: ```python indices_flat = np.flatnonzero(mask) ``` I also tried to run a basic performance test using the `%timeit` magic command in a Jupyter notebook, and it showed that both methods take several seconds for the larger array. Is there a more efficient way to handle this, or is this expected behavior when dealing with such large datasets? If it helps, my system has 16GB of RAM and I'm running Ubuntu 20.04. Any insights or suggestions would be greatly appreciated. Any suggestions would be helpful. I'm working on a CLI tool that needs to handle this. My team is using Python for this CLI tool. Has anyone else encountered this?