Strange lag and crash when using AVFoundation to process video frames on iOS 16
I'm migrating some code and I'm working on a personal project and I'm currently implementing a feature that processes video frames using AVFoundation, but I've been encountering significant lag and occasional crashes when I try to handle the frames in real-time... My setup involves an `AVCaptureVideoDataOutput` to get the frames, and I'm processing them in the delegate method. Here's a snippet of my code: ```swift class VideoProcessingViewController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate { var captureSession: AVCaptureSession! override func viewDidLoad() { super.viewDidLoad() setupCaptureSession() } func setupCaptureSession() { // Setup code for camera capture captureSession = AVCaptureSession() guard let videoCaptureDevice = AVCaptureDevice.default(for: .video) else { return } let videoInput: AVCaptureDeviceInput do { videoInput = try AVCaptureDeviceInput(device: videoCaptureDevice) } catch { return } if (captureSession.canAddInput(videoInput)) { captureSession.addInput(videoInput) } else { return } let videoOutput = AVCaptureVideoDataOutput() videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoQueue")) if (captureSession.canAddOutput(videoOutput)) { captureSession.addOutput(videoOutput) } captureSession.startRunning() } func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { // Simulating heavy processing let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)! let ciImage = CIImage(cvPixelBuffer: pixelBuffer) // Process the ciImage way too much here... // This causes the lag and sometimes crashes } } ``` Despite running this on an iPhone 12, I notice that the frame processing takes about 200-300 ms per frame, which isn't acceptable for a smooth experience. Additionally, if I try to process too many frames in a row, the app crashes with an "EXC_BAD_ACCESS" error, mostly when I attempt to access properties of `ciImage`. I've tried moving the processing to a background queue, but it still doesn't resolve the lag issue. I've also checked to ensure that the camera permissions are granted, and I've got proper error handling in place. Any insights on how to improve performance or prevent the crashes? Would using Metal or offloading processing to a separate thread help in this case? What's the best practice here? I'm on Linux using the latest version of Swift. Has anyone dealt with something similar? This is my first time working with Swift stable.