How to solve: Rendering a Canvas to an Image in a way that doesn't lock up Flutter UI


Asked by nietaki on December 08, 2021 (source).

I need to draw some images in Flutter using geometric primitives, to both show in-app and cache for later use. What I'm doing right now is something similar to this:

import 'dart:ui';

final imageDimension = 600;
final recorder = PictureRecorder();
final canvas = Canvas(
        Rect.fromPoints(const Offset(0.0, 0.0),
            Offset(imageDimension.toDouble(), imageDimension.toDouble())));

/// tens of thousands of canvas operations here:
// canvas.drawCircle(...);
// canvas.drawLine(...);
final picture = recorder.endRecording();

// the following call can take ~10s
final image = await picture.toImage(imageDimension, imageDimension);
final dataBytes = await image.toByteData(format: ImageByteFormat.png);

Here's an example of the outcome:

rendered canvas

I know the image operations in this amount are heavy and I don't mind them taking some time. The problem is since they're CPU bound, they lock up the UI (even though they are async and outside of any widget build methods). There doesn't seem to be any way to break the picture.toImage() call into smaller batches to make the UI more responsive.

My question is: Is there a way in Flutter to render a complex image built from geometric primitives in a way that doesn't impact the UI responsiveness?

My first idea was to do the heavy calculation inside a compute() isolate, but that won't work since the calculations use some native code:

E/flutter (20500): [ERROR:flutter/lib/ui/] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object extends NativeWrapper - Library:'dart:ui' Class: Picture)
E/flutter (20500): #0      spawnFunction (dart:_internal-patch/internal_patch.dart:190:54)
E/flutter (20500): #1      Isolate.spawn (dart:isolate-patch/isolate_patch.dart:362:7)
E/flutter (20500): #2      compute (package:flutter/src/foundation/_isolates_io.dart:22:41)
E/flutter (20500): #3      PatternRenderer.renderImage (package:bastono/services/pattern_renderer.dart:95:32)
E/flutter (20500): #4      TrackRenderService.getTrackRenderImage (package:bastono/services/track_render_service.dart:111:49)
E/flutter (20500): <asynchronous suspension>

I think there is an alternative approach with CustomPaint painting just a couple of elements every frame and after it's all done screenshot it somehow using RenderRepaintBoundary.toImage(), but there's a couple of problems with this approach:

  • It's much more complicated and relies on using heuristics to choose the amount of elements that can be rendered to the canvas per frame (I think?)
  • I think the Widget would need to be visible in order for it to get rendered so I couldn't really use it to render images in the background (?)
  • I'm not exactly sure how I'd get RenderRepaintBoundary for the widget I use for rendering.

Edit: it seems like the screenshot package allows for taking screenshots that are not rendered on the screen. I don't know about its performance characteristics yet, but it seems like it could work together with the CustomPaint class. It still feels like a very convoluted workaround though, I'd be happy to see other options.


Question answered by nietaki (source).

Probably the best approach is moving rendering from dart:ui to a library that can be used in the compute isolate.

The image package is written in pure Dart, so it should work well and it does support the primitive operations I need.

I'm currently using the Canvas solution from the other answer, and it works for the most part, but I will switch to this soon.