It's a pretty big app. I'd be curious to see how big the equivalent app built using Web technologies directly would be.
That said, yes, if you want small download sizes on the Web then Flutter isn't the technology for you.
Concretely: our minimal overhead is about 300KB gzipped currently. We expect that to go down a bit before we exit beta, but a "hello world" in Flutter for Web will never get down to the few dozen bytes that you could get using pure HTML.
Fundamentally our approach will always have a large overhead compared to other Web technologies because we don't use most of the built-in features that browsers bring to the table like CSS and HTML and so on. We have our own layout, our own compositing, our own gesture detector, etc. We even blink our own cursor. The only features we use are the lowest-level ones like `canvas` that are needed to actually push pixels to the screen, or ARIA which is needed to expose the accessibility tree.
On the plus side that means you can do anything you want. You want the text cursor to look like a wiggling snake with eyes that follow the mouse cursor? No problem. You want a checkbox that exactly matches the Material Design spec? Or that exactly matches the iOS design language? No problem. The cost is that all that code has to ship with your app.
IMO it'd make more sense if they gave priority to desktop. The world desperately needs a good Electron alternative.