Can Python Scale For Mainstream Pool Use

I’ve seen a lot of debate on this so I’m curious what everyone’s opinion is (since many here are developers themselves). Do you think the python reference pool code released by the Chia dev team can be adapted so that it can scale to handle a large amount of farmers? Yes or no and why? If you think there is a limit what would you estimate that limit to be?

Hopefully there will also be some helpful tips people can suggest for pool devs who are using Python for their pool.

Yes. Python powers the backend of Instagram amongst many other massive web apps, scaling isn’t really an issue in any language nowadays, it’s always possible one way or another:

https://instagram-engineering.com/profiling-cpython-at-instagram-89d4cbeeb898

2 Likes

If not there is always PyPy (JIT based) or Cython (C based) which are notable faster if you are not IO or OS bound. Just how much of the chia is pure CPU bound ?

Also some Python extensions (say NumPy) are native code anyway, so may not run faster or require a re-implementation

2 Likes

I’m late to this party, by here’s my $0.02.

On Python: Scaling web applications today isn’t about language choice, it’s about application design. I think a pool implemented in Python will work fine, provided it’s designed to support the load & features that a production pool requires. Having said that, VMs are expensive. I wouldn’t be surprised if serious Python-based pools end up rewriting particularly hot code in a more efficient language for cost savings (which, for CPU-bound workloads, can be very significant).

On the reference pool: I don’t think the reference pool is designed for production use. IIRC the dev team has explicitly stated that it’s meant to be a reference, and that they expect serious pool operators to have to make modifications to support robustness / security / new features.

1 Like