Has anyone tried the chiapos speed improvements?

CYPER · June 8, 2021, 7:18am

How do I revert back to the original chiapos?

Abhram · June 8, 2021, 6:08pm

69chargr · June 9, 2021, 2:49am

My results = 1 K32 plot in 1990 sec!!
Proofs 416 / 400, 1.04

Plot Name: plot-k32-2021-06-08-22-39-8747d228fe630e529388ee82e25daea5a29f6e5b362126b938c295507611df94
[P1] Table 1 took 24.8096 sec
[P1] Table 2 took 94.9047 sec, found 4295005556 matches
[P1] Lost 38245 matches due to 32-bit overflow.
[P1] Table 3 took 150.257 sec, found 4295028207 matches
[P1] Lost 60857 matches due to 32-bit overflow.
[P1] Table 4 took 200.829 sec, found 4295013337 matches
[P1] Lost 45380 matches due to 32-bit overflow.
[P1] Table 5 took 192.758 sec, found 4295087110 matches
[P1] Lost 120375 matches due to 32-bit overflow.
[P1] Table 6 took 171.707 sec, found 4295036538 matches
[P1] Lost 69557 matches due to 32-bit overflow.
[P1] Table 7 took 120.433 sec, found 4294961094 matches
Phase 1 took 956.058 sec
[P2] max_table_size = 4295087110
[P2] Table 7 scan took 10.9199 sec
[P2] Table 7 rewrite took 46.4142 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 28.6829 sec
[P2] Table 6 rewrite took 53.0487 sec, dropped 581337606 entries (13.5351 %)
[P2] Table 5 scan took 26.4871 sec
[P2] Table 5 rewrite took 49.8847 sec, dropped 762038528 entries (17.7421 %)
[P2] Table 4 scan took 28.3256 sec
[P2] Table 4 rewrite took 48.7869 sec, dropped 828877948 entries (19.2986 %)
[P2] Table 3 scan took 25.4886 sec
[P2] Table 3 rewrite took 48.9579 sec, dropped 855106230 entries (19.9092 %)
[P2] Table 2 scan took 24.9134 sec
[P2] Table 2 rewrite took 48.8068 sec, dropped 865575282 entries (20.1531 %)
Phase 2 took 450.005 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 47.5261 sec, wrote 3429430274 right entries
[P3-2] Table 2 took 31.5062 sec, wrote 3429430274 left entries, 3429430274 final
[P3-1] Table 3 took 51.2182 sec, wrote 3439921977 right entries
[P3-2] Table 3 took 34.6017 sec, wrote 3439921977 left entries, 3439921977 final
[P3-1] Table 4 took 54.3665 sec, wrote 3466135389 right entries
[P3-2] Table 4 took 32.8395 sec, wrote 3466135389 left entries, 3466135389 final
[P3-1] Table 5 took 52.9282 sec, wrote 3533048582 right entries
[P3-2] Table 5 took 34.0876 sec, wrote 3533048582 left entries, 3533048582 final
[P3-1] Table 6 took 55.1114 sec, wrote 3713698932 right entries
[P3-2] Table 6 took 35.2223 sec, wrote 3713698932 left entries, 3713698932 final
[P3-1] Table 7 took 65.1647 sec, wrote 4294961094 right entries
[P3-2] Table 7 took 51.6132 sec, wrote 4294961094 left entries, 4294961094 final
Phase 3 took 548.402 sec, wrote 21877196248 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 35.8864 sec, final plot size is 108835684962 bytes
Total plot creation time was 1990.39 sec

69chargr · June 9, 2021, 5:58am

Second plot was faster. I was using the PC during the first plot.

Interesting note: I was getting 43/day using the original peachy combined modified chiapos. If I divide seconds in a day 84,600/1969.03, I get… 43.88

So unless this new plotter lends itself to parallel processes, are we really gaining anything in most use cases?

Plot Name: plot-k32-2021-06-09-00-09-d24436b4bd4f0871b26a68eafb3fa60769a1d3d35f5e506028b34ae34047cdd0
[P1] Table 1 took 12.2591 sec
[P1] Table 2 took 96.7521 sec, found 4294950613 matches
[P1] Table 3 took 154.236 sec, found 4294871360 matches
[P1] Table 4 took 193.88 sec, found 4294970466 matches
[P1] Lost 2857 matches due to 32-bit overflow.
[P1] Table 5 took 190.932 sec, found 4294968294 matches
[P1] Lost 745 matches due to 32-bit overflow.
[P1] Table 6 took 177.279 sec, found 4294993872 matches
[P1] Lost 26367 matches due to 32-bit overflow.
[P1] Table 7 took 122.585 sec, found 4295042485 matches
Phase 1 took 947.996 sec
[P2] max_table_size = 4295042485
[P2] Table 7 scan took 13.0649 sec
[P2] Table 7 rewrite took 37.295 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 28.7185 sec
[P2] Table 6 rewrite took 52.859 sec, dropped 581269867 entries (13.5337 %)
[P2] Table 5 scan took 26.8092 sec
[P2] Table 5 rewrite took 50.0234 sec, dropped 761953248 entries (17.7406 %)
[P2] Table 4 scan took 25.6456 sec
[P2] Table 4 rewrite took 49.4492 sec, dropped 828870553 entries (19.2986 %)
[P2] Table 3 scan took 25.3471 sec
[P2] Table 3 rewrite took 49.0754 sec, dropped 855036472 entries (19.9083 %)
[P2] Table 2 scan took 26.2195 sec
[P2] Table 2 rewrite took 48.8021 sec, dropped 865574957 entries (20.1533 %)
Phase 2 took 442.474 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 44.9643 sec, wrote 3429375656 right entries
[P3-2] Table 2 took 32.8737 sec, wrote 3429375656 left entries, 3429375656 final
[P3-1] Table 3 took 50.5931 sec, wrote 3439834888 right entries
[P3-2] Table 3 took 33.6685 sec, wrote 3439834888 left entries, 3439834888 final
[P3-1] Table 4 took 51.9865 sec, wrote 3466099913 right entries
[P3-2] Table 4 took 33.1702 sec, wrote 3466099913 left entries, 3466099913 final
[P3-1] Table 5 took 52.1563 sec, wrote 3533015046 right entries
[P3-2] Table 5 took 34.0539 sec, wrote 3533015046 left entries, 3533015046 final
[P3-1] Table 6 took 55.4397 sec, wrote 3713724005 right entries
[P3-2] Table 6 took 34.3975 sec, wrote 3713724005 left entries, 3713724005 final
[P3-1] Table 7 took 62.5906 sec, wrote 4295042485 right entries
[P3-2] Lost 75189 entries due to 32-bit overflow.
[P3-2] Table 7 took 53.3106 sec, wrote 4294967296 left entries, 4294967296 final
Phase 3 took 541.674 sec, wrote 21877016804 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 36.839 sec, final plot size is 108834963207 bytes
Total plot creation time was 1969.03 sec

luckidog · June 9, 2021, 7:25am

   $ plotman analyze $(ls -rt plotter/*log | tail -n 40)    ################   # <---- New Hotness
    +-------+----+-------------+--------------+--------------+--------------+--------------+---------------+
    | Slice | n  |   %usort    |   phase 1    |   phase 2    |   phase 3    |   phase 4    |  total time   |
    +=======+====+=============+==============+==============+==============+==============+===============+
    | x     | 33 | μ=100.0 σ=0 | μ=7.6K σ=160 | μ=6.5K σ=147 | μ=5.9K σ=168 | μ=610.7 σ=21 | μ=20.7K σ=332 |
    +-------+----+-------------+--------------+--------------+--------------+--------------+---------------+


$ plotman analyze  $(ls -rt plotter-standard/*log | tail -n 32)   ######################### # <---- Stock busted
    +-------+----+-------------+--------------+--------------+--------------+---------------+--------------+
    | Slice | n  |   %usort    |   phase 1    |   phase 2    |   phase 3    |    phase 4    |  total time  |
    +=======+====+=============+==============+==============+==============+===============+==============+
    | x     | 31 | μ=100.0 σ=0 | μ=10.3K σ=2K | μ=5.3K σ=607 | μ=12.2K σ=4K | μ=842.7 σ=354 | μ=28.7K σ=6K |
    +-------+----+-------------+--------------+--------------+--------------+---------------+--------------+

w4d3 · June 9, 2021, 11:04am

Lol I’ll quote this weirdo for ya

w4d3 · June 9, 2021, 11:12am

Which “modified chiapos” are you referring to? I’m only aware of the one from pechy’s combined repo, are there others?

69chargr · June 9, 2021, 12:33pm

luckidog · June 9, 2021, 3:23pm

Or

cd chiapos
git checkout 1.0.2
pip install .

sc0 · June 10, 2021, 4:46pm

What’s the difference between the ‘main’ and ‘combined’ repo for this?

Harris · June 10, 2021, 6:46pm

The combined branch is similar to main except for the addition of a faster sort developed by another user (link below). It should perform better than main.

w4d3 · June 11, 2021, 5:40am

Thanks for the assist bud!