Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.399885468s FAIL (2025-10-20 17:24:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.399931445s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.325908665s FAIL (2025-10-20 17:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325938641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.033465456s FAIL (2025-10-20 17:23:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.033498359s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.185005938s FAIL (2025-10-20 17:23:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.185035443s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.407975594s FAIL (2025-10-20 17:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.408006133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.269836588s FAIL (2025-10-20 17:22:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.269871367s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.482829586s FAIL (2025-10-20 17:22:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.48286293s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.316699547s FAIL (2025-10-20 17:21:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.316729133s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.210982129s FAIL (2025-10-20 17:21:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.211012006s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.08820367s FAIL (2025-10-20 17:20:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.088239858s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.567498095s FAIL (2025-10-20 17:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.567531669s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.206878176s FAIL (2025-10-20 17:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.20691641s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.071948656s FAIL (2025-10-20 17:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.071977691s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.368265809s FAIL (2025-10-20 17:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.368294083s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.082328511s FAIL (2025-10-20 17:18:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.082388605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.374154109s FAIL (2025-10-20 17:17:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.37418524s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.366212059s FAIL (2025-10-20 17:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.366239461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.136274171s FAIL (2025-10-20 17:17:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.13631022s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.159305691s FAIL (2025-10-20 17:16:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.159340919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.164455442s FAIL (2025-10-20 17:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.164490428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.97306875s FAIL (2025-10-20 17:16:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.973103024s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.19354571s FAIL (2025-10-20 17:16:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.193575086s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.30643537s FAIL (2025-10-20 17:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.306469233s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.157807255s FAIL (2025-10-20 17:15:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.157854345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.785034102s FAIL (2025-10-20 17:14:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.785067887s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.23101656s FAIL (2025-10-20 17:14:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231048623s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.039180311s FAIL (2025-10-20 17:13:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.039225128s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.221311328s FAIL (2025-10-20 17:13:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.221340554s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.129191356s FAIL (2025-10-20 17:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129219248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.151289886s FAIL (2025-10-20 17:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.15132354s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.975407523s FAIL (2025-10-20 17:12:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.97543261s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.432419193s FAIL (2025-10-20 17:12:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.432456392s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.421893149s FAIL (2025-10-20 17:12:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.421933256s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.116236454s FAIL (2025-10-20 17:11:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.116269536s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.063692356s FAIL (2025-10-20 17:11:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.063726803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.365861564s FAIL (2025-10-20 17:10:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.365891329s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.124247134s FAIL (2025-10-20 17:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.124283803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.290119616s FAIL (2025-10-20 17:10:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.290154133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.664538983s FAIL (2025-10-20 17:09:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.664572878s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.226536533s FAIL (2025-10-20 17:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.226563865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.068052811s FAIL (2025-10-20 17:09:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.068092847s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.364771234s FAIL (2025-10-20 17:08:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.36480063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.196777817s FAIL (2025-10-20 17:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.196808947s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.149564803s FAIL (2025-10-20 17:08:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.149628345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.251463718s FAIL (2025-10-20 17:07:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.251492032s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.201696042s FAIL (2025-10-20 17:07:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.20172711s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.238354226s FAIL (2025-10-20 17:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.238377801s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.453272033s FAIL (2025-10-20 17:04:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.453296189s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000358859s FAIL (2025-10-20 17:04:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000379949s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.909451861s FAIL (2025-10-20 14:49:36 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 6.909484643s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)