Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.356642589s FAIL (2025-10-20 17:48:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.356670592s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.152144416s FAIL (2025-10-20 17:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.152183642s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.359621437s FAIL (2025-10-20 17:47:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.359657515s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.282149067s FAIL (2025-10-20 17:46:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.282182122s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.052386218s FAIL (2025-10-20 17:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.052421254s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.382765486s FAIL (2025-10-20 17:45:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.382791487s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.127862763s FAIL (2025-10-20 17:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.127894793s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.133113685s FAIL (2025-10-20 17:45:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.133148401s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.022550993s FAIL (2025-10-20 17:44:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.02259751s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.187562086s FAIL (2025-10-20 17:44:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187594528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.032893376s FAIL (2025-10-20 17:43:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.032919427s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.360669363s FAIL (2025-10-20 17:43:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.360697776s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.025066582s FAIL (2025-10-20 17:43:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.0251079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.990065065s FAIL (2025-10-20 17:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.990097307s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.111508823s FAIL (2025-10-20 17:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.111538037s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.293469293s FAIL (2025-10-20 17:41:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.294412769s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.022178775s FAIL (2025-10-20 17:41:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.02221311s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.206646545s FAIL (2025-10-20 17:41:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.206677814s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.372483098s FAIL (2025-10-20 17:41:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.37250522s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.1659173s FAIL (2025-10-20 17:40:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.165947738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.375822375s FAIL (2025-10-20 17:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.376042043s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.089828691s FAIL (2025-10-20 17:39:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.089852657s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.080851207s FAIL (2025-10-20 17:39:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.080885833s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.410860155s FAIL (2025-10-20 17:38:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.41088948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.204370969s FAIL (2025-10-20 17:38:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.204405816s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.771972835s FAIL (2025-10-20 17:37:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771998524s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.174546039s FAIL (2025-10-20 17:37:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.174580275s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.75620814s FAIL (2025-10-20 17:36:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.756235261s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.116239538s FAIL (2025-10-20 17:36:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.116288229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.415544434s FAIL (2025-10-20 17:35:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.415581375s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.101243986s FAIL (2025-10-20 17:35:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.10127213s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.549618507s FAIL (2025-10-20 17:35:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.549653121s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.294920365s FAIL (2025-10-20 17:34:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.294960842s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.458072246s FAIL (2025-10-20 17:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.458093768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.149692471s FAIL (2025-10-20 17:33:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.149727027s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.140253179s FAIL (2025-10-20 17:33:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.140286522s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.11123311s FAIL (2025-10-20 17:32:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.11126454s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.101772676s FAIL (2025-10-20 17:32:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.101807782s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.402738852s FAIL (2025-10-20 17:32:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.402787293s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.103138206s FAIL (2025-10-20 17:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.103174555s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.367721851s FAIL (2025-10-20 17:31:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.367752128s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.091983312s FAIL (2025-10-20 17:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.092012868s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.447565622s FAIL (2025-10-20 17:30:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.44760051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.317378592s FAIL (2025-10-20 17:30:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.317420493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.148141799s FAIL (2025-10-20 17:29:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.148171475s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.058256073s FAIL (2025-10-20 17:28:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.058303492s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.331217741s FAIL (2025-10-20 17:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.331246766s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.278895182s FAIL (2025-10-20 17:28:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e starting upload Oct 20 17:07:22.290572384 (651h36m7.752664867s) [upload=573243]: Uploading segments... Oct 20 17:07:22.290611870 (651h36m7.752704353s) [upload=573243]: Got next segment. Inline: false Oct 20 17:07:22.290645023 (651h36m7.752737507s) [upload=573243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 20 17:07:22.290651015 (651h36m7.752743498s) [upload=573243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 20 17:07:23.663328653 (651h36m9.125421166s) [seg_pos={0 0}, upload=573243]: Begin upload segment... Oct 20 17:07:23.663406633 (651h36m9.125499106s) [seg_pos={0 0}, upload=573243]: Done begin upload segment. Oct 20 17:07:23.663442671 (651h36m9.125535144s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664168974 (651h36m9.126261447s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.664669607 (651h36m9.12676208s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665187343 (651h36m9.127279826s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665511901 (651h36m9.127604374s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.665899328 (651h36m9.127991801s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666282578 (651h36m9.128375051s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.666636282 (651h36m9.128728755s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667140843 (651h36m9.129233316s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.667918994 (651h36m9.130011477s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668549374 (651h36m9.130641858s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.668950689 (651h36m9.131043172s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669455600 (651h36m9.131548083s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.669546133 (651h36m9.131638616s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670135064 (651h36m9.132227537s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:23.670524335 (651h36m9.132616818s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Uploading piece... Oct 20 17:07:39.081587577 (651h36m24.54368006s) [upload=573243]: Next returned nil segment Oct 20 17:07:39.081611963 (651h36m24.543704437s) [upload=573243]: Waiting for error group managing segments... Oct 20 17:07:41.933983393 (651h36m27.396075876s) [noise=true , address=77.90.31.14:4662 , piece_id=MGTMNXXDRMJVWMGAPGZ7TITNKV34GIJNNCNF6HELMG2QWISADMMA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:46.418908042 (651h36m31.881000525s) [noise=true , address=77.90.31.21:4879 , piece_id=NE32VKU5Y5WKDCN5Q4Z4426MYFTSW2JD65G52WHQWSH5AQPXJOOA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:47.945682340 (651h36m33.407774813s) [noise=true , address=145.79.227.19:4077 , piece_id=2HLR5JFCMCE3TVP4JSWB6SDC2H7BHP7W42XHJEILAZGILDBEQ6PA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:48.081389863 (651h36m33.543482347s) [noise=true , address=145.79.227.6:4257 , piece_id=EMQGGIF2NCJ746YZ4D5FZRFFVZLI6JNUNNJAIKVOY3DTAELSRP3Q, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.014359976 (651h36m34.476452459s) [noise=true , address=145.79.227.14:4840 , piece_id=SUSZYHF7CJ7QEJD4O5A4OMSOHRFH55PXJJT2FTZQV3CUGJCTIUFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.553746602 (651h36m35.015839085s) [noise=true , address=77.90.31.17:4771 , piece_id=M6QPWFXO5K3VJGF7OTIBRSJN7EV47VRHBKFL5DMPFADQHYCO7U4A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.645207982 (651h36m35.107300455s) [noise=true , address=77.90.31.16:4735 , piece_id=ZYMRZMPD6BRQ6YBB7UBKSV6JWNAZNWIOZYXZQE5MM7IJDKMPP4WQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:49.962342927 (651h36m35.4244354s) [noise=true , address=145.79.227.11:4731 , piece_id=YE6RICRM4KSUSCTL7F2RX3FMHSFRS2ELD7QKACEB3Q5DW6K5TS6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.102101054 (651h36m35.564193527s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYGWQNJ6ISGSU2BASHWWAHZYWZ2D5US7EQT5AVSFL552XREVWKRQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:50.656974057 (651h36m36.11906654s) [noise=true , address=45.91.116.4:4938 , piece_id=GVCINZLOFHWH46PW3YRZBZKJHIXOPAEM5B5LMVRX2256L225UWAQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: <nil> Oct 20 17:07:51.421289780 (651h36m36.883382263s) [noise=true , address=45.91.116.11:4119 , piece_id=CUCKNF4W7UAYA4MZVLGKPALFIYXAIAE5LEJ437GFLP6V2CANMXPA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421341999 (651h36m36.883434482s) [noise=true , address=45.91.116.10:4082 , piece_id=7SIT5DWSJ7XCSCPIUJ3ZLXUZB5G2W455JVIFMLXOQOL73BV34A6A, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->45.91.116.10:4082: use of closed network connection Oct 20 17:07:51.421412643 (651h36m36.883505116s) [noise=true , address=77.90.31.18:4807 , piece_id=GU6WYQZP6A52MXJQJ4WRK25X5RJWVM3UR3GAH2QA74AHQ43CL6DA, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: expected piece hash; context deadline exceeded; EOF Oct 20 17:07:51.421447399 (651h36m36.883539883s) [noise=true , address=145.79.227.18:4005 , piece_id=GCKQ2JVL7UY7SVCQVI2ZOUHOOC3WEWXIWOXYRXHMLMI7AZLXOJZQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32922->145.79.227.18:4005: use of closed network connection Oct 20 17:07:51.421618105 (651h36m36.883710588s) [noise=true , address=77.90.31.16:4734 , piece_id=MKUE6LYTA24S6QO3GF7ETNDYGYKGJGECVIRQPXLXC4XNA3ITG5BQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51468->77.90.31.16:4734: use of closed network connection Oct 20 17:07:51.421657980 (651h36m36.883750453s) [noise=true , address=77.90.31.10:4517 , piece_id=DRQWHUW5MZHELNFO3SMFZSYFZGFP7EU52XYEPVZGHKGPTEAFCHFQ, seg_pos={0 0}, upload=573243]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33856->77.90.31.10:4517: use of closed network connection Oct 20 17:07:51.421709749 (651h36m36.883802233s) [seg_pos={0 0}, upload=573243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 20 17:07:51.670217656 (651h36m37.132310159s) [upload=573243]: Done uploading segments... Oct 20 17:07:51.670300122 (651h36m37.132392615s) [upload=573243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.27893051s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.239533652s FAIL (2025-10-20 17:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.239567756s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.431827452s FAIL (2025-10-20 17:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.43185708s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)