Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.842246384s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:22:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.842288054s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.9631952s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:21:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.96322134s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.775856201s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:21:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.775884725s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............................X.......X........XX.X..X........] duration: 3.50313928s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:21:30 UTC) orbiter warning: context deadline exceeded Duration: 3.503157965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.725547763s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:21:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.725575336s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.850799879s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:20:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.85082691s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 12.959553751s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:20:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.959617622s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 19.149351068s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:20:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.149375404s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............................X.......X........XX.X..X.......] duration: 3.424750353s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:20:28 UTC) orbiter warning: context deadline exceeded Duration: 3.424783385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.985968483s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:20:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.985994603s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.157450078s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:19:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.157478932s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.872429735s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.872450164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............................X.......X........XX.X..X......] duration: 3.439325605s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:19:31 UTC) orbiter warning: context deadline exceeded Duration: 3.439358969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.83147484s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.831502604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................X.......X........XX.X..X.....] duration: 3.439182824s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:18:56 UTC) orbiter warning: context deadline exceeded Duration: 3.439211409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.848146035s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:18:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.848178146s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.915410451s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:18:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.91544733s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................................X.......X........XX.X..X....] duration: 3.675806891s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:18:04 UTC) orbiter warning: context deadline exceeded Duration: 3.675865204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.803483973s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:17:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.803509242s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.823856682s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:17:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.824200669s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................................X.......X........XX.X..X...] duration: 3.386893575s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:17:21 UTC) orbiter warning: context deadline exceeded Duration: 3.386920545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 11.135533857s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:17:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.135674424s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.738473731s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.738505603s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................................X.......X........XX.X..X..] duration: 3.577157997s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:16:27 UTC) orbiter warning: context deadline exceeded Duration: 3.577184166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 12.341948078s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:16:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.341965772s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.818337053s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:15:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.818363353s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 19.257367244s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:15:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.257397772s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................................X.......X........XX.X..X.] duration: 3.455797733s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:15:52 UTC) orbiter warning: context deadline exceeded Duration: 3.455823643s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................................X.......X........XX.X..X] duration: 5.001146217s trace: https://jaeger.dev.storj.io/trace/2453b61de7a1b1d6 FAIL (2025-11-30 14:15:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001164792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.990752944s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:14:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.990783184s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................................X.......X........XX.X..] duration: 4.660366376s trace: https://jaeger.dev.storj.io/trace/308484a6249eb0f4 FAIL (2025-11-30 14:14:53 UTC) orbiter warning: context deadline exceeded Duration: 4.660398026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 12.983717631s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:14:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.983766835s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................................X.......X........XX.X.] duration: 3.52145249s trace: https://jaeger.dev.storj.io/trace/308484a6249eb0f4 FAIL (2025-11-30 14:14:26 UTC) orbiter warning: context deadline exceeded Duration: 3.521472559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.756532066s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:14:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.756574115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.912292101s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:13:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.912321708s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 10.598859322s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:13:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.598888087s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......................................X.......X........XX.X] duration: 5.001684035s trace: https://jaeger.dev.storj.io/trace/308484a6249eb0f4 FAIL (2025-11-30 14:13:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001710737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 13.705407624s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:12:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.705430828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.936219478s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:12:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.936247631s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 19.06134952s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:12:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.061380961s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................X.......X........XX.] duration: 3.455522468s trace: https://jaeger.dev.storj.io/trace/ffa8991a8dccab2 FAIL (2025-11-30 14:12:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455549609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................................X.......X........XX] duration: 5.000823515s trace: https://jaeger.dev.storj.io/trace/ffa8991a8dccab2 FAIL (2025-11-30 14:12:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000853342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.090162561s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:11:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.090215212s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00115586s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:11:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001181407s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................X.......X........X] duration: 5.001497843s trace: https://jaeger.dev.storj.io/trace/4bf1e019377c4efd FAIL (2025-11-30 14:11:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001510026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.741401426s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.7414302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.83008085s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:11:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.830110607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.82893295s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:10:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.828982083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..........................................X.......X........] duration: 3.580743542s trace: https://jaeger.dev.storj.io/trace/71a223f8d367c9d8 FAIL (2025-11-30 14:10:22 UTC) orbiter warning: context deadline exceeded Duration: 3.580774122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.899537747s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:09:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.899567154s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)