Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...XXX.....X.....X.XX....X...X.X.........X.................] duration: 5.819861457s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:44:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.819887646s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X...............X...............................] duration: 3.692860542s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:43:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.692886773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X...XXX.....X.....X.XX....X...X.X.........X................] duration: 5.944205487s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.944235445s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X...............X..............................] duration: 3.673965405s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:43:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673987617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X...............X.............................] duration: 3.409436991s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:43:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.409459674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X...XXX.....X.....X.XX....X...X.X.........X...............] duration: 6.796434976s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:43:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.796465844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X...............X............................] duration: 3.360250933s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:42:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360272905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X...XXX.....X.....X.XX....X...X.X.........X..............] duration: 5.816253177s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:41:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.816283445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X...XXX.....X.....X.XX....X...X.X.........X.............] duration: 7.184681158s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 7.184707357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X...............X...........................] duration: 3.780515262s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:41:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780537194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X...XXX.....X.....X.XX....X...X.X.........X............] duration: 5.842559286s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.842589612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X...............X..........................] duration: 3.422132295s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:40:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422169015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X...............X.........................] duration: 3.549209954s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:39:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549257182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X...XXX.....X.....X.XX....X...X.X.........X...........] duration: 5.75394848s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:39:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.753973076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X...XXX.....X.....X.XX....X...X.X.........X..........] duration: 6.293867386s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.293901331s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X...............X........................] duration: 4.503740817s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:38:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.503769193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X...............X.......................] duration: 3.574399404s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:38:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57461789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X...XXX.....X.....X.XX....X...X.X.........X.........] duration: 6.03382645s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:38:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.033856287s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X...............X......................] duration: 3.649661276s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:37:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.649692467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X...XXX.....X.....X.XX....X...X.X.........X........] duration: 7.445881475s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:37:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 7.445907334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X...XXX.....X.....X.XX....X...X.X.........X.......] duration: 6.177986115s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.178017473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X...XXX.....X.....X.XX....X...X.X.........X......] duration: 6.182110908s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:36:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.182144562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X...XXX.....X.....X.XX....X...X.X.........X.....] duration: 5.78886757s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.788896273s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X...............X.....................] duration: 3.558662107s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:36:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558692186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X...............X....................] duration: 3.628977269s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:35:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629025732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X...XXX.....X.....X.XX....X...X.X.........X....] duration: 6.184330076s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:35:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.184376544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............X...XXX.....X.....X.XX....X...X.X.........X...] duration: 5.919670136s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.919703089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X...............X...................] duration: 3.625408826s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:34:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625436089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............X...XXX.....X.....X.XX....X...X.X.........X..] duration: 5.853927845s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:34:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 5.853963903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................X...XXX.....X.....X.XX....X...X.X.........X.] duration: 6.057026216s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:34:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 6.057056164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X...............X..................] duration: 3.425661276s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:33:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425685461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................X...XXX.....X.....X.XX....X...X.X.........X] duration: 30.003147743s trace: https://jaeger.dev.storj.io/trace/47f23d30b4c5c0b FAIL (2026-01-07 20:33:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _ObjectBegin Jan 7 20:32:47.264392726 (797h3m47.135475761s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:32:47.510204882 (797h3m47.381287948s) [seg_pos={0 0}, upload=697136]: Begin upload segment... Jan 7 20:32:47.510269766 (797h3m47.381352801s) [seg_pos={0 0}, upload=697136]: Done begin upload segment. Jan 7 20:32:47.510316304 (797h3m47.38139934s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510324631 (797h3m47.381407666s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.510476540 (797h3m47.381559585s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511058208 (797h3m47.382141243s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511454131 (797h3m47.382537166s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.511789490 (797h3m47.382872525s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512084622 (797h3m47.383167667s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512234347 (797h3m47.383317422s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512372650 (797h3m47.383455685s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512636433 (797h3m47.383719468s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512703641 (797h3m47.383786676s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.512917608 (797h3m47.384000643s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513395428 (797h3m47.384478473s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513677756 (797h3m47.384760801s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.513999668 (797h3m47.385082703s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:47.514321080 (797h3m47.385404115s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Uploading piece... Jan 7 20:32:49.214835566 (797h3m49.085918611s) [upload=697136]: Next returned nil segment Jan 7 20:32:49.214848721 (797h3m49.085931756s) [upload=697136]: Waiting for error group managing segments... Jan 7 20:32:49.503314115 (797h3m49.37439718s) [noise=true , address=145.79.227.17:4972 , piece_id=476ZNUADRWEWBCYSTZ2YTVUCMGYQOELH77WS6EFMQCWLN3P2SYHQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.504178039 (797h3m49.375261084s) [noise=true , address=145.79.227.5:4221 , piece_id=ENNCEHKLPBPYMJTUEMCNRNDBMFPSEMLLB5IGFBET2QKE3GHNJWWQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.509096434 (797h3m49.38017947s) [noise=true , address=145.79.227.16:4909 , piece_id=HPBJT2COKIE7LM6VL2UPA5TNJXTFSHYHFE4ASPYWFGXDBBYLLQSA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.545514290 (797h3m49.416597325s) [noise=true , address=145.79.227.43:4158 , piece_id=3UHLK7DETMN6CGETHEO5GGD3W2KDA3TVL6NUABOSOCQCTZXCKA5A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.594197481 (797h3m49.465280536s) [noise=true , address=145.79.227.33:4869 , piece_id=JSHURCUMILFCAPGHM6VNEKSIXHOWSXZOHRBXGNESXYJPP22VMS3Q, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.598082398 (797h3m49.469165443s) [noise=true , address=145.79.227.15:4876 , piece_id=3CS32UQFVF2QBT3RBGQIRYEMGGBYZVYPFNRTCOLIQG7ETVKLVMAQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.626462168 (797h3m49.497545203s) [noise=true , address=145.79.227.52:4447 , piece_id=S2R7ZVCOOZXUXEYCYPCSQWY4ZI7ZHETXRPORT27KPUHEG3LEO5OA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.634227063 (797h3m49.505310098s) [noise=true , address=145.79.227.9:4367 , piece_id=YINRMOQ7JMG2X27F3YFR4G6F42UVQPSWVDFBLN55KAGXHDPRGYGQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.714235121 (797h3m49.585318156s) [noise=true , address=145.79.227.11:4729 , piece_id=GHLPXY4SZVEI2UFXGDK3WIMQ7QAUNYRWLAWTAEHYHP5CDBRQVHHA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.870677512 (797h3m49.741760547s) [noise=true , address=77.90.31.11:4554 , piece_id=V4MIFRIWYMQKP7S6HH7WIJKR3PUYJZ6MRLSH6HJ2G3X34TG72OFA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:49.932690023 (797h3m49.803773058s) [noise=true , address=45.91.116.11:4118 , piece_id=IVXOJJH3LUQLGYWUWSNFP2WITNRQS74ZKJAPRQEUQNERZV6TNYBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.123060429 (797h3m49.994143464s) [noise=true , address=145.79.227.27:4691 , piece_id=C6MOSMKVHFHM4C2HU2LCLTGWVNIENWW3I3VGFLT7WZGW5G4BXUNQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236423189 (797h3m50.107506224s) [noise=true , address=145.79.227.47:4266 , piece_id=AHTXF273ZHA5NISQKMYN2E5YBDPPWXXMWTX4UCONFPFAEPBZYF3A, seg_pos={0 0}, upload=697136]: Done uploading piece. err: <nil> Jan 7 20:32:50.236444269 (797h3m50.107527304s) [seg_pos={0 0}, upload=697136]: Segment reached optimal threshold of 13 pieces. Jan 7 20:32:50.236637267 (797h3m50.107720302s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35524->145.79.227.21:4475: use of closed network connection Jan 7 20:32:50.236643479 (797h3m50.107726524s) [noise=true , address=145.79.227.21:4475 , piece_id=TO66QAZXFC54EMF6N72ARAPDENHQHARH6WIF4DXWD4FI3O4S4MBQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236851314 (797h3m50.107934349s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->145.79.227.22:4510: use of closed network connection Jan 7 20:32:50.236886571 (797h3m50.107969607s) [noise=true , address=145.79.227.22:4510 , piece_id=ZDRABEIAZGDQ6UFWW7XL2MWA6ZQ6JAUINEFRX3HR2TBQVHWAYVCQ, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236943270 (797h3m50.108026345s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34922->145.79.227.41:4086: use of closed network connection Jan 7 20:32:50.236954561 (797h3m50.108037616s) [noise=true , address=145.79.227.41:4086 , piece_id=DATWV33MSESFXTYTAYWDRL4RXJ4VA5HPBBNYKY2ANSDQKW33UZOA, seg_pos={0 0}, upload=697136]: Long tail cancellation due to optimal threshold Jan 7 20:32:50.236989698 (797h3m50.108072743s) [seg_pos={0 0}, upload=697136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:32:50.237074831 (797h3m50.108157866s) [upload=697136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:32:50.237081613 (797h3m50.108164648s) [upload=697136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:32:50.535795140 (797h3m50.406878175s) [upload=697136]: Done uploading segments... Jan 7 20:32:50.535817193 (797h3m50.406900228s) [upload=697136]: Upload finished. err: <nil> Duration:* 30.003174665s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 253.260226ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae FAIL (2026-01-07 20:32:58 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 253.292238ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 54.371256698s remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/521974c405507e8c FAIL (2026-01-07 20:32:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 54.371286043s edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X...............X.................] duration: 3.572093304s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:32:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572120394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................X...XXX.....X.....X.XX....X...X.X.........] duration: 9.326325333s trace: https://jaeger.dev.storj.io/trace/cd6e308e65ee284 FAIL (2026-01-07 20:32:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ) context deadline exceeded ======LOGS====== (796h56m10.740637397s) [upload=697024]: Done starting upload Jan 7 20:25:10.869631399 (796h56m10.740714444s) [upload=697024]: Uploading segments... Jan 7 20:25:10.869846669 (796h56m10.740929714s) [upload=697024]: Got next segment. Inline: false Jan 7 20:25:10.869887407 (796h56m10.740970442s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 7 20:25:10.869892266 (796h56m10.740975301s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:25:11.157301280 (796h56m11.028384325s) [seg_pos={0 0}, upload=697024]: Begin upload segment... Jan 7 20:25:11.157366845 (796h56m11.02844989s) [seg_pos={0 0}, upload=697024]: Done begin upload segment. Jan 7 20:25:11.157413022 (796h56m11.028496058s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.157472105 (796h56m11.02855514s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158476869 (796h56m11.029559944s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158614791 (796h56m11.029697836s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159113280 (796h56m11.030196325s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159488765 (796h56m11.03057181s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159850624 (796h56m11.030933659s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160187897 (796h56m11.031270942s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160556718 (796h56m11.031639753s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160849156 (796h56m11.031932201s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161149938 (796h56m11.032232983s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161579036 (796h56m11.032662081s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161983535 (796h56m11.03306657s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162589720 (796h56m11.033672765s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162972349 (796h56m11.034055394s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.163313028 (796h56m11.034396073s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:12.867076208 (796h56m12.738159243s) [upload=697024]: Next returned nil segment Jan 7 20:25:12.867093641 (796h56m12.738176686s) [upload=697024]: Waiting for error group managing segments... Jan 7 20:25:13.149062194 (796h56m13.020145229s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155055716 (796h56m13.026138751s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155201694 (796h56m13.026284729s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.158713822 (796h56m13.029796867s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.170052662 (796h56m13.041135697s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.178076031 (796h56m13.049159066s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.179479633 (796h56m13.050562668s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.181930329 (796h56m13.053013364s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.228955643 (796h56m13.100038688s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.231649403 (796h56m13.102732438s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.242411746 (796h56m13.113494781s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.523728877 (796h56m13.394811922s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587694969 (796h56m13.458778014s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587715758 (796h56m13.458798793s) [seg_pos={0 0}, upload=697024]: Segment reached optimal threshold of 13 pieces. Jan 7 20:25:13.587981194 (796h56m13.459064229s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:57754->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 7 20:25:13.587990711 (796h56m13.459073746s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.587994198 (796h56m13.459077263s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection Jan 7 20:25:13.588004779 (796h56m13.459087824s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588155796 (796h56m13.459238851s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jan 7 20:25:13.588165053 (796h56m13.459248098s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588215500 (796h56m13.459298545s) [seg_pos={0 0}, upload=697024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:25:13.588306032 (796h56m13.459389077s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:25:13.588313427 (796h56m13.459396472s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:25:13.883078035 (796h56m13.75416108s) [upload=697024]: Done uploading segments... Jan 7 20:25:13.883104524 (796h56m13.754187559s) [upload=697024]: Upload finished. err: <nil> Duration: 9.326352275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..................X...XXX.....X.....X.XX....X...X.X........] duration: 6.147295964s trace: https://jaeger.dev.storj.io/trace/cd6e308e65ee284 FAIL (2026-01-07 20:31:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ) context deadline exceeded ======LOGS====== (796h56m10.740637397s) [upload=697024]: Done starting upload Jan 7 20:25:10.869631399 (796h56m10.740714444s) [upload=697024]: Uploading segments... Jan 7 20:25:10.869846669 (796h56m10.740929714s) [upload=697024]: Got next segment. Inline: false Jan 7 20:25:10.869887407 (796h56m10.740970442s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 7 20:25:10.869892266 (796h56m10.740975301s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:25:11.157301280 (796h56m11.028384325s) [seg_pos={0 0}, upload=697024]: Begin upload segment... Jan 7 20:25:11.157366845 (796h56m11.02844989s) [seg_pos={0 0}, upload=697024]: Done begin upload segment. Jan 7 20:25:11.157413022 (796h56m11.028496058s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.157472105 (796h56m11.02855514s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158476869 (796h56m11.029559944s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158614791 (796h56m11.029697836s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159113280 (796h56m11.030196325s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159488765 (796h56m11.03057181s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159850624 (796h56m11.030933659s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160187897 (796h56m11.031270942s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160556718 (796h56m11.031639753s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160849156 (796h56m11.031932201s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161149938 (796h56m11.032232983s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161579036 (796h56m11.032662081s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161983535 (796h56m11.03306657s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162589720 (796h56m11.033672765s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162972349 (796h56m11.034055394s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.163313028 (796h56m11.034396073s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:12.867076208 (796h56m12.738159243s) [upload=697024]: Next returned nil segment Jan 7 20:25:12.867093641 (796h56m12.738176686s) [upload=697024]: Waiting for error group managing segments... Jan 7 20:25:13.149062194 (796h56m13.020145229s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155055716 (796h56m13.026138751s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155201694 (796h56m13.026284729s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.158713822 (796h56m13.029796867s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.170052662 (796h56m13.041135697s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.178076031 (796h56m13.049159066s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.179479633 (796h56m13.050562668s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.181930329 (796h56m13.053013364s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.228955643 (796h56m13.100038688s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.231649403 (796h56m13.102732438s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.242411746 (796h56m13.113494781s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.523728877 (796h56m13.394811922s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587694969 (796h56m13.458778014s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587715758 (796h56m13.458798793s) [seg_pos={0 0}, upload=697024]: Segment reached optimal threshold of 13 pieces. Jan 7 20:25:13.587981194 (796h56m13.459064229s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:57754->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 7 20:25:13.587990711 (796h56m13.459073746s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.587994198 (796h56m13.459077263s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection Jan 7 20:25:13.588004779 (796h56m13.459087824s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588155796 (796h56m13.459238851s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jan 7 20:25:13.588165053 (796h56m13.459248098s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588215500 (796h56m13.459298545s) [seg_pos={0 0}, upload=697024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:25:13.588306032 (796h56m13.459389077s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:25:13.588313427 (796h56m13.459396472s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:25:13.883078035 (796h56m13.75416108s) [upload=697024]: Done uploading segments... Jan 7 20:25:13.883104524 (796h56m13.754187559s) [upload=697024]: Upload finished. err: <nil> Duration: 6.147317915s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 266.403986ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae FAIL (2026-01-07 20:31:59 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 266.442749ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..................X...XXX.....X.....X.XX....X...X.X.......] duration: 14.731810227s trace: https://jaeger.dev.storj.io/trace/cd6e308e65ee284 FAIL (2026-01-07 20:31:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ) context deadline exceeded ======LOGS====== (796h56m10.740637397s) [upload=697024]: Done starting upload Jan 7 20:25:10.869631399 (796h56m10.740714444s) [upload=697024]: Uploading segments... Jan 7 20:25:10.869846669 (796h56m10.740929714s) [upload=697024]: Got next segment. Inline: false Jan 7 20:25:10.869887407 (796h56m10.740970442s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 7 20:25:10.869892266 (796h56m10.740975301s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:25:11.157301280 (796h56m11.028384325s) [seg_pos={0 0}, upload=697024]: Begin upload segment... Jan 7 20:25:11.157366845 (796h56m11.02844989s) [seg_pos={0 0}, upload=697024]: Done begin upload segment. Jan 7 20:25:11.157413022 (796h56m11.028496058s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.157472105 (796h56m11.02855514s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158476869 (796h56m11.029559944s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158614791 (796h56m11.029697836s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159113280 (796h56m11.030196325s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159488765 (796h56m11.03057181s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159850624 (796h56m11.030933659s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160187897 (796h56m11.031270942s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160556718 (796h56m11.031639753s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160849156 (796h56m11.031932201s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161149938 (796h56m11.032232983s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161579036 (796h56m11.032662081s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161983535 (796h56m11.03306657s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162589720 (796h56m11.033672765s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162972349 (796h56m11.034055394s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.163313028 (796h56m11.034396073s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:12.867076208 (796h56m12.738159243s) [upload=697024]: Next returned nil segment Jan 7 20:25:12.867093641 (796h56m12.738176686s) [upload=697024]: Waiting for error group managing segments... Jan 7 20:25:13.149062194 (796h56m13.020145229s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155055716 (796h56m13.026138751s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155201694 (796h56m13.026284729s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.158713822 (796h56m13.029796867s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.170052662 (796h56m13.041135697s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.178076031 (796h56m13.049159066s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.179479633 (796h56m13.050562668s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.181930329 (796h56m13.053013364s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.228955643 (796h56m13.100038688s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.231649403 (796h56m13.102732438s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.242411746 (796h56m13.113494781s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.523728877 (796h56m13.394811922s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587694969 (796h56m13.458778014s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587715758 (796h56m13.458798793s) [seg_pos={0 0}, upload=697024]: Segment reached optimal threshold of 13 pieces. Jan 7 20:25:13.587981194 (796h56m13.459064229s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:57754->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 7 20:25:13.587990711 (796h56m13.459073746s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.587994198 (796h56m13.459077263s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection Jan 7 20:25:13.588004779 (796h56m13.459087824s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588155796 (796h56m13.459238851s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jan 7 20:25:13.588165053 (796h56m13.459248098s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588215500 (796h56m13.459298545s) [seg_pos={0 0}, upload=697024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:25:13.588306032 (796h56m13.459389077s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:25:13.588313427 (796h56m13.459396472s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:25:13.883078035 (796h56m13.75416108s) [upload=697024]: Done uploading segments... Jan 7 20:25:13.883104524 (796h56m13.754187559s) [upload=697024]: Upload finished. err: <nil> Duration: 14.731841887s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X...............X................] duration: 3.492123958s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:31:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492149527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X...............X...............] duration: 3.537916784s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:31:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537945468s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 278.920156ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae FAIL (2026-01-07 20:31:02 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 278.950595ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X...............X..............] duration: 3.547161461s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:30:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547500144s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 171.374655ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/521974c405507e8c FAIL (2026-01-07 20:30:50 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 171.402568ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X..................X...XXX.....X.....X.XX....X...X.X......] duration: 6.138344105s trace: https://jaeger.dev.storj.io/trace/cd6e308e65ee284 FAIL (2026-01-07 20:30:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ) context deadline exceeded ======LOGS====== (796h56m10.740637397s) [upload=697024]: Done starting upload Jan 7 20:25:10.869631399 (796h56m10.740714444s) [upload=697024]: Uploading segments... Jan 7 20:25:10.869846669 (796h56m10.740929714s) [upload=697024]: Got next segment. Inline: false Jan 7 20:25:10.869887407 (796h56m10.740970442s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 7 20:25:10.869892266 (796h56m10.740975301s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:25:11.157301280 (796h56m11.028384325s) [seg_pos={0 0}, upload=697024]: Begin upload segment... Jan 7 20:25:11.157366845 (796h56m11.02844989s) [seg_pos={0 0}, upload=697024]: Done begin upload segment. Jan 7 20:25:11.157413022 (796h56m11.028496058s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.157472105 (796h56m11.02855514s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158476869 (796h56m11.029559944s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158614791 (796h56m11.029697836s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159113280 (796h56m11.030196325s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159488765 (796h56m11.03057181s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159850624 (796h56m11.030933659s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160187897 (796h56m11.031270942s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160556718 (796h56m11.031639753s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160849156 (796h56m11.031932201s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161149938 (796h56m11.032232983s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161579036 (796h56m11.032662081s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161983535 (796h56m11.03306657s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162589720 (796h56m11.033672765s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162972349 (796h56m11.034055394s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.163313028 (796h56m11.034396073s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:12.867076208 (796h56m12.738159243s) [upload=697024]: Next returned nil segment Jan 7 20:25:12.867093641 (796h56m12.738176686s) [upload=697024]: Waiting for error group managing segments... Jan 7 20:25:13.149062194 (796h56m13.020145229s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155055716 (796h56m13.026138751s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155201694 (796h56m13.026284729s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.158713822 (796h56m13.029796867s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.170052662 (796h56m13.041135697s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.178076031 (796h56m13.049159066s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.179479633 (796h56m13.050562668s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.181930329 (796h56m13.053013364s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.228955643 (796h56m13.100038688s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.231649403 (796h56m13.102732438s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.242411746 (796h56m13.113494781s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.523728877 (796h56m13.394811922s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587694969 (796h56m13.458778014s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587715758 (796h56m13.458798793s) [seg_pos={0 0}, upload=697024]: Segment reached optimal threshold of 13 pieces. Jan 7 20:25:13.587981194 (796h56m13.459064229s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:57754->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 7 20:25:13.587990711 (796h56m13.459073746s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.587994198 (796h56m13.459077263s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection Jan 7 20:25:13.588004779 (796h56m13.459087824s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588155796 (796h56m13.459238851s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jan 7 20:25:13.588165053 (796h56m13.459248098s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588215500 (796h56m13.459298545s) [seg_pos={0 0}, upload=697024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:25:13.588306032 (796h56m13.459389077s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:25:13.588313427 (796h56m13.459396472s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:25:13.883078035 (796h56m13.75416108s) [upload=697024]: Done uploading segments... Jan 7 20:25:13.883104524 (796h56m13.754187559s) [upload=697024]: Upload finished. err: <nil> Duration: 6.138369284s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 271.546868ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae FAIL (2026-01-07 20:30:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 271.57487ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X...............X.............] duration: 3.59414874s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:30:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594170772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X...............X............] duration: 4.567652221s trace: https://jaeger.dev.storj.io/trace/1bc89d1a74533810 FAIL (2026-01-07 20:30:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.567673012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X..................X...XXX.....X.....X.XX....X...X.X.....] duration: 6.315948262s trace: https://jaeger.dev.storj.io/trace/cd6e308e65ee284 FAIL (2026-01-07 20:30:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ) context deadline exceeded ======LOGS====== (796h56m10.740637397s) [upload=697024]: Done starting upload Jan 7 20:25:10.869631399 (796h56m10.740714444s) [upload=697024]: Uploading segments... Jan 7 20:25:10.869846669 (796h56m10.740929714s) [upload=697024]: Got next segment. Inline: false Jan 7 20:25:10.869887407 (796h56m10.740970442s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 7 20:25:10.869892266 (796h56m10.740975301s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 7 20:25:11.157301280 (796h56m11.028384325s) [seg_pos={0 0}, upload=697024]: Begin upload segment... Jan 7 20:25:11.157366845 (796h56m11.02844989s) [seg_pos={0 0}, upload=697024]: Done begin upload segment. Jan 7 20:25:11.157413022 (796h56m11.028496058s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.157472105 (796h56m11.02855514s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158476869 (796h56m11.029559944s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.158614791 (796h56m11.029697836s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159113280 (796h56m11.030196325s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159488765 (796h56m11.03057181s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.159850624 (796h56m11.030933659s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160187897 (796h56m11.031270942s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160556718 (796h56m11.031639753s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.160849156 (796h56m11.031932201s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161149938 (796h56m11.032232983s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161579036 (796h56m11.032662081s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.161983535 (796h56m11.03306657s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162589720 (796h56m11.033672765s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.162972349 (796h56m11.034055394s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:11.163313028 (796h56m11.034396073s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Uploading piece... Jan 7 20:25:12.867076208 (796h56m12.738159243s) [upload=697024]: Next returned nil segment Jan 7 20:25:12.867093641 (796h56m12.738176686s) [upload=697024]: Waiting for error group managing segments... Jan 7 20:25:13.149062194 (796h56m13.020145229s) [noise=true , address=145.79.227.24:4583 , piece_id=W7VYZS2DFSZP2CI22Y7MUQUMTDQAEJUALEKFUMONRIXR3Z3R6KJA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155055716 (796h56m13.026138751s) [noise=true , address=145.79.227.14:4838 , piece_id=XRVGLKD2YDSDOWRLA56FC4AK2DX63NXXVBQYIJZVNEAXZVOJGEUA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.155201694 (796h56m13.026284729s) [noise=true , address=145.79.227.13:4804 , piece_id=OPPHBI3EGEYINYE6BODJ4HOUTKDEFTCVLNK7ACK7HK5E3RGF7M6A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.158713822 (796h56m13.029796867s) [noise=true , address=145.79.227.40:4049 , piece_id=UXHKS3W2Y3QIY6BMNJ4JVEO3D6O2YKERVPNA23OGT2WKFEPY5JHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.170052662 (796h56m13.041135697s) [noise=true , address=145.79.227.6:4259 , piece_id=DOHSGZPSRM6KE3WQS6W4WAKTSP4LRVUITGHOZT6BVKVJMUNLRS2Q, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.178076031 (796h56m13.049159066s) [noise=true , address=145.79.227.47:4265 , piece_id=M5227EEIGEDG64YYRIZ6C4RV3V32XULQ5QYNLZSXDUUNKW33UC3A, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.179479633 (796h56m13.050562668s) [noise=true , address=145.79.227.41:4087 , piece_id=A6CAE6W3YG3HIHPMUVSZ6FF7IF4IAR5PQV7TUTZSNPNWHATZMDWA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.181930329 (796h56m13.053013364s) [noise=true , address=145.79.227.48:4302 , piece_id=G5VHSASS2DPI4HH62JAPO6FGK6RTNY7MT6UN6VWLDS533D4NNBTA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.228955643 (796h56m13.100038688s) [noise=true , address=145.79.227.44:4193 , piece_id=O5JSNX7RR6OHYE7WXJKGPCISUAO4QVJ76CFRQP2AHLQGRNWCKNHA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.231649403 (796h56m13.102732438s) [noise=true , address=145.79.227.26:4653 , piece_id=3XOMCWDKA22ZEBAG2B27TURJYW4C6T7JIANKV44AF3ALCIA2GLGA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.242411746 (796h56m13.113494781s) [noise=true , address=145.79.227.20:4438 , piece_id=BBDKGYIFYF4X4AVGLR5JOASD2LYELO6AGMRNKFFRRR2LHPNPYFRQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.523728877 (796h56m13.394811922s) [noise=true , address=77.90.31.15:4700 , piece_id=S2XEMAAC5PEHPWFRJSA4HH2NIMOPRH6UBZOSDZOU2TLUXVOOJ4IQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587694969 (796h56m13.458778014s) [noise=true , address=45.91.116.10:4081 , piece_id=74LTLCW6YFDAJB237RE3NHI7HOCXVRIWBFOT26ZQZL6GGCCYEMOA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: <nil> Jan 7 20:25:13.587715758 (796h56m13.458798793s) [seg_pos={0 0}, upload=697024]: Segment reached optimal threshold of 13 pieces. Jan 7 20:25:13.587981194 (796h56m13.459064229s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:57754->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 7 20:25:13.587990711 (796h56m13.459073746s) [noise=true , address=45.91.116.5:4298 , piece_id=JGSQ5XLGV2DE42NK4G4U3PYMMXC3R5CM543IIRRS6OYMIEQPGNXA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.587994198 (796h56m13.459077263s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38430->45.91.116.6:4975: use of closed network connection Jan 7 20:25:13.588004779 (796h56m13.459087824s) [noise=true , address=45.91.116.6:4975 , piece_id=PRLYCHCGMWASDIJEURFYR5XMKZATXHWSCKEC6ZX64NOQR7725SGQ, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588155796 (796h56m13.459238851s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jan 7 20:25:13.588165053 (796h56m13.459248098s) [noise=true , address=77.90.31.18:4805 , piece_id=HDUMHWBUYCDANNKWQDYDJ2LVCIQOAB6V4LK6ZDYEL2YDD4SZCYEA, seg_pos={0 0}, upload=697024]: Long tail cancellation due to optimal threshold Jan 7 20:25:13.588215500 (796h56m13.459298545s) [seg_pos={0 0}, upload=697024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 7 20:25:13.588306032 (796h56m13.459389077s) [upload=697024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 7 20:25:13.588313427 (796h56m13.459396472s) [upload=697024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 7 20:25:13.883078035 (796h56m13.75416108s) [upload=697024]: Done uploading segments... Jan 7 20:25:13.883104524 (796h56m13.754187559s) [upload=697024]: Upload finished. err: <nil> Duration: 6.31600039s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 172.480259ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/521974c405507e8c FAIL (2026-01-07 20:29:56 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 172.624364ms edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)