Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.711292244s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:30:17 UTC) orbiter warning: context deadline exceeded Duration: 11.711325157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 14.121639255s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.121689891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [................XXX.XX.......X..XX......X.X...X.......X.....] duration: 3.953315011s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:29:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.953338216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 22.6080698s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 22.608101269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.................XXX.XX.......X..XX......X.X...X.......X....] duration: 3.578112591s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:29:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578163287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 15.332230602s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:29:24 UTC) orbiter warning: context deadline exceeded Duration: 15.33225577s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..................XXX.XX.......X..XX......X.X...X.......X...] duration: 5.001358149s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:29:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001390781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 12.656876781s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:29:00 UTC) orbiter warning: context deadline exceeded Duration: 12.656906179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 15.038051173s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 15.038078235s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...................XXX.XX.......X..XX......X.X...X.......X..] duration: 3.548680898s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:28:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548784145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 13.794934953s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:28:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 13.794966212s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...................XXX.XX.......X..XX......X.X...X.......X.] duration: 3.642011325s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:28:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642028158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.222646609s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:27:51 UTC) orbiter warning: context deadline exceeded Duration: 7.222691465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 14.018271795s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:27:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.018303425s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX...................XXX.XX.......X..XX......X.X...X.......X] duration: 5.00188917s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:27:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001914798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.16616056s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:27:22 UTC) orbiter warning: context deadline exceeded Duration: 7.166190326s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 14.927526415s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:26:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.92755564s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 12.201225552s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:26:35 UTC) orbiter warning: context deadline exceeded Duration: 12.201271118s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 14.332502825s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.33270413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX...................XXX.XX.......X..XX......X.X...X.......] duration: 3.446779138s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:26:28 UTC) orbiter warning: context deadline exceeded Duration: 3.447033453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 14.557872789s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:26:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.557909129s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X.XX...................XXX.XX.......X..XX......X.X...X......] duration: 3.560124467s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:26:05 UTC) orbiter warning: context deadline exceeded Duration: 3.560153333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 14.523518734s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.523550504s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 18.190152875s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:25:42 UTC) orbiter warning: context deadline exceeded Duration: 18.190193202s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX.XX...................XXX.XX.......X..XX......X.X...X.....] duration: 3.642787963s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:25:03 UTC) orbiter warning: context deadline exceeded Duration: 3.642804595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XX.XX...................XXX.XX.......X..XX......X.X...X....] duration: 3.415103783s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:24:43 UTC) orbiter warning: context deadline exceeded Duration: 3.415131766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 14.196553625s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.196590565s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.741977183s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:24:37 UTC) orbiter warning: context deadline exceeded Duration: 11.7420021s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X.XX.XX...................XXX.XX.......X..XX......X.X...X...] duration: 3.514344896s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:24:18 UTC) orbiter warning: context deadline exceeded Duration: 3.514372218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 14.282823318s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.282853495s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.903599771s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:23:35 UTC) orbiter warning: context deadline exceeded Duration: 11.903620511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 14.184513339s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.184543285s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X.XX.XX...................XXX.XX.......X..XX......X.X...X..] duration: 3.553662103s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:23:16 UTC) orbiter warning: context deadline exceeded Duration: 3.553689024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 14.331970822s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:22:47 UTC) orbiter warning: context deadline exceeded Duration: 14.331999609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 14.422852768s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.422879569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.X.XX.XX...................XXX.XX.......X..XX......X.X...X.] duration: 3.504801924s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:22:22 UTC) orbiter warning: context deadline exceeded Duration: 3.504834817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 14.455263804s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:22:12 UTC) orbiter warning: context deadline exceeded Duration: 14.455297319s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.789722934s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:21:45 UTC) orbiter warning: context deadline exceeded Duration: 21.789750417s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X.X.XX.XX...................XXX.XX.......X..XX......X.X...X] duration: 5.001933575s trace: https://jaeger.dev.storj.io/trace/1491a82f0dd8ba6f FAIL (2025-11-22 16:21:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001958271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 14.284065893s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.284096982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 14.428333324s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:20:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.428391665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X.X.XX.XX...................XXX.XX.......X..XX......X.X...] duration: 3.518119704s trace: https://jaeger.dev.storj.io/trace/65a7ba69260aef83 FAIL (2025-11-22 16:20:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518150822s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.321489043s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:20:29 UTC) orbiter warning: context deadline exceeded Duration: 7.321516635s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X.X.XX.XX...................XXX.XX.......X..XX......X.X..] duration: 3.605435137s trace: https://jaeger.dev.storj.io/trace/65a7ba69260aef83 FAIL (2025-11-22 16:19:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605506945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...........................................X...............] duration: 6.868807798s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:19:44 UTC) orbiter warning: context deadline exceeded Duration: 6.868838787s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 14.256177108s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.256201565s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...........................................X..............] duration: 12.653174933s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:19:28 UTC) orbiter warning: context deadline exceeded Duration: 12.653239686s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....X.X.XX.XX...................XXX.XX.......X..XX......X.X.] duration: 5.00145828s trace: https://jaeger.dev.storj.io/trace/65a7ba69260aef83 FAIL (2025-11-22 16:19:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001519286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 17.008690633s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 17.008724998s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...........................................X.............] duration: 16.22976807s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:18:46 UTC) orbiter warning: context deadline exceeded Duration: 16.229794019s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)