Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 14.589858545s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.589889374s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X....X.X.XX.XX...................XXX.XX.......X..XX......X.X] duration: 5.001318503s trace: https://jaeger.dev.storj.io/trace/65a7ba69260aef83 FAIL (2025-11-22 16:18:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001343601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................................X............] duration: 14.861408174s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:18:10 UTC) orbiter warning: context deadline exceeded Duration: 14.861432841s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 14.171038068s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:17:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.171058178s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XX....X.X.XX.XX...................XXX.XX.......X..XX......X.] duration: 5.001434664s trace: https://jaeger.dev.storj.io/trace/730efa44f76d1e81 FAIL (2025-11-22 16:17:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001461205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...X...........................................X...........] duration: 9.96154533s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:17:22 UTC) orbiter warning: context deadline exceeded Duration: 9.96157188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 14.123749332s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.123781834s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...X...........................................X..........] duration: 16.805814512s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:16:59 UTC) orbiter warning: context deadline exceeded Duration: 16.805845242s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.XX....X.X.XX.XX...................XXX.XX.......X..XX......X] duration: 5.001564437s trace: https://jaeger.dev.storj.io/trace/730efa44f76d1e81 FAIL (2025-11-22 16:16:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001591008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 14.527458983s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.527487097s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...X...........................................X.........] duration: 19.839540478s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:16:12 UTC) orbiter warning: context deadline exceeded Duration: 19.839572659s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.003787761s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:15:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 30.003802089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.XX....X.X.XX.XX...................XXX.XX.......X..XX......] duration: 3.441151621s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:15:37 UTC) orbiter warning: context deadline exceeded Duration: 3.441191366s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...X...........................................X........] duration: 18.529820896s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:15:26 UTC) orbiter warning: context deadline exceeded Duration: 18.529857677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.006023589s trace: https://jaeger.dev.storj.io/trace/1a5dbd88951e1d7b FAIL (2025-11-22 16:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 3PGH63RMZ37WZTBM7JY3NL7RGC6E4HRXKYFUQBDLF43GHQOQ4AOQ) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: PUQ6Q5NIPXPTTSTD66VHRNKBGUEM6DVX7Y4QHIDPWCUBIDGPMF3Q) context deadline exceeded ======LOGS====== .932653685 (320h10m38.484124146s) [upload=281970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:14:40.179792370 (320h10m38.731262831s) [seg_pos={0 0}, upload=281970]: Begin upload segment... Nov 22 16:14:40.179840963 (320h10m38.731311414s) [seg_pos={0 0}, upload=281970]: Done begin upload segment. Nov 22 16:14:40.179868185 (320h10m38.731338646s) [noise=true , address=145.79.227.31:4797 , piece_id=ZLTEQBMQ34EBQPMYASKTCPGGBBX45UB6SPPJLFC22NFHKBXKV7OA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.179937847 (320h10m38.731408308s) [noise=true , address=45.91.116.4:4938 , piece_id=RPLWRHLU3CFHDXSK6GLLPKOE2FRKAF4MIQHXP4CWCLEIHYI5KIOA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.180449512 (320h10m38.731919973s) [noise=true , address=145.79.227.28:4725 , piece_id=5RF7PVNYUKCLTBRAKJVDY2UGIU7LVNFANJT2J4RBJSHDCCQHZI2Q, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.180911191 (320h10m38.732381652s) [noise=true , address=145.79.227.15:4876 , piece_id=KIPT4IPUJ2LMF6DW6BDXVNXKZG62XKMBAGLPOG4HE5BZTT5SEWMA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.181322394 (320h10m38.732792855s) [noise=true , address=45.91.116.6:4975 , piece_id=WZ4S2VTZ5CRVP7XTLPIIXHD2DERTUVMDZ55W652E56DJPFS5QR2A, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.181714821 (320h10m38.733185282s) [noise=true , address=77.90.31.18:4807 , piece_id=EQFQUCJU2EONSCRWKCOOP2IC6LVB36CUWFTUXJRF7Y7TY7VTUDIA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.182152625 (320h10m38.733623086s) [noise=true , address=145.79.227.5:4223 , piece_id=UMWNYPFFKG5A4CCIXPROE4LQQAB5Q6RNSFX76LFRUVL4LHNZMA7Q, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.182582413 (320h10m38.734052874s) [noise=true , address=145.79.227.21:4474 , piece_id=CRWT4SSLN67BJOWTOHRGKX566TL6DQ2RTMM3R47IJ4PW2PDVE6WQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.183093396 (320h10m38.734563857s) [noise=true , address=145.79.227.14:4839 , piece_id=PAWXAWVVP47ARKSKT7T7KOZBUSMHKQKZB77SMIEDA2XU5UMWT2AQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.184356191 (320h10m38.735826652s) [noise=true , address=145.79.227.16:4911 , piece_id=HUX7RM4L23XR7JXKE7KS5GLPTNS6QXHYX6TNP6QGKR5W2MW5XHEQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.185699078 (320h10m38.737169539s) [noise=true , address=145.79.227.7:4294 , piece_id=3PGH63RMZ37WZTBM7JY3NL7RGC6E4HRXKYFUQBDLF43GHQOQ4AOQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.187353949 (320h10m38.73882441s) [noise=true , address=145.79.227.17:4971 , piece_id=PUQ6Q5NIPXPTTSTD66VHRNKBGUEM6DVX7Y4QHIDPWCUBIDGPMF3Q, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.188869946 (320h10m38.740340397s) [noise=true , address=45.91.116.7:4011 , piece_id=GPUFKBUUYOGSAALDB6GBF77N4CJF44RNUHGV2BXJQ3PT3UVTKBWQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.190747782 (320h10m38.742218243s) [noise=true , address=77.90.31.16:4734 , piece_id=DQCRSUQRR76AIJ436LEJGFWWLRHCRX7G4AEUE4ER7Y5ZZMKB4ZJQ, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.192267735 (320h10m38.743738196s) [noise=true , address=77.90.31.11:4554 , piece_id=AL5RF4QYXK4WQIEPMFWAGL2KNOC5QEB65RB6K55R6PEFTRWLT7KA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:40.194039179 (320h10m38.74550964s) [noise=true , address=145.79.227.10:4401 , piece_id=HBV5IJSPVH4DOCYFR727Y2PL6HXBPPB5FUHKNOO4AOFWM7XHRCLA, seg_pos={0 0}, upload=281970]: Uploading piece... Nov 22 16:14:42.012962224 (320h10m40.564432675s) [upload=281970]: Next returned nil segment Nov 22 16:14:42.012988695 (320h10m40.564459156s) [upload=281970]: Waiting for error group managing segments... Nov 22 16:14:42.318782549 (320h10m40.87025301s) [noise=true , address=45.91.116.4:4938 , piece_id=RPLWRHLU3CFHDXSK6GLLPKOE2FRKAF4MIQHXP4CWCLEIHYI5KIOA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:42.342743285 (320h10m40.894213746s) [noise=true , address=77.90.31.16:4734 , piece_id=DQCRSUQRR76AIJ436LEJGFWWLRHCRX7G4AEUE4ER7Y5ZZMKB4ZJQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:42.347646230 (320h10m40.899116691s) [noise=true , address=45.91.116.6:4975 , piece_id=WZ4S2VTZ5CRVP7XTLPIIXHD2DERTUVMDZ55W652E56DJPFS5QR2A, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:42.359492668 (320h10m40.910963129s) [noise=true , address=45.91.116.7:4011 , piece_id=GPUFKBUUYOGSAALDB6GBF77N4CJF44RNUHGV2BXJQ3PT3UVTKBWQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:42.435819999 (320h10m40.98729045s) [noise=true , address=77.90.31.18:4807 , piece_id=EQFQUCJU2EONSCRWKCOOP2IC6LVB36CUWFTUXJRF7Y7TY7VTUDIA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:42.499277263 (320h10m41.050747734s) [noise=true , address=77.90.31.11:4554 , piece_id=AL5RF4QYXK4WQIEPMFWAGL2KNOC5QEB65RB6K55R6PEFTRWLT7KA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:49.043988509 (320h10m47.59545897s) [noise=true , address=145.79.227.15:4876 , piece_id=KIPT4IPUJ2LMF6DW6BDXVNXKZG62XKMBAGLPOG4HE5BZTT5SEWMA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:50.695390125 (320h10m49.246860586s) [noise=true , address=145.79.227.16:4911 , piece_id=HUX7RM4L23XR7JXKE7KS5GLPTNS6QXHYX6TNP6QGKR5W2MW5XHEQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:53.082374397 (320h10m51.633844858s) [noise=true , address=145.79.227.7:4294 , piece_id=3PGH63RMZ37WZTBM7JY3NL7RGC6E4HRXKYFUQBDLF43GHQOQ4AOQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:53.160741612 (320h10m51.712212083s) [noise=true , address=145.79.227.17:4971 , piece_id=PUQ6Q5NIPXPTTSTD66VHRNKBGUEM6DVX7Y4QHIDPWCUBIDGPMF3Q, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:14:54.291693129 (320h10m52.84316361s) [noise=true , address=145.79.227.5:4223 , piece_id=UMWNYPFFKG5A4CCIXPROE4LQQAB5Q6RNSFX76LFRUVL4LHNZMA7Q, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:15:00.639246192 (320h10m59.190716653s) [noise=true , address=145.79.227.31:4797 , piece_id=ZLTEQBMQ34EBQPMYASKTCPGGBBX45UB6SPPJLFC22NFHKBXKV7OA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:15:04.120017254 (320h11m2.671487725s) [noise=true , address=145.79.227.14:4839 , piece_id=PAWXAWVVP47ARKSKT7T7KOZBUSMHKQKZB77SMIEDA2XU5UMWT2AQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: <nil> Nov 22 16:15:04.120072700 (320h11m2.671543161s) [seg_pos={0 0}, upload=281970]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:04.120432966 (320h11m2.671903437s) [noise=true , address=145.79.227.10:4401 , piece_id=HBV5IJSPVH4DOCYFR727Y2PL6HXBPPB5FUHKNOO4AOFWM7XHRCLA, seg_pos={0 0}, upload=281970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:60620->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:60620->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60620->145.79.227.10:4401: use of closed network connection Nov 22 16:15:04.120445240 (320h11m2.671915701s) [noise=true , address=145.79.227.10:4401 , piece_id=HBV5IJSPVH4DOCYFR727Y2PL6HXBPPB5FUHKNOO4AOFWM7XHRCLA, seg_pos={0 0}, upload=281970]: Long tail cancellation due to optimal threshold Nov 22 16:15:04.121434854 (320h11m2.672905315s) [noise=true , address=145.79.227.28:4725 , piece_id=5RF7PVNYUKCLTBRAKJVDY2UGIU7LVNFANJT2J4RBJSHDCCQHZI2Q, seg_pos={0 0}, upload=281970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:33790->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:33790->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33790->145.79.227.28:4725: use of closed network connection Nov 22 16:15:04.121449773 (320h11m2.672920244s) [noise=true , address=145.79.227.28:4725 , piece_id=5RF7PVNYUKCLTBRAKJVDY2UGIU7LVNFANJT2J4RBJSHDCCQHZI2Q, seg_pos={0 0}, upload=281970]: Long tail cancellation due to optimal threshold Nov 22 16:15:04.121692275 (320h11m2.673162766s) [noise=true , address=145.79.227.21:4474 , piece_id=CRWT4SSLN67BJOWTOHRGKX566TL6DQ2RTMM3R47IJ4PW2PDVE6WQ, seg_pos={0 0}, upload=281970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:56480->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:56480->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56480->145.79.227.21:4474: use of closed network connection Nov 22 16:15:04.121703115 (320h11m2.673173586s) [noise=true , address=145.79.227.21:4474 , piece_id=CRWT4SSLN67BJOWTOHRGKX566TL6DQ2RTMM3R47IJ4PW2PDVE6WQ, seg_pos={0 0}, upload=281970]: Long tail cancellation due to optimal threshold Nov 22 16:15:04.121747209 (320h11m2.67321767s) [seg_pos={0 0}, upload=281970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:04.122150257 (320h11m2.673620718s) [upload=281970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:04.122160406 (320h11m2.673630877s) [upload=281970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:04.421456788 (320h11m2.972927249s) [upload=281970]: Done uploading segments... Nov 22 16:15:04.421495262 (320h11m2.972965723s) [upload=281970]: Upload finished. err: <nil> Duration:* 30.006049217s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...X...........................................X.......] duration: 17.440339203s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:14:50 UTC) orbiter warning: context deadline exceeded Duration: 17.440375403s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X.XX....X.X.XX.XX...................XXX.XX.......X..XX.....] duration: 3.435559051s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:14:34 UTC) orbiter warning: context deadline exceeded Duration: 3.43561635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.X.XX....X.X.XX.XX...................XXX.XX.......X..XX....] duration: 3.55309023s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:14:17 UTC) orbiter warning: context deadline exceeded Duration: 3.553121691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...X...........................................X......] duration: 11.946854374s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:14:09 UTC) orbiter warning: context deadline exceeded Duration: 11.946876127s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...X...........................................X.....] duration: 11.652656111s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:13:49 UTC) orbiter warning: context deadline exceeded Duration: 11.652675267s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X.X.XX....X.X.XX.XX...................XXX.XX.......X..XX...] duration: 5.001641696s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:13:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001665049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X...X...........................................X....] duration: 19.08623297s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:13:38 UTC) orbiter warning: context deadline exceeded Duration: 19.086259693s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X.X.XX....X.X.XX.XX...................XXX.XX.......X..XX..] duration: 3.403669568s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:12:53 UTC) orbiter warning: context deadline exceeded Duration: 3.40370191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......X...X...........................................X...] duration: 21.35080561s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:12:31 UTC) orbiter warning: context deadline exceeded Duration: 21.350878799s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X.X.XX....X.X.XX.XX...................XXX.XX.......X..XX.] duration: 3.496604692s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:12:20 UTC) orbiter warning: context deadline exceeded Duration: 3.496638146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X......X...X...........................................X..] duration: 16.512022367s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:12:06 UTC) orbiter warning: context deadline exceeded Duration: 16.512051502s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [....X.X.XX....X.X.XX.XX...................XXX.XX.......X..XX] duration: 5.000777108s trace: https://jaeger.dev.storj.io/trace/205b04fb7b44e544 FAIL (2025-11-22 16:11:40 UTC) orbiter warning: context deadline exceeded Duration: 5.000800473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.....X.X.XX....X.X.XX.XX...................XXX.XX.......X..X] duration: 5.001675604s trace: https://jaeger.dev.storj.io/trace/2f48c7833ccb65fa FAIL (2025-11-22 16:11:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002003499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X......X...X...........................................X.] duration: 7.738984349s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:11:18 UTC) orbiter warning: context deadline exceeded Duration: 7.739004518s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X......X...X...........................................X] duration: 30.000834325s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:11:10 UTC) orbiter warning: context deadline exceeded Duration: 30.000856178s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.019679967s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:10:49 UTC) orbiter warning: context deadline exceeded Duration: 6.019709814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X......X...X...........................................] duration: 16.834802547s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:10:20 UTC) orbiter warning: context deadline exceeded Duration: 16.83483609s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [......X.X.XX....X.X.XX.XX...................XXX.XX.......X..] duration: 3.53128563s trace: https://jaeger.dev.storj.io/trace/1656221134f740c9 FAIL (2025-11-22 16:10:13 UTC) orbiter warning: context deadline exceeded Duration: 3.531330846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 14.955849863s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:10:01 UTC) orbiter warning: context deadline exceeded Duration: 14.955878739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X......X...X..........................................] duration: 7.378482122s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:09:55 UTC) orbiter warning: context deadline exceeded Duration: 7.378621296s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.X......X...X.........................................] duration: 6.760469327s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:09:31 UTC) orbiter warning: context deadline exceeded Duration: 6.760501599s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.......X.X.XX....X.X.XX.XX...................XXX.XX.......X.] duration: 3.747166205s trace: https://jaeger.dev.storj.io/trace/1656221134f740c9 FAIL (2025-11-22 16:09:19 UTC) orbiter warning: context deadline exceeded Duration: 3.747197185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.801740953s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:09:18 UTC) orbiter warning: context deadline exceeded Duration: 5.801769907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....X.X......X...X........................................] duration: 18.058450786s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:09:10 UTC) orbiter warning: context deadline exceeded Duration: 18.05848528s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.817806644s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:08:21 UTC) orbiter warning: context deadline exceeded Duration: 5.817836751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [........X.X.XX....X.X.XX.XX...................XXX.XX.......X] duration: 5.000929898s trace: https://jaeger.dev.storj.io/trace/1656221134f740c9 FAIL (2025-11-22 16:08:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000958544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X....X.X......X...X.......................................] duration: 17.914614217s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:08:04 UTC) orbiter warning: context deadline exceeded Duration: 17.914904469s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X....X.X......X...X......................................] duration: 19.174007197s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:07:37 UTC) orbiter warning: context deadline exceeded Duration: 19.174036345s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X........X.X.XX....X.X.XX.XX...................XXX.XX.......] duration: 3.72981188s trace: https://jaeger.dev.storj.io/trace/4c5b769530e50f21 FAIL (2025-11-22 16:07:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.729842278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.284081872s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:07:26 UTC) orbiter warning: context deadline exceeded Duration: 6.284110148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X........X.X.XX....X.X.XX.XX...................XXX.XX......] duration: 3.537777123s trace: https://jaeger.dev.storj.io/trace/4c5b769530e50f21 FAIL (2025-11-22 16:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537815136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.827195391s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:06:26 UTC) orbiter warning: context deadline exceeded Duration: 5.827222714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.X....X.X......X...X.....................................] duration: 14.388356128s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:06:20 UTC) orbiter warning: context deadline exceeded Duration: 14.388385704s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X........X.X.XX....X.X.XX.XX...................XXX.XX.....] duration: 3.975717057s trace: https://jaeger.dev.storj.io/trace/4c5b769530e50f21 FAIL (2025-11-22 16:06:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.975744308s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.337273569s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:05:55 UTC) orbiter warning: context deadline exceeded Duration: 6.337296854s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)