Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 700904 reports (Page 140 of 14019)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.675670454s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:59:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.675693279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 5.689879536s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.689894875s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................XX.X.X.........................] duration: 3.589053575s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589075837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.370235281s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:58:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.370257114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.914304774s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:58:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.914328691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 5.804588814s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:58:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.804615546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.919482656s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:58:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.919502756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.73238177s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:57:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.732422008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................XX.X.X........................] duration: 3.47609316s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:57:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476108258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.345565926s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.345581165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................XX.X.X.......................] duration: 3.531066837s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:57:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531093898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.933754651s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:57:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933983783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 5.697011038s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.697057488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.47787499s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:56:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.47790118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................XX.X.X......................] duration: 3.497514437s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:56:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498266198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 5.74863508s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:56:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.748659568s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.704902722s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:55:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.704923221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................XX.X.X.....................] duration: 3.640736837s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:55:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.640758179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.389816276s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:55:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.389840112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.332941494s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:55:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.33297041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 5.760131708s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.760157988s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................XX.X.X....................] duration: 3.462727053s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:55:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462755348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.027243644s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:54:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.027280656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................XX.X.X...................] duration: 3.551987321s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:54:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552013842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.693819501s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:54:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.693841244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................XX.X.X..................] duration: 3.851744539s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:54:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.851762513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.714400468s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:53:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.714423963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.502833023s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:53:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.502858352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 5.699001728s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.699036534s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.769758222s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:53:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.76977167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.419074085s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:53:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.419099645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................................XX.X.X.................] duration: 3.541240688s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:53:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541253834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.329578083s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.329604012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.628456352s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:52:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.628540944s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.851850285s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:52:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.851871797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.795295399s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:52:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.795319966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.08080019s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:52:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.080821222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................................XX.X.X................] duration: 3.52163371s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:51:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521669168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.458093652s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:51:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.458121285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.613462086s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:51:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.613487824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.756593827s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:51:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.756615047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.725285119s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:51:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.725306761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.208367244s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.208459211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.694822657s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:51:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.694846321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................................XX.X.X...............] duration: 3.448760517s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:50:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448785547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 5.72825022s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:50:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.728277293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.728430395s trace: https://jaeger.dev.storj.io/trace/276b2e32db11ea61 FAIL (2026-04-11 00:50:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 1) ======LOGS====== BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece... Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments... Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil> Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil> Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments... Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1) Duration: 5.728451935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.523549844s trace: https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc FAIL (2026-04-11 00:49:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment... Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment. Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece... Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments... Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil> Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments... Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.523569883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.86250412s trace: https://jaeger.dev.storj.io/trace/2ead608de6f962d2 FAIL (2026-04-11 00:49:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86252612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................XX.X.X..............] duration: 3.379001848s trace: https://jaeger.dev.storj.io/trace/5366ea4092e184b8 FAIL (2026-04-11 00:49:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379023991s core uplink us-select-1 us1-placement-20

1
...
138
139
140
141
142
...
14019