Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.720700703s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:44:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.720726864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................................XX.X.X.......] duration:
3.492566497s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:44:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.492591625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
5.763435024s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:43:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.763464069s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.494009667s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:43:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.49403751s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................................................XX.X.X......] duration:
3.715905139s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:43:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.715925989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
7.119145029s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:43:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 7.119174837s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.717294028s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:43:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.717331189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................................................XX.X.X.....] duration:
3.429343603s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:42:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.429366587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.287034868s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:42:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.287051189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
5.73021897s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:42:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.730232106s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................................................XX.X.X....] duration:
3.453999626s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:42:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.454025867s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.280253101s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:42:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.280276817s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.638799775s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:42:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.638825134s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
6.16128444s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:41:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 6.161309288s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................................................XX.X.X...] duration:
3.498742185s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:41:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.498757455s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.771083332s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:41:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.771104682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
5.76490754s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:41:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.764929382s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................................................XX.X.X..] duration:
3.656906186s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:41:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.656928008s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
5.655329517s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:41:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.655500626s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.442501159s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:41:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.442526318s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.640299049s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:40:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.640320639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.484861521s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:40:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.484887801s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.823958981s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:40:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.823989761s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
5.8727317s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:40:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.872759513s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....................................................XX.X.X.] duration:
3.57185969s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:40:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.571883085s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.513370871s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:40:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.513397462s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
5.726272262s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:39:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.726293892s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.807296634s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:39:53 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.807318756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.623150341s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:39:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.623173285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......................................................XX.X.X] duration:
5.002034468s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:39:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00205636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
5.737485583s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:39:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.737508938s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
26.976713604s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:39:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.976753791s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
5.764846775s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:39:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.764883185s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.500435751s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:38:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.500460699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......................................................XX.X.] duration:
3.520134914s trace:
https://jaeger.dev.storj.io/trace/4d0062f369162a92
FAIL (2026-04-11 00:38:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.520162387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.819006915s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:38:30 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.819019751s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.460825967s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:38:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.460874041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
5.776810985s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:37:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.77683325s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [........................................................XX.X] duration:
5.000451864s trace:
https://jaeger.dev.storj.io/trace/4d0062f369162a92
FAIL (2026-04-11 00:37:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000474608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
5.708172651s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:37:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.70819327s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
15.344298256s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:37:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.344318213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.873673498s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:37:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.873699239s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
8.304610348s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:36:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 8.304630768s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................................XX.] duration:
4.476818785s trace:
https://jaeger.dev.storj.io/trace/26a985775675b83e
FAIL (2026-04-11 00:36:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.476836579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
18.005412656s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:36:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.005436793s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
16.885208192s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:36:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 16.885227869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
23.058370755s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:36:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 23.058386796s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.281430055s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:36:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.281456487s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
36.844531725s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:36:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.844543297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.232944563s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:35:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 30.232964801s
core
uplink
us-select-1
|