Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.237291855s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:07:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.237318267s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.645818767s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:07:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.645885365s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
5.752453931s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:07:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.752482746s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.765388282s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:06:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.765416066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.774755711s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:06:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.774769628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................XX.X.X.....................................] duration:
3.551180557s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:06:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.55120284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.877645311s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:06:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.87767098s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.938686521s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:06:10 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.938702501s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
5.731141629s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:06:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.731162821s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................XX.X.X....................................] duration:
3.612447698s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:05:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.612469471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.387372774s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:05:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.387397171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.841109212s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:05:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.841144499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................XX.X.X...................................] duration:
4.648044381s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:04:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.64806374s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
5.737190298s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:04:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.737213492s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.335378577s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:04:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.335402843s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................XX....................................] duration:
5.781621068s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:04:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.781650777s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.629709481s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:04:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.6297301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................XX.X.X..................................] duration:
3.716119864s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:04:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.716145213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
6.750300175s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:04:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.750330743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
5.771373253s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:03:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.771402961s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....................XX.X.X.................................] duration:
3.757831156s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:03:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.757850995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
9.602541901s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:03:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.602561148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.245205813s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:03:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.245226464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
5.688108386s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:02:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.688132662s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......................XX.X.X................................] duration:
3.592244956s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:02:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.59226779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.24991379s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:02:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.249933841s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.848946235s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:02:12 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.848967076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
6.206927433s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:02:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.206952402s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......................XX.X.X...............................] duration:
3.535403707s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:02:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.535430529s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
5.801278646s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:01:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.801303295s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.413463205s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:01:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.413488133s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.67302948s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:01:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.673045351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
5.720236843s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:01:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.720260659s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.33569686s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:01:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.335722732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........................XX.X.X..............................] duration:
3.477587789s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:00:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.477636392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.190702902s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:00:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.190719315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........................XX.X.X.............................] duration:
3.649475074s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:00:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.649499853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.709155593s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:00:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.709221941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
5.833791034s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:00:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.833821964s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........................XX.X.X............................] duration:
3.516404334s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:59:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.516434783s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
5.80953281s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:59:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.809550746s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.36726568s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:59:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.367291972s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........................XX.X.X...........................] duration:
3.877977292s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:59:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.877998974s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................XX.X.X..........................] duration:
3.504114959s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:59:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.504138645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.675670454s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:59:05 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.675693279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
5.689879536s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:59:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.689894875s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................XX.X.X.........................] duration:
3.589053575s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 00:58:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.589075837s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.370235281s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 00:58:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.370257114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.914304774s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 00:58:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.914328691s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................XX............................] duration:
5.804588814s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 00:58:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.804615546s
core
uplink
us-select-1
|