Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X..X......X......X...X...X........X.....X.X..........] duration: 4.54290051s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:26:34 UTC) orbiter warning: context deadline exceeded Duration: 4.542924285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...............................X....X........] duration: 39.361410581s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 39.361441038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX..........XX.] duration: 12.106315791s trace: https://jaeger.dev.storj.io/trace/190d2f9c84ed83c7 FAIL (2025-12-30 10:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA) context deadline exceeded ======LOGS====== Dec 30 10:24:58.744655103 (594h55m58.615738148s) [upload=521524]: Starting upload Dec 30 10:24:58.744753912 (594h55m58.615836957s) [upload=521524]: Done starting upload Dec 30 10:24:58.744850887 (594h55m58.615933932s) [upload=521524]: Uploading segments... Dec 30 10:24:58.745087968 (594h55m58.616171003s) [upload=521524]: Got next segment. Inline: false Dec 30 10:24:58.745119989 (594h55m58.616203024s) [upload=521524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:24:58.745124468 (594h55m58.616207513s) [upload=521524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:24:58.992226222 (594h55m58.863309267s) [seg_pos={0 0}, upload=521524]: Begin upload segment... Dec 30 10:24:58.992270437 (594h55m58.863353482s) [seg_pos={0 0}, upload=521524]: Done begin upload segment. Dec 30 10:24:58.992294492 (594h55m58.863377527s) [noise=true , address=145.79.227.38:4977 , piece_id=K7SKWMO3M7IFTT7PQARD6ZBGRVIRLHQ2OJS4ENARYJOTG2O7GSSQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.992379404 (594h55m58.863462439s) [noise=true , address=77.90.31.13:4629 , piece_id=2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.992834601 (594h55m58.863917646s) [noise=true , address=145.79.227.51:4411 , piece_id=H23SUTS6SL3W5QUWKW4WI6PVS7QYAXMSNMEVPOBU6MVM7QBQYFMQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993253017 (594h55m58.864336062s) [noise=true , address=145.79.227.19:4079 , piece_id=OJEUO3V465IDRDYFZWSA5BZGQYOITDNI5HAQTAOCNVYS5FNXKSIA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993610568 (594h55m58.864693613s) [noise=true , address=145.79.227.4:4187 , piece_id=NDZK7EWOICKZOU2TCR4T7EC2NT5U675E5RB275B6LP7V6BVU4I5A, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993979521 (594h55m58.865062566s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.994292076 (594h55m58.865375121s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.994592678 (594h55m58.865675713s) [noise=true , address=145.79.227.22:4510 , piece_id=MWHGYFOLXUWMWZJCEUXJYLLZ5UWEX6767BFILZ4JO4ITBKGRJ5GA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.995066009 (594h55m58.866149045s) [noise=true , address=145.79.227.37:4565 , piece_id=I5K6D3QCZCED6IBKANIF3U5I6FIUTGDVXA3FIUXEY632QGTWOWLQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.996022330 (594h55m58.867105365s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.997581669 (594h55m58.868664705s) [noise=true , address=145.79.227.12:4765 , piece_id=BR6INZJ3ED33OC4R56LKOHH2EOF3AXYNTFDNAWBQTR5RWIWRMCUQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.997613851 (594h55m58.868696896s) [noise=true , address=145.79.227.3:4149 , piece_id=CSBC63HVTS4QICPA6GXJMCT67A47ROCNNTO7T7NZS7MAHVPN36LQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.999034757 (594h55m58.870117802s) [noise=true , address=145.79.227.52:4445 , piece_id=VSSSEUYAPRISKCFC5FWU3LFPTP7GKSTRXAKLZGELZZ3WZFQAJVOQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.000231605 (594h55m58.87131466s) [noise=true , address=145.79.227.45:4230 , piece_id=V2MHJNS6KCKHTQNKFL2SUKQVNKHJI2FL7XFCZWHTHW7IRDERF3BQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.001707997 (594h55m58.872791042s) [noise=true , address=77.90.31.15:4700 , piece_id=VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.003546738 (594h55m58.874629793s) [noise=true , address=145.79.227.5:4223 , piece_id=C2GPPP6YJLWMSWQMNPQH6F73K46KUB2CPZDBV25J3MH2MKFM37EQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:25:00.694351440 (594h56m0.565434475s) [upload=521524]: Next returned nil segment Dec 30 10:25:00.694364084 (594h56m0.565447129s) [upload=521524]: Waiting for error group managing segments... Dec 30 10:25:00.984720823 (594h56m0.855803858s) [noise=true , address=145.79.227.37:4565 , piece_id=I5K6D3QCZCED6IBKANIF3U5I6FIUTGDVXA3FIUXEY632QGTWOWLQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.990548259 (594h56m0.861631294s) [noise=true , address=145.79.227.45:4230 , piece_id=V2MHJNS6KCKHTQNKFL2SUKQVNKHJI2FL7XFCZWHTHW7IRDERF3BQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.990608855 (594h56m0.8616919s) [noise=true , address=145.79.227.51:4411 , piece_id=H23SUTS6SL3W5QUWKW4WI6PVS7QYAXMSNMEVPOBU6MVM7QBQYFMQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.998092365 (594h56m0.86917541s) [noise=true , address=145.79.227.19:4079 , piece_id=OJEUO3V465IDRDYFZWSA5BZGQYOITDNI5HAQTAOCNVYS5FNXKSIA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.998183228 (594h56m0.869266263s) [noise=true , address=145.79.227.4:4187 , piece_id=NDZK7EWOICKZOU2TCR4T7EC2NT5U675E5RB275B6LP7V6BVU4I5A, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.011619012 (594h56m0.882702058s) [noise=true , address=145.79.227.3:4149 , piece_id=CSBC63HVTS4QICPA6GXJMCT67A47ROCNNTO7T7NZS7MAHVPN36LQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.013758586 (594h56m0.884841621s) [noise=true , address=145.79.227.38:4977 , piece_id=K7SKWMO3M7IFTT7PQARD6ZBGRVIRLHQ2OJS4ENARYJOTG2O7GSSQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.019541939 (594h56m0.890624984s) [noise=true , address=145.79.227.52:4445 , piece_id=VSSSEUYAPRISKCFC5FWU3LFPTP7GKSTRXAKLZGELZZ3WZFQAJVOQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.023367213 (594h56m0.894450248s) [noise=true , address=145.79.227.5:4223 , piece_id=C2GPPP6YJLWMSWQMNPQH6F73K46KUB2CPZDBV25J3MH2MKFM37EQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.067098104 (594h56m0.938181149s) [noise=true , address=145.79.227.12:4765 , piece_id=BR6INZJ3ED33OC4R56LKOHH2EOF3AXYNTFDNAWBQTR5RWIWRMCUQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.075403799 (594h56m0.946486844s) [noise=true , address=145.79.227.22:4510 , piece_id=MWHGYFOLXUWMWZJCEUXJYLLZ5UWEX6767BFILZ4JO4ITBKGRJ5GA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.239126395 (594h56m1.11020944s) [noise=true , address=77.90.31.13:4629 , piece_id=2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.291716386 (594h56m1.162799461s) [noise=true , address=77.90.31.15:4700 , piece_id=VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.291759508 (594h56m1.162842573s) [seg_pos={0 0}, upload=521524]: Segment reached optimal threshold of 13 pieces. Dec 30 10:25:01.291967534 (594h56m1.163050589s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection Dec 30 10:25:01.291977593 (594h56m1.163060648s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292035373 (594h56m1.163118418s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 10:25:01.292048247 (594h56m1.163131302s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292230484 (594h56m1.163313529s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 10:25:01.292237567 (594h56m1.163320612s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292259389 (594h56m1.163342424s) [seg_pos={0 0}, upload=521524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:25:01.292340323 (594h56m1.163423358s) [upload=521524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:25:01.292346836 (594h56m1.163429881s) [upload=521524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:25:01.598357342 (594h56m1.469440388s) [upload=521524]: Done uploading segments... Dec 30 10:25:01.598383943 (594h56m1.469466988s) [upload=521524]: Upload finished. err: <nil> Duration: 12.106389631s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X..X......X......X...X...X........X.....X.X.........] duration: 3.379361173s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:25:54 UTC) orbiter warning: context deadline exceeded Duration: 3.379392864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX..........XX] duration: 30.002857455s trace: https://jaeger.dev.storj.io/trace/190d2f9c84ed83c7 FAIL (2025-12-30 10:25:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA) context deadline exceeded ======LOGS====== Dec 30 10:24:58.744655103 (594h55m58.615738148s) [upload=521524]: Starting upload Dec 30 10:24:58.744753912 (594h55m58.615836957s) [upload=521524]: Done starting upload Dec 30 10:24:58.744850887 (594h55m58.615933932s) [upload=521524]: Uploading segments... Dec 30 10:24:58.745087968 (594h55m58.616171003s) [upload=521524]: Got next segment. Inline: false Dec 30 10:24:58.745119989 (594h55m58.616203024s) [upload=521524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:24:58.745124468 (594h55m58.616207513s) [upload=521524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:24:58.992226222 (594h55m58.863309267s) [seg_pos={0 0}, upload=521524]: Begin upload segment... Dec 30 10:24:58.992270437 (594h55m58.863353482s) [seg_pos={0 0}, upload=521524]: Done begin upload segment. Dec 30 10:24:58.992294492 (594h55m58.863377527s) [noise=true , address=145.79.227.38:4977 , piece_id=K7SKWMO3M7IFTT7PQARD6ZBGRVIRLHQ2OJS4ENARYJOTG2O7GSSQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.992379404 (594h55m58.863462439s) [noise=true , address=77.90.31.13:4629 , piece_id=2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.992834601 (594h55m58.863917646s) [noise=true , address=145.79.227.51:4411 , piece_id=H23SUTS6SL3W5QUWKW4WI6PVS7QYAXMSNMEVPOBU6MVM7QBQYFMQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993253017 (594h55m58.864336062s) [noise=true , address=145.79.227.19:4079 , piece_id=OJEUO3V465IDRDYFZWSA5BZGQYOITDNI5HAQTAOCNVYS5FNXKSIA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993610568 (594h55m58.864693613s) [noise=true , address=145.79.227.4:4187 , piece_id=NDZK7EWOICKZOU2TCR4T7EC2NT5U675E5RB275B6LP7V6BVU4I5A, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.993979521 (594h55m58.865062566s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.994292076 (594h55m58.865375121s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.994592678 (594h55m58.865675713s) [noise=true , address=145.79.227.22:4510 , piece_id=MWHGYFOLXUWMWZJCEUXJYLLZ5UWEX6767BFILZ4JO4ITBKGRJ5GA, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.995066009 (594h55m58.866149045s) [noise=true , address=145.79.227.37:4565 , piece_id=I5K6D3QCZCED6IBKANIF3U5I6FIUTGDVXA3FIUXEY632QGTWOWLQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.996022330 (594h55m58.867105365s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.997581669 (594h55m58.868664705s) [noise=true , address=145.79.227.12:4765 , piece_id=BR6INZJ3ED33OC4R56LKOHH2EOF3AXYNTFDNAWBQTR5RWIWRMCUQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.997613851 (594h55m58.868696896s) [noise=true , address=145.79.227.3:4149 , piece_id=CSBC63HVTS4QICPA6GXJMCT67A47ROCNNTO7T7NZS7MAHVPN36LQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:58.999034757 (594h55m58.870117802s) [noise=true , address=145.79.227.52:4445 , piece_id=VSSSEUYAPRISKCFC5FWU3LFPTP7GKSTRXAKLZGELZZ3WZFQAJVOQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.000231605 (594h55m58.87131466s) [noise=true , address=145.79.227.45:4230 , piece_id=V2MHJNS6KCKHTQNKFL2SUKQVNKHJI2FL7XFCZWHTHW7IRDERF3BQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.001707997 (594h55m58.872791042s) [noise=true , address=77.90.31.15:4700 , piece_id=VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:24:59.003546738 (594h55m58.874629793s) [noise=true , address=145.79.227.5:4223 , piece_id=C2GPPP6YJLWMSWQMNPQH6F73K46KUB2CPZDBV25J3MH2MKFM37EQ, seg_pos={0 0}, upload=521524]: Uploading piece... Dec 30 10:25:00.694351440 (594h56m0.565434475s) [upload=521524]: Next returned nil segment Dec 30 10:25:00.694364084 (594h56m0.565447129s) [upload=521524]: Waiting for error group managing segments... Dec 30 10:25:00.984720823 (594h56m0.855803858s) [noise=true , address=145.79.227.37:4565 , piece_id=I5K6D3QCZCED6IBKANIF3U5I6FIUTGDVXA3FIUXEY632QGTWOWLQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.990548259 (594h56m0.861631294s) [noise=true , address=145.79.227.45:4230 , piece_id=V2MHJNS6KCKHTQNKFL2SUKQVNKHJI2FL7XFCZWHTHW7IRDERF3BQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.990608855 (594h56m0.8616919s) [noise=true , address=145.79.227.51:4411 , piece_id=H23SUTS6SL3W5QUWKW4WI6PVS7QYAXMSNMEVPOBU6MVM7QBQYFMQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.998092365 (594h56m0.86917541s) [noise=true , address=145.79.227.19:4079 , piece_id=OJEUO3V465IDRDYFZWSA5BZGQYOITDNI5HAQTAOCNVYS5FNXKSIA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:00.998183228 (594h56m0.869266263s) [noise=true , address=145.79.227.4:4187 , piece_id=NDZK7EWOICKZOU2TCR4T7EC2NT5U675E5RB275B6LP7V6BVU4I5A, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.011619012 (594h56m0.882702058s) [noise=true , address=145.79.227.3:4149 , piece_id=CSBC63HVTS4QICPA6GXJMCT67A47ROCNNTO7T7NZS7MAHVPN36LQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.013758586 (594h56m0.884841621s) [noise=true , address=145.79.227.38:4977 , piece_id=K7SKWMO3M7IFTT7PQARD6ZBGRVIRLHQ2OJS4ENARYJOTG2O7GSSQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.019541939 (594h56m0.890624984s) [noise=true , address=145.79.227.52:4445 , piece_id=VSSSEUYAPRISKCFC5FWU3LFPTP7GKSTRXAKLZGELZZ3WZFQAJVOQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.023367213 (594h56m0.894450248s) [noise=true , address=145.79.227.5:4223 , piece_id=C2GPPP6YJLWMSWQMNPQH6F73K46KUB2CPZDBV25J3MH2MKFM37EQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.067098104 (594h56m0.938181149s) [noise=true , address=145.79.227.12:4765 , piece_id=BR6INZJ3ED33OC4R56LKOHH2EOF3AXYNTFDNAWBQTR5RWIWRMCUQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.075403799 (594h56m0.946486844s) [noise=true , address=145.79.227.22:4510 , piece_id=MWHGYFOLXUWMWZJCEUXJYLLZ5UWEX6767BFILZ4JO4ITBKGRJ5GA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.239126395 (594h56m1.11020944s) [noise=true , address=77.90.31.13:4629 , piece_id=2NAYT6LH623GWZ6YCX2DROUDA66LSWGCKZZQHNFAGGZIPHUKLLSA, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.291716386 (594h56m1.162799461s) [noise=true , address=77.90.31.15:4700 , piece_id=VXZ4DF4WECOLXEQUO42GXINIAKXSKOSB3GXYYII3TOPENQF2PQCQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: <nil> Dec 30 10:25:01.291759508 (594h56m1.162842573s) [seg_pos={0 0}, upload=521524]: Segment reached optimal threshold of 13 pieces. Dec 30 10:25:01.291967534 (594h56m1.163050589s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->45.91.116.8:4046: use of closed network connection Dec 30 10:25:01.291977593 (594h56m1.163060648s) [noise=true , address=45.91.116.8:4046 , piece_id=OJ7MIF3DZYV5ETIE5L356CWACQMXOC3KNEHZTUSMJNETCQFOS4FQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292035373 (594h56m1.163118418s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 10:25:01.292048247 (594h56m1.163131302s) [noise=true , address=45.91.116.14:4192 , piece_id=RORFGP2IE2EKPQWGLGUG6OEJXOOLWOU5EYXY7DEP4GEZ2ZIQ3KBQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292230484 (594h56m1.163313529s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 10:25:01.292237567 (594h56m1.163320612s) [noise=true , address=145.79.227.29:4943 , piece_id=HQBTDYVNWNU6WBBIQ73MLJ7ZD7B6PRYQURWF55P6KOPI22LQOSQQ, seg_pos={0 0}, upload=521524]: Long tail cancellation due to optimal threshold Dec 30 10:25:01.292259389 (594h56m1.163342424s) [seg_pos={0 0}, upload=521524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:25:01.292340323 (594h56m1.163423358s) [upload=521524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:25:01.292346836 (594h56m1.163429881s) [upload=521524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:25:01.598357342 (594h56m1.469440388s) [upload=521524]: Done uploading segments... Dec 30 10:25:01.598383943 (594h56m1.469466988s) [upload=521524]: Upload finished. err: <nil> Duration: 30.00288175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...............................X....X.......] duration: 44.771180303s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 44.771199599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........X..X......X......X...X...X........X.....X.X........] duration: 3.423631901s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:24:42 UTC) orbiter warning: context deadline exceeded Duration: 3.423656027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...............................X....X......] duration: 46.993361036s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 46.993387426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX..........X] duration: 30.002615571s trace: https://jaeger.dev.storj.io/trace/4942db35157765b9 FAIL (2025-12-30 10:24:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: AGHBK2NI7WF5SBTDZTCOOUAUOVOR7C3L2S5OU67ENOZMEBPNYLPQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: TSRO2CLONYIZNN2DMOC5MHMBA3FHS37LNRSJGUXTRRNWND27M7PQ) context deadline exceeded ======LOGS====== Dec 30 10:24:00.522304216 (594h55m0.393387261s) [upload=521511]: Starting upload Dec 30 10:24:00.522373348 (594h55m0.393456383s) [upload=521511]: Done starting upload Dec 30 10:24:00.522416519 (594h55m0.393499555s) [upload=521511]: Uploading segments... Dec 30 10:24:00.522569361 (594h55m0.393652396s) [upload=521511]: Got next segment. Inline: false Dec 30 10:24:00.522606281 (594h55m0.393689316s) [upload=521511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:24:00.522610770 (594h55m0.393693805s) [upload=521511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:24:00.766466753 (594h55m0.637549799s) [seg_pos={0 0}, upload=521511]: Begin upload segment... Dec 30 10:24:00.766515436 (594h55m0.637598481s) [seg_pos={0 0}, upload=521511]: Done begin upload segment. Dec 30 10:24:00.766547767 (594h55m0.637630813s) [noise=true , address=145.79.227.24:4581 , piece_id=R7UDQDUNZZXD42LAEA6WUEULDSZK4W7IG3YDF24AAVTLY555XF5Q, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.766651415 (594h55m0.63773445s) [noise=true , address=45.91.116.10:4083 , piece_id=KFUVJ72K7AKFEKVOHDRTQ66XUXLBQ6CKURP2POMRQSDLGNMACE4Q, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.767297786 (594h55m0.638380831s) [noise=true , address=145.79.227.27:4691 , piece_id=L37DDUSE3MTJTGKT7Y5237W3MHDORCV4ZHOLQSV54CWCX5DYCT6A, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.767667059 (594h55m0.638750104s) [noise=true , address=145.79.227.4:4186 , piece_id=SBPBHFBOT4EWKWNEPY33DF7KPB23CMPPHZCCA45X2ROC6ETJZGQA, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.768016724 (594h55m0.639099759s) [noise=true , address=77.90.31.14:4664 , piece_id=ZMB35HGUWNOW6M46YNELQ2CQEQNBN2XWW5UYEQXILAYVBDJFUN6Q, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.768409883 (594h55m0.639492928s) [noise=true , address=145.79.227.10:4401 , piece_id=4L6XZNKC5SRZS4CZOGAV3EJW3D4P22DKKK5HRU53X6IQZTSTN5YQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.768762585 (594h55m0.63984563s) [noise=true , address=77.90.31.10:4518 , piece_id=TSRO2CLONYIZNN2DMOC5MHMBA3FHS37LNRSJGUXTRRNWND27M7PQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.769137308 (594h55m0.640220343s) [noise=true , address=45.91.116.13:4153 , piece_id=YHHSTNHJMIOVWMLZXIN6VVVXED3DVQLAAS7SLN23U7ILKOA733WA, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.770462531 (594h55m0.641545576s) [noise=true , address=145.79.227.14:4838 , piece_id=YWA62SPHHQS4SLHKMPZVPORJIXWB6RE62UGFL2ZG4RQTIWN4R25A, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.770698311 (594h55m0.641781346s) [noise=true , address=145.79.227.36:4517 , piece_id=PG7DSTISGSKSERXW2LIZZVT2ZGY6IXXVEPYV6VIAUMUDWI2STNFQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.771692715 (594h55m0.64277576s) [noise=true , address=145.79.227.21:4473 , piece_id=LVA4B5NEFA6IET5FW2U7CKJAMCNVTCCVCZB6QGDKOXPRDM3HLUDA, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.772259293 (594h55m0.643342338s) [noise=true , address=145.79.227.7:4294 , piece_id=YI5PWEOHAZUM7IJ2UPI5SOWK73UVYP3DJ5OPP4JL5B2FW5BPH4LA, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.773878246 (594h55m0.644961281s) [noise=true , address=145.79.227.41:4085 , piece_id=XWPDKUMXRDNHTNBSOH25JNJXHWQAD46EZV4P7P3BK33GYF3GAJXQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.775286758 (594h55m0.646369793s) [noise=true , address=145.79.227.18:4007 , piece_id=RYCAQPKNYLB2RQVCOGPCCA24HWWAYLKAU6LYLJH2OZ77L4MWP7ZQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.776691573 (594h55m0.647774608s) [noise=true , address=145.79.227.11:4729 , piece_id=LKSGOT2BG4RJWT7RTZ2I5GVXOKGDGHMIHWM2ARZTDMTMYTHRGG6A, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:00.778400096 (594h55m0.649483141s) [noise=true , address=77.90.31.11:4556 , piece_id=AGHBK2NI7WF5SBTDZTCOOUAUOVOR7C3L2S5OU67ENOZMEBPNYLPQ, seg_pos={0 0}, upload=521511]: Uploading piece... Dec 30 10:24:02.385388528 (594h55m2.256471573s) [upload=521511]: Next returned nil segment Dec 30 10:24:02.385425107 (594h55m2.256508152s) [upload=521511]: Waiting for error group managing segments... Dec 30 10:24:02.766594440 (594h55m2.637677495s) [noise=true , address=145.79.227.41:4085 , piece_id=XWPDKUMXRDNHTNBSOH25JNJXHWQAD46EZV4P7P3BK33GYF3GAJXQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.768192573 (594h55m2.639275608s) [noise=true , address=145.79.227.11:4729 , piece_id=LKSGOT2BG4RJWT7RTZ2I5GVXOKGDGHMIHWM2ARZTDMTMYTHRGG6A, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.768257086 (594h55m2.639340131s) [noise=true , address=145.79.227.18:4007 , piece_id=RYCAQPKNYLB2RQVCOGPCCA24HWWAYLKAU6LYLJH2OZ77L4MWP7ZQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.771993430 (594h55m2.643076465s) [noise=true , address=145.79.227.14:4838 , piece_id=YWA62SPHHQS4SLHKMPZVPORJIXWB6RE62UGFL2ZG4RQTIWN4R25A, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.782160811 (594h55m2.653243856s) [noise=true , address=145.79.227.10:4401 , piece_id=4L6XZNKC5SRZS4CZOGAV3EJW3D4P22DKKK5HRU53X6IQZTSTN5YQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.782434091 (594h55m2.653517136s) [noise=true , address=145.79.227.36:4517 , piece_id=PG7DSTISGSKSERXW2LIZZVT2ZGY6IXXVEPYV6VIAUMUDWI2STNFQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.786050417 (594h55m2.657133462s) [noise=true , address=145.79.227.4:4186 , piece_id=SBPBHFBOT4EWKWNEPY33DF7KPB23CMPPHZCCA45X2ROC6ETJZGQA, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.786553014 (594h55m2.657636059s) [noise=true , address=145.79.227.24:4581 , piece_id=R7UDQDUNZZXD42LAEA6WUEULDSZK4W7IG3YDF24AAVTLY555XF5Q, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.787763649 (594h55m2.658846684s) [noise=true , address=145.79.227.27:4691 , piece_id=L37DDUSE3MTJTGKT7Y5237W3MHDORCV4ZHOLQSV54CWCX5DYCT6A, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.796595427 (594h55m2.667678472s) [noise=true , address=145.79.227.21:4473 , piece_id=LVA4B5NEFA6IET5FW2U7CKJAMCNVTCCVCZB6QGDKOXPRDM3HLUDA, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:02.844538789 (594h55m2.715621834s) [noise=true , address=145.79.227.7:4294 , piece_id=YI5PWEOHAZUM7IJ2UPI5SOWK73UVYP3DJ5OPP4JL5B2FW5BPH4LA, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:03.002701919 (594h55m2.873784964s) [noise=true , address=77.90.31.11:4556 , piece_id=AGHBK2NI7WF5SBTDZTCOOUAUOVOR7C3L2S5OU67ENOZMEBPNYLPQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:03.065625807 (594h55m2.936708852s) [noise=true , address=77.90.31.10:4518 , piece_id=TSRO2CLONYIZNN2DMOC5MHMBA3FHS37LNRSJGUXTRRNWND27M7PQ, seg_pos={0 0}, upload=521511]: Done uploading piece. err: <nil> Dec 30 10:24:03.065651426 (594h55m2.936734471s) [seg_pos={0 0}, upload=521511]: Segment reached optimal threshold of 13 pieces. Dec 30 10:24:03.065993948 (594h55m2.937077003s) [noise=true , address=45.91.116.10:4083 , piece_id=KFUVJ72K7AKFEKVOHDRTQ66XUXLBQ6CKURP2POMRQSDLGNMACE4Q, seg_pos={0 0}, upload=521511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 10:24:03.066006963 (594h55m2.937090028s) [noise=true , address=45.91.116.10:4083 , piece_id=KFUVJ72K7AKFEKVOHDRTQ66XUXLBQ6CKURP2POMRQSDLGNMACE4Q, seg_pos={0 0}, upload=521511]: Long tail cancellation due to optimal threshold Dec 30 10:24:03.066040066 (594h55m2.937123111s) [noise=true , address=77.90.31.14:4664 , piece_id=ZMB35HGUWNOW6M46YNELQ2CQEQNBN2XWW5UYEQXILAYVBDJFUN6Q, seg_pos={0 0}, upload=521511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:58156->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:24:03.066050415 (594h55m2.93713345s) [noise=true , address=77.90.31.14:4664 , piece_id=ZMB35HGUWNOW6M46YNELQ2CQEQNBN2XWW5UYEQXILAYVBDJFUN6Q, seg_pos={0 0}, upload=521511]: Long tail cancellation due to optimal threshold Dec 30 10:24:03.066224126 (594h55m2.937307171s) [noise=true , address=45.91.116.13:4153 , piece_id=YHHSTNHJMIOVWMLZXIN6VVVXED3DVQLAAS7SLN23U7ILKOA733WA, seg_pos={0 0}, upload=521511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 10:24:03.066233374 (594h55m2.937316419s) [noise=true , address=45.91.116.13:4153 , piece_id=YHHSTNHJMIOVWMLZXIN6VVVXED3DVQLAAS7SLN23U7ILKOA733WA, seg_pos={0 0}, upload=521511]: Long tail cancellation due to optimal threshold Dec 30 10:24:03.066282347 (594h55m2.937365382s) [seg_pos={0 0}, upload=521511]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:24:03.066380875 (594h55m2.93746391s) [upload=521511]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:24:03.066388249 (594h55m2.937471294s) [upload=521511]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:24:03.362079055 (594h55m3.2331621s) [upload=521511]: Done uploading segments... Dec 30 10:24:03.362097940 (594h55m3.233180975s) [upload=521511]: Upload finished. err: <nil> Duration: 30.00264073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.........X..X......X......X...X...X........X.....X.X.......] duration: 3.578007119s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:24:08 UTC) orbiter warning: context deadline exceeded Duration: 3.578039682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX.........X..X......X......X...X...X........X.....X.X......] duration: 3.845391476s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:23:48 UTC) orbiter warning: context deadline exceeded Duration: 3.845424348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...............................X....X.....] duration: 28.54820116s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 28.548227659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX..........] duration: 5.89124457s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 5.891271864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.XX.........X..X......X......X...X...X........X.....X.X.....] duration: 3.487386032s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:22:59 UTC) orbiter warning: context deadline exceeded Duration: 3.487412441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...............................X....X....] duration: 15.688799885s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:22:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 15.688831294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.XX.........X..X......X......X...X...X........X.....X.X....] duration: 3.644765388s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:22:43 UTC) orbiter warning: context deadline exceeded Duration: 3.64478738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX.........] duration: 5.618793166s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:22:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 5.618816659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX........] duration: 6.134781095s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:22:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 6.134806954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X.XX.........X..X......X......X...X...X........X.....X.X...] duration: 3.546795157s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:22:06 UTC) orbiter warning: context deadline exceeded Duration: 3.54682361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX.......] duration: 5.656509853s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:21:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 5.65653954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...............................X....X...] duration: 33.210917521s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:21:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 33.210945233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X.XX.........X..X......X......X...X...X........X.....X.X..] duration: 3.485068718s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:21:46 UTC) orbiter warning: context deadline exceeded Duration: 3.485145826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX......] duration: 5.960232426s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:21:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 5.960319301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...............................X....X..] duration: 27.325119531s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:20:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 27.325161973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X..X.XX.........X..X......X......X...X...X........X.....X.X.] duration: 3.464052309s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:20:46 UTC) orbiter warning: context deadline exceeded Duration: 3.464083589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX.....] duration: 6.344535292s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 6.344561341s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X..X.XX.........X..X......X......X...X...X........X.....X.X] duration: 5.001364896s trace: https://jaeger.dev.storj.io/trace/61cc99a6fd913394 FAIL (2025-12-30 10:20:09 UTC) orbiter warning: context deadline exceeded Duration: 5.001407708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...............................X....X.] duration: 41.008728207s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:20:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 41.008795505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X..X.XX.........X..X......X......X...X...X........X.....X.] duration: 3.521079269s trace: https://jaeger.dev.storj.io/trace/15774b55b16bb2b FAIL (2025-12-30 10:19:27 UTC) orbiter warning: context deadline exceeded Duration: 3.521106833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...............................X....X] duration: 1m0.003728592s trace: https://jaeger.dev.storj.io/trace/36447761231729b5 FAIL (2025-12-30 10:19:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055283542 (594h49m24.926366577s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055542806 (594h49m24.926625841s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055618040 (594h49m24.926701085s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055824894 (594h49m24.926907929s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.055993424 (594h49m24.927076459s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056098504 (594h49m24.927181539s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056344222 (594h49m24.927427257s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056371203 (594h49m24.927454248s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056598817 (594h49m24.927681862s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056757739 (594h49m24.927840784s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.056849815 (594h49m24.92793285s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057142021 (594h49m24.928225066s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:25.057533206 (594h49m24.928616252s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Uploading piece... Dec 30 10:18:26.475161744 (594h49m26.346244779s) [upload=521427]: Next returned nil segment Dec 30 10:18:26.475447969 (594h49m26.346531004s) [upload=521427]: Waiting for error group managing segments... Dec 30 10:18:26.831316274 (594h49m26.702399309s) [noise=true , address=64.49.13.138:4488 , piece_id=COVRX7GX7BMHFY7QTC36NE4PTFG5BC4C3O3HHARTGPLXZDXLG27A, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.894582995 (594h49m26.76566604s) [noise=true , address=145.79.227.35:4497 , piece_id=CLRVQA5T5J35J6ZAFHECCF7XOWQCF4S5OHWIK47HWIBYEUW3NEOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.918740866 (594h49m26.789823901s) [noise=true , address=145.79.227.43:4174 , piece_id=S6JTVYP36MNX5QZSYQV4OKPVUWL53EXK5HMAAZEKLYJAWPHN3KTA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:26.921926010 (594h49m26.793009055s) [noise=true , address=145.79.227.15:4888 , piece_id=OLUD2R2RXDNMBHKLITIMYWLFY4QP2ELPUJVFTQNP3E3ABU4VBLOA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.076204533 (594h49m26.947287568s) [noise=true , address=77.90.31.19:4854 , piece_id=A2DHGWOEMFQ6SKGP6WQDEVLRTTA4ZQGYF544OULIXXT2WUQHJ2PQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.094090801 (594h49m26.965173846s) [noise=true , address=77.90.31.21:4896 , piece_id=QVK5YVGK3USSJ6GATJO5AAJBZCCFY3HWOL344KC6NSHVYT6BELOQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.203869118 (594h49m27.074952163s) [noise=true , address=77.90.31.16:4749 , piece_id=KOTMEVOUBISDK4IZWQF3PL3QD4PZSFIPLGKNQKDGIQ3P453SGRLQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.211126839 (594h49m27.082209884s) [noise=true , address=45.91.116.7:4041 , piece_id=H2PSAXDR3PIWIEQJRBHKCMM6JBGS44UTROGQFLA7D454WUCIJPSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.238330168 (594h49m27.109413203s) [noise=true , address=77.90.31.13:4638 , piece_id=SCNMRX4ASD4XDZT5CIPNO2U2SA472STJLK4XIOXQORUPZENGQW4Q, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.321157616 (594h49m27.192240671s) [noise=true , address=145.79.227.40:4067 , piece_id=645BEL4H5SPX62WBC6KAK5LMYTBQPK5YH2PRGJZVZKFRRUIUM5IQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.640643030 (594h49m27.511726065s) [noise=true , address=77.90.31.18:4840 , piece_id=WLBLBX6SVZUU7OGCM4PE5RCQPC53RZRUIEPFGHLHOWYW2IPUX7SQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.840100812 (594h49m27.711183847s) [noise=true , address=145.79.227.34:4937 , piece_id=DNZC6WPMPTHJSK5HHIUA6ISMPS7WZR6VQESB6WDNQTYZZNSIFKBQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:27.899976846 (594h49m27.771059881s) [noise=true , address=145.79.227.30:4790 , piece_id=NV56NORAANPY3TOU3QYXVKC2U6NCQWSLRRBPNQQUFNF3RARIUJHQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086153168 (594h49m27.957236203s) [noise=true , address=145.79.227.33:4899 , piece_id=KJTVLYZSP742K72TQ6QQJCVK4G77W3TUKCPWRNSV2EUNSOWPACSA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: <nil> Dec 30 10:18:28.086177103 (594h49m27.957260138s) [seg_pos={0 0}, upload=521427]: Segment reached optimal threshold of 14 pieces. Dec 30 10:18:28.086378908 (594h49m27.957461943s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59950->64.49.13.169:4711: use of closed network connection Dec 30 10:18:28.086388476 (594h49m27.957471511s) [noise=true , address=64.49.13.169:4711 , piece_id=5BOMSSYDQFZTYVPRVQHMGTOGTONH7WKV6EIMGASZXFJXULZKHXSQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086398325 (594h49m27.95748136s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46460->64.49.13.169:4709: use of closed network connection Dec 30 10:18:28.086403495 (594h49m27.95748653s) [noise=true , address=64.49.13.169:4709 , piece_id=KZBFQE4NIR7ECYQZ45GNRDYCZXGBVDOQJJTIKZ3GK5DVTWPPSUIA, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086559551 (594h49m27.957642587s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ): protocol: expected piece hash; context canceled; EOF Dec 30 10:18:28.086569772 (594h49m27.957652807s) [noise=true , address=77.90.31.14:4691 , piece_id=LISUDDXMTGSSXYECAAUIBV6W5SBH6HCXTEXNYM6QNRO7SMMNAARQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086701021 (594h49m27.957784066s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36812->64.49.13.169:4708: use of closed network connection Dec 30 10:18:28.086709227 (594h49m27.957792272s) [noise=true , address=64.49.13.169:4708 , piece_id=FTTNMNRNOT7D4NFLPC4CVFRKTNTK5WKNMUX363PC2AJPE7TLR3TQ, seg_pos={0 0}, upload=521427]: Long tail cancellation due to optimal threshold Dec 30 10:18:28.086751838 (594h49m27.957834883s) [seg_pos={0 0}, upload=521427]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:18:28.086846377 (594h49m27.957929412s) [upload=521427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:18:28.086881244 (594h49m27.957964289s) [upload=521427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:18:28.388573168 (594h49m28.259656223s) [upload=521427]: Done uploading segments... Dec 30 10:18:28.388597293 (594h49m28.259680338s) [upload=521427]: Upload finished. err: <nil> Duration: 1m0.003739954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX....] duration: 5.933355295s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:19:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 5.933382366s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X..X.XX.........X..X......X......X...X...X........X.....X] duration: 5.000548011s trace: https://jaeger.dev.storj.io/trace/15774b55b16bb2b FAIL (2025-12-30 10:18:39 UTC) orbiter warning: context deadline exceeded Duration: 5.000660566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............................X....] duration: 25.036023881s trace: https://jaeger.dev.storj.io/trace/4fb89a9e54e769f5 FAIL (2025-12-30 10:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UhZuvUKofduRzgCWi3WQcfZVuekxBJUXydn3H1ktQ7AmRRvcx, Piece ID: 75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 30 10:14:31.374464365 (594h45m31.2455474s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374502719 (594h45m31.245585754s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374952003 (594h45m31.246035038s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375337639 (594h45m31.246420674s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375427309 (594h45m31.246510354s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375821950 (594h45m31.246904995s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376210771 (594h45m31.247293806s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376573892 (594h45m31.247656937s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377101036 (594h45m31.248184071s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377422367 (594h45m31.248505413s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377897122 (594h45m31.248980157s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378294578 (594h45m31.249377613s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378613306 (594h45m31.249696351s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378740998 (594h45m31.249824034s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379196656 (594h45m31.250279691s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379542404 (594h45m31.25062544s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379985118 (594h45m31.251068153s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.380375170 (594h45m31.251458215s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:32.330373077 (594h45m32.201456112s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 10:14:32.819622480 (594h45m32.690705515s) [upload=521369]: Next returned nil segment Dec 30 10:14:32.819634714 (594h45m32.690717759s) [upload=521369]: Waiting for error group managing segments... Dec 30 10:14:33.269321643 (594h45m33.140404688s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.296166971 (594h45m33.167250006s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.566310470 (594h45m33.437393515s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.586516255 (594h45m33.457599301s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.647495109 (594h45m33.518578154s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.111736063 (594h45m33.982819098s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.175714929 (594h45m34.046797964s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.219078501 (594h45m34.090161536s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.255956813 (594h45m34.127039848s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.285262166 (594h45m34.156345211s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.374195129 (594h45m34.245278174s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.445126625 (594h45m34.31620966s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.486740434 (594h45m34.357823469s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842111971 (594h45m34.713195016s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842136328 (594h45m34.713219393s) [seg_pos={0 0}, upload=521369]: Segment reached optimal threshold of 14 pieces. Dec 30 10:14:34.842397145 (594h45m34.71348019s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842406633 (594h45m34.713489688s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842616282 (594h45m34.713699327s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842635157 (594h45m34.713718193s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842689882 (594h45m34.713772917s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842699821 (594h45m34.713782856s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842728506 (594h45m34.713811551s) [seg_pos={0 0}, upload=521369]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:14:34.843191367 (594h45m34.714274412s) [upload=521369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:14:34.843200284 (594h45m34.714283319s) [upload=521369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:14:35.131924283 (594h45m35.003007328s) [upload=521369]: Done uploading segments... Dec 30 10:14:35.131946265 (594h45m35.0030293s) [upload=521369]: Upload finished. err: <nil> Duration: 25.036064909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX...] duration: 6.116686541s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:18:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 6.116728619s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X..X.XX.........X..X......X......X...X...X........X.....] duration: 3.532056991s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:18:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532100414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X....X..X.XX.........X..X......X......X...X...X........X....] duration: 4.171815315s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:17:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.171841906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............................X...] duration: 47.883945421s trace: https://jaeger.dev.storj.io/trace/4fb89a9e54e769f5 FAIL (2025-12-30 10:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UhZuvUKofduRzgCWi3WQcfZVuekxBJUXydn3H1ktQ7AmRRvcx, Piece ID: 75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 30 10:14:31.374464365 (594h45m31.2455474s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374502719 (594h45m31.245585754s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374952003 (594h45m31.246035038s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375337639 (594h45m31.246420674s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375427309 (594h45m31.246510354s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375821950 (594h45m31.246904995s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376210771 (594h45m31.247293806s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376573892 (594h45m31.247656937s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377101036 (594h45m31.248184071s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377422367 (594h45m31.248505413s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377897122 (594h45m31.248980157s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378294578 (594h45m31.249377613s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378613306 (594h45m31.249696351s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378740998 (594h45m31.249824034s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379196656 (594h45m31.250279691s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379542404 (594h45m31.25062544s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379985118 (594h45m31.251068153s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.380375170 (594h45m31.251458215s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:32.330373077 (594h45m32.201456112s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 10:14:32.819622480 (594h45m32.690705515s) [upload=521369]: Next returned nil segment Dec 30 10:14:32.819634714 (594h45m32.690717759s) [upload=521369]: Waiting for error group managing segments... Dec 30 10:14:33.269321643 (594h45m33.140404688s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.296166971 (594h45m33.167250006s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.566310470 (594h45m33.437393515s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.586516255 (594h45m33.457599301s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.647495109 (594h45m33.518578154s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.111736063 (594h45m33.982819098s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.175714929 (594h45m34.046797964s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.219078501 (594h45m34.090161536s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.255956813 (594h45m34.127039848s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.285262166 (594h45m34.156345211s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.374195129 (594h45m34.245278174s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.445126625 (594h45m34.31620966s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.486740434 (594h45m34.357823469s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842111971 (594h45m34.713195016s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842136328 (594h45m34.713219393s) [seg_pos={0 0}, upload=521369]: Segment reached optimal threshold of 14 pieces. Dec 30 10:14:34.842397145 (594h45m34.71348019s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842406633 (594h45m34.713489688s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842616282 (594h45m34.713699327s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842635157 (594h45m34.713718193s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842689882 (594h45m34.713772917s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842699821 (594h45m34.713782856s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842728506 (594h45m34.713811551s) [seg_pos={0 0}, upload=521369]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:14:34.843191367 (594h45m34.714274412s) [upload=521369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:14:34.843200284 (594h45m34.714283319s) [upload=521369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:14:35.131924283 (594h45m35.003007328s) [upload=521369]: Done uploading segments... Dec 30 10:14:35.131946265 (594h45m35.0030293s) [upload=521369]: Upload finished. err: <nil> Duration: 47.883962242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX..] duration: 6.100571012s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:17:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 6.100606409s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XX....X..X.XX.........X..X......X......X...X...X........X...] duration: 4.344473142s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:17:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.344517386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............................X..] duration: 50.256852986s trace: https://jaeger.dev.storj.io/trace/4fb89a9e54e769f5 FAIL (2025-12-30 10:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UhZuvUKofduRzgCWi3WQcfZVuekxBJUXydn3H1ktQ7AmRRvcx, Piece ID: 75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 30 10:14:31.374464365 (594h45m31.2455474s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374502719 (594h45m31.245585754s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374952003 (594h45m31.246035038s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375337639 (594h45m31.246420674s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375427309 (594h45m31.246510354s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375821950 (594h45m31.246904995s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376210771 (594h45m31.247293806s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376573892 (594h45m31.247656937s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377101036 (594h45m31.248184071s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377422367 (594h45m31.248505413s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377897122 (594h45m31.248980157s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378294578 (594h45m31.249377613s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378613306 (594h45m31.249696351s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378740998 (594h45m31.249824034s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379196656 (594h45m31.250279691s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379542404 (594h45m31.25062544s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379985118 (594h45m31.251068153s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.380375170 (594h45m31.251458215s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:32.330373077 (594h45m32.201456112s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 10:14:32.819622480 (594h45m32.690705515s) [upload=521369]: Next returned nil segment Dec 30 10:14:32.819634714 (594h45m32.690717759s) [upload=521369]: Waiting for error group managing segments... Dec 30 10:14:33.269321643 (594h45m33.140404688s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.296166971 (594h45m33.167250006s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.566310470 (594h45m33.437393515s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.586516255 (594h45m33.457599301s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.647495109 (594h45m33.518578154s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.111736063 (594h45m33.982819098s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.175714929 (594h45m34.046797964s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.219078501 (594h45m34.090161536s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.255956813 (594h45m34.127039848s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.285262166 (594h45m34.156345211s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.374195129 (594h45m34.245278174s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.445126625 (594h45m34.31620966s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.486740434 (594h45m34.357823469s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842111971 (594h45m34.713195016s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842136328 (594h45m34.713219393s) [seg_pos={0 0}, upload=521369]: Segment reached optimal threshold of 14 pieces. Dec 30 10:14:34.842397145 (594h45m34.71348019s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842406633 (594h45m34.713489688s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842616282 (594h45m34.713699327s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842635157 (594h45m34.713718193s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842689882 (594h45m34.713772917s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842699821 (594h45m34.713782856s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842728506 (594h45m34.713811551s) [seg_pos={0 0}, upload=521369]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:14:34.843191367 (594h45m34.714274412s) [upload=521369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:14:34.843200284 (594h45m34.714283319s) [upload=521369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:14:35.131924283 (594h45m35.003007328s) [upload=521369]: Done uploading segments... Dec 30 10:14:35.131946265 (594h45m35.0030293s) [upload=521369]: Upload finished. err: <nil> Duration: 50.256868195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX....X..X.XX.........X..X......X......X...X...X........X..] duration: 3.420617745s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:16:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420659442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............................X.] duration: 36.380662697s trace: https://jaeger.dev.storj.io/trace/4fb89a9e54e769f5 FAIL (2025-12-30 10:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UhZuvUKofduRzgCWi3WQcfZVuekxBJUXydn3H1ktQ7AmRRvcx, Piece ID: 75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 30 10:14:31.374464365 (594h45m31.2455474s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374502719 (594h45m31.245585754s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374952003 (594h45m31.246035038s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375337639 (594h45m31.246420674s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375427309 (594h45m31.246510354s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375821950 (594h45m31.246904995s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376210771 (594h45m31.247293806s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376573892 (594h45m31.247656937s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377101036 (594h45m31.248184071s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377422367 (594h45m31.248505413s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377897122 (594h45m31.248980157s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378294578 (594h45m31.249377613s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378613306 (594h45m31.249696351s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378740998 (594h45m31.249824034s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379196656 (594h45m31.250279691s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379542404 (594h45m31.25062544s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379985118 (594h45m31.251068153s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.380375170 (594h45m31.251458215s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:32.330373077 (594h45m32.201456112s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 10:14:32.819622480 (594h45m32.690705515s) [upload=521369]: Next returned nil segment Dec 30 10:14:32.819634714 (594h45m32.690717759s) [upload=521369]: Waiting for error group managing segments... Dec 30 10:14:33.269321643 (594h45m33.140404688s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.296166971 (594h45m33.167250006s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.566310470 (594h45m33.437393515s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.586516255 (594h45m33.457599301s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.647495109 (594h45m33.518578154s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.111736063 (594h45m33.982819098s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.175714929 (594h45m34.046797964s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.219078501 (594h45m34.090161536s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.255956813 (594h45m34.127039848s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.285262166 (594h45m34.156345211s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.374195129 (594h45m34.245278174s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.445126625 (594h45m34.31620966s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.486740434 (594h45m34.357823469s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842111971 (594h45m34.713195016s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842136328 (594h45m34.713219393s) [seg_pos={0 0}, upload=521369]: Segment reached optimal threshold of 14 pieces. Dec 30 10:14:34.842397145 (594h45m34.71348019s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842406633 (594h45m34.713489688s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842616282 (594h45m34.713699327s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842635157 (594h45m34.713718193s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842689882 (594h45m34.713772917s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842699821 (594h45m34.713782856s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842728506 (594h45m34.713811551s) [seg_pos={0 0}, upload=521369]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:14:34.843191367 (594h45m34.714274412s) [upload=521369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:14:34.843200284 (594h45m34.714283319s) [upload=521369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:14:35.131924283 (594h45m35.003007328s) [upload=521369]: Done uploading segments... Dec 30 10:14:35.131946265 (594h45m35.0030293s) [upload=521369]: Upload finished. err: <nil> Duration: 36.380678167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX.] duration: 9.911011033s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 9.911031742s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX....X..X.XX.........X..X......X......X...X...X........X.] duration: 4.268364637s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:15:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.268394163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX....X..X.XX.........X..X......X......X...X...X........X] duration: 5.001279263s trace: https://jaeger.dev.storj.io/trace/5ec6e2df9c243e46 FAIL (2025-12-30 10:15:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001298981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....XX] duration: 30.003310295s trace: https://jaeger.dev.storj.io/trace/1f189a9f8487b78f FAIL (2025-12-30 10:15:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:15:01.143711132 (594h46m1.014794177s) [upload=521377]: Starting upload Dec 30 10:15:01.143780925 (594h46m1.01486397s) [upload=521377]: Done starting upload Dec 30 10:15:01.143942602 (594h46m1.015025647s) [upload=521377]: Uploading segments... Dec 30 10:15:01.144039667 (594h46m1.015122702s) [upload=521377]: Got next segment. Inline: false Dec 30 10:15:01.144080194 (594h46m1.015163239s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:15:01.144087598 (594h46m1.015170633s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:15:01.390750386 (594h46m1.261833431s) [seg_pos={0 0}, upload=521377]: Begin upload segment... Dec 30 10:15:01.390834346 (594h46m1.261917391s) [seg_pos={0 0}, upload=521377]: Done begin upload segment. Dec 30 10:15:01.390877969 (594h46m1.261961014s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.390964143 (594h46m1.262047188s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.392342107 (594h46m1.263425152s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393033363 (594h46m1.264116408s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.393522304 (594h46m1.264605349s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394110655 (594h46m1.2651937s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394151612 (594h46m1.265234658s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.394788064 (594h46m1.265871109s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395358781 (594h46m1.266441827s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.395918678 (594h46m1.267001723s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396195124 (594h46m1.267278169s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396393731 (594h46m1.267476777s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396841444 (594h46m1.267924489s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.396907720 (594h46m1.267990766s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397416510 (594h46m1.268499555s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:01.397759222 (594h46m1.268842267s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Uploading piece... Dec 30 10:15:03.032280705 (594h46m2.90336375s) [upload=521377]: Next returned nil segment Dec 30 10:15:03.032300443 (594h46m2.903383488s) [upload=521377]: Waiting for error group managing segments... Dec 30 10:15:03.440128716 (594h46m3.311211761s) [noise=true , address=145.79.227.4:4186 , piece_id=VKDAPHRDDGAFTPFR47HZJOES6HWBQG2TMTXKDE4LVSM5IF5V3XNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.471550829 (594h46m3.342633874s) [noise=true , address=145.79.227.18:4007 , piece_id=222MYXQSTVKKTOSFJNPQNUTXI6MPMDYBHKY463P4J5FCARCGLXNA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.478142571 (594h46m3.349225616s) [noise=true , address=145.79.227.21:4475 , piece_id=Q3KPJJUD66ATMB7X23GKPQQSJBGSR6JTKCPREOISMQW4QRXHHJWA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.481267491 (594h46m3.352350536s) [noise=true , address=145.79.227.45:4230 , piece_id=VZOVXKPYT27LBWMOMVPC2UASCQ4Y4O3DKTSQX3QIE5LGSNA7NYWQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.482468167 (594h46m3.353551212s) [noise=true , address=145.79.227.6:4258 , piece_id=YAE3H6TOFGJ2GIKPLUSI27NWIL2SUWZQHOEGFO6AZRKVDGORHLSA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.488447082 (594h46m3.359530127s) [noise=true , address=145.79.227.8:4329 , piece_id=SEKSXXNR5BP62F6INABJ4WKLBODWXGDR6GJMHG2W7RECJZF7YYLQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.495617195 (594h46m3.366700251s) [noise=true , address=145.79.227.19:4078 , piece_id=7UPDBPVE7RUAMB44AYPD7ARPRZKZL3KBPE4XFJZOFN7ADUZBIAXA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.510382010 (594h46m3.381465055s) [noise=true , address=145.79.227.7:4294 , piece_id=BP4FXQM6MHM3T3V22DD234D7X3YJZEQKOUAUFE3LEOL6RUE26WAA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.524846743 (594h46m3.395929788s) [noise=true , address=145.79.227.9:4365 , piece_id=UYFCFZF5O5Y3TXUBZC754M7NTSQ7AQAXU7JKQTZVU4XF7GCYCTNQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.631477227 (594h46m3.502560272s) [noise=true , address=77.90.31.10:4517 , piece_id=ZQCENPTCVEJLPCTWYRAPIVDOKMACJERECR7HTCDVSMI2EPTJFYVQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.686168547 (594h46m3.557251602s) [noise=true , address=77.90.31.21:4880 , piece_id=C77WTT4MJHJHT6T3GO6RQX3PDF23PTQ6M24JS2QKZPP2RGR3DEMA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.781890388 (594h46m3.652973433s) [noise=true , address=45.91.116.13:4155 , piece_id=X7HR3R43647ZDPV5UZ45JWVYSJ7ZPC46MQWDPFXVQY7QKR4UX4PQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909408485 (594h46m3.78049154s) [noise=true , address=45.91.116.8:4047 , piece_id=EIJWE22C5TU4W6VVME4KBK5RUZWG2NHCW3VBN27J2QE3H2I427WQ, seg_pos={0 0}, upload=521377]: Done uploading piece. err: <nil> Dec 30 10:15:03.909435206 (594h46m3.780518261s) [seg_pos={0 0}, upload=521377]: Segment reached optimal threshold of 13 pieces. Dec 30 10:15:03.909769341 (594h46m3.780852407s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.909779100 (594h46m3.780862155s) [noise=true , address=145.79.227.34:4906 , piece_id=YNFEUVCHE7VOJFKBV2ZRTR2U5BBEKKFT7LPHUW3OTPNEXUOPCCJA, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910289251 (594h46m3.781372297s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33134->45.91.116.4:4937: use of closed network connection Dec 30 10:15:03.910299481 (594h46m3.781382526s) [noise=true , address=45.91.116.4:4937 , piece_id=6LRRZ5ADVDN73OFPK4F6KZL47KWL6GLAHR3DBCRRQISKKKGGY64A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910704873 (594h46m3.781787918s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 10:15:03.910727687 (594h46m3.781810732s) [noise=true , address=145.79.227.44:4193 , piece_id=WVLZOEVZQCGRIOZFJG5IP2JNNCDSRZI4RFJ3O4AJVKRQUI2UID2A, seg_pos={0 0}, upload=521377]: Long tail cancellation due to optimal threshold Dec 30 10:15:03.910767973 (594h46m3.781851018s) [seg_pos={0 0}, upload=521377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:15:03.910882721 (594h46m3.781965767s) [upload=521377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:15:03.910889675 (594h46m3.78197272s) [upload=521377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:15:04.206727800 (594h46m4.077810845s) [upload=521377]: Done uploading segments... Dec 30 10:15:04.206756385 (594h46m4.07783943s) [upload=521377]: Upload finished. err: <nil> Duration: 30.003332999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............................X] duration: 1m0.002431585s trace: https://jaeger.dev.storj.io/trace/4fb89a9e54e769f5 FAIL (2025-12-30 10:15:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UhZuvUKofduRzgCWi3WQcfZVuekxBJUXydn3H1ktQ7AmRRvcx, Piece ID: 75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 30 10:14:31.374464365 (594h45m31.2455474s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374502719 (594h45m31.245585754s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.374952003 (594h45m31.246035038s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375337639 (594h45m31.246420674s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375427309 (594h45m31.246510354s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.375821950 (594h45m31.246904995s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376210771 (594h45m31.247293806s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.376573892 (594h45m31.247656937s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377101036 (594h45m31.248184071s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377422367 (594h45m31.248505413s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.377897122 (594h45m31.248980157s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378294578 (594h45m31.249377613s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378613306 (594h45m31.249696351s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.378740998 (594h45m31.249824034s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379196656 (594h45m31.250279691s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379542404 (594h45m31.25062544s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.379985118 (594h45m31.251068153s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:31.380375170 (594h45m31.251458215s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Uploading piece... Dec 30 10:14:32.330373077 (594h45m32.201456112s) [noise=true , address=193.149.164.143:4700 , piece_id=2YGBEJ6CLWJ5CS5LTOGKSCMOCP7RHPR55F6EJ5XXQIXZU7GB4BJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 10:14:32.819622480 (594h45m32.690705515s) [upload=521369]: Next returned nil segment Dec 30 10:14:32.819634714 (594h45m32.690717759s) [upload=521369]: Waiting for error group managing segments... Dec 30 10:14:33.269321643 (594h45m33.140404688s) [noise=true , address=145.79.227.6:4285 , piece_id=4WIAOF4NL3Q6EKCFBDON76E77RJUWZAVI3OG64RFMT54MT7T7AWA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.296166971 (594h45m33.167250006s) [noise=true , address=64.49.13.154:4932 , piece_id=75WFGJDYW6QH2A5NBTEWRTBSEQPONPJDB6KJDDNTNH7TWWCZRYYA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.566310470 (594h45m33.437393515s) [noise=true , address=77.90.31.18:4834 , piece_id=MGXRDKQ6LRJCMGDNXBFSWAUTTKKVVB3ANRRPDZR5CDMO57HZYYUQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.586516255 (594h45m33.457599301s) [noise=true , address=45.91.116.6:4998 , piece_id=WXLZKNDISLPV4E37EWX3OUSDP2SQ2KUBSXHBWX2PDKA2ZIDOTIHQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:33.647495109 (594h45m33.518578154s) [noise=true , address=45.91.116.8:4059 , piece_id=KOYKCG4B35ZC6BDJF4GEYLRZHNGUTGXUJWKAECZQ2TTZGBEBVKQQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.111736063 (594h45m33.982819098s) [noise=true , address=77.90.31.14:4694 , piece_id=P4FLW4UQXU6CEMIM4U2QAEFATS22GJIUQAZGWU6EKHOJFIKGKOOQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.175714929 (594h45m34.046797964s) [noise=true , address=145.79.227.34:4940 , piece_id=ECW3EYNRP6VZIZQ74NINYJ6TLFQXYCUNOMC45ISNDPABVT5NKAPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.219078501 (594h45m34.090161536s) [noise=true , address=145.79.227.44:4211 , piece_id=MQYPLOJBPKVYPADLNZK7UUANAOC3BQ4QJRYIS2G7HEYWVDVDDPBQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.255956813 (594h45m34.127039848s) [noise=true , address=145.79.227.30:4792 , piece_id=H7F2CIO66SSHSXRKTOWQVXX5LOZRWTLJ4KULRKJ2AGBECDCWOSVQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.285262166 (594h45m34.156345211s) [noise=true , address=145.79.227.33:4895 , piece_id=LLZS6QFSAYKA75HQVJXLKPMBKZ2SHMIQVZFTRRN7GUPW6ISREM4Q, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.374195129 (594h45m34.245278174s) [noise=true , address=145.79.227.35:4504 , piece_id=PUAG4UMO3QS6QNUOQNT2VGCP2UPZD33A7TVTBXDXFQZTOJC7CJPA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.445126625 (594h45m34.31620966s) [noise=true , address=77.90.31.12:4603 , piece_id=VWMPASOL5QJBU2FHSH4GO4J4JEQVF4NAGEHJEJDAYRUGBJRNOLEQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.486740434 (594h45m34.357823469s) [noise=true , address=193.149.164.143:4704 , piece_id=O6NCIMUY2M6MRDGWPN4A2I35MZMBNYZKWT42XM3UJCP7E3NBNBJA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842111971 (594h45m34.713195016s) [noise=true , address=145.79.227.49:4363 , piece_id=S5NHRXTMSMEFST6BRPAGE767KBGW75YJVZFBTVPXTIVW7VISS75A, seg_pos={0 0}, upload=521369]: Done uploading piece. err: <nil> Dec 30 10:14:34.842136328 (594h45m34.713219393s) [seg_pos={0 0}, upload=521369]: Segment reached optimal threshold of 14 pieces. Dec 30 10:14:34.842397145 (594h45m34.71348019s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842406633 (594h45m34.713489688s) [noise=true , address=77.90.31.18:4833 , piece_id=QDVA6DVTQM254LRDA3Q4DKMPUXBODPV4KSHYF47UTJ653QMIVPYQ, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842616282 (594h45m34.713699327s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842635157 (594h45m34.713718193s) [noise=true , address=45.91.116.13:4186 , piece_id=C3XEE4ESCUJCEI6NVQUYYUP7FE6YWXK6OK4ZRVB7UGTH252KSJLA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842689882 (594h45m34.713772917s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: expected piece hash; context canceled; EOF Dec 30 10:14:34.842699821 (594h45m34.713782856s) [noise=true , address=77.90.31.11:4582 , piece_id=Q7OZTH33I5Q35NTLNAACAZ2OUQMO4W6FRW6OSCBEWG3ZRUTKCZIA, seg_pos={0 0}, upload=521369]: Long tail cancellation due to optimal threshold Dec 30 10:14:34.842728506 (594h45m34.713811551s) [seg_pos={0 0}, upload=521369]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:14:34.843191367 (594h45m34.714274412s) [upload=521369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:14:34.843200284 (594h45m34.714283319s) [upload=521369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:14:35.131924283 (594h45m35.003007328s) [upload=521369]: Done uploading segments... Dec 30 10:14:35.131946265 (594h45m35.0030293s) [upload=521369]: Upload finished. err: <nil> Duration: 1m0.002635884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XX....X..X.XX.........X..X......X......X...X...X........] duration: 3.426955584s trace: https://jaeger.dev.storj.io/trace/4546990854ed2716 FAIL (2025-12-30 10:14:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426981574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X....X.X..X.X..............XX.X.XX.X.X.X.XX.X.....X] duration: 30.003696175s trace: https://jaeger.dev.storj.io/trace/85527966813b08d FAIL (2025-12-30 10:14:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: M7GKCA4ADT4TPO7AN2IYVGM4IJ56U67CVGQSCAJY2UJPGO7C2YPA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AOOTF5AXYTVMUFEZDF5MFJ7T5SMRMXNDQKNNP5GWNWOJHMJ6EZMQ) context deadline exceeded ======LOGS====== c 30 10:13:53.579891007 (594h44m53.450974052s) [upload=521361]: Starting upload Dec 30 10:13:53.579955931 (594h44m53.451039016s) [upload=521361]: Done starting upload Dec 30 10:13:53.580079476 (594h44m53.451162521s) [upload=521361]: Uploading segments... Dec 30 10:13:53.580222018 (594h44m53.451305053s) [upload=521361]: Got next segment. Inline: false Dec 30 10:13:53.580267324 (594h44m53.451350359s) [upload=521361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:13:53.580274537 (594h44m53.451357613s) [upload=521361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:13:53.820985483 (594h44m53.692068528s) [seg_pos={0 0}, upload=521361]: Begin upload segment... Dec 30 10:13:53.821043412 (594h44m53.692126458s) [seg_pos={0 0}, upload=521361]: Done begin upload segment. Dec 30 10:13:53.821088960 (594h44m53.692171995s) [noise=true , address=145.79.227.15:4875 , piece_id=OOCYS4HUIF34KB4KZB5QTDNZHH2T4ZLKVVRGUT3YCTF5274GSQIA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.821116271 (594h44m53.692199316s) [noise=true , address=77.90.31.18:4805 , piece_id=M7GKCA4ADT4TPO7AN2IYVGM4IJ56U67CVGQSCAJY2UJPGO7C2YPA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.821620512 (594h44m53.692703557s) [noise=true , address=145.79.227.47:4266 , piece_id=2NA5CVVDJ6Q5NPLT45S52WTJWUP2DY637543NIPDCRYIGVY77PSQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.822043427 (594h44m53.693126472s) [noise=true , address=145.79.227.16:4910 , piece_id=SH3BYGBAELYAAA37RV4SMDPQZCZKDFHFJDZMORJJFFKQVEJHDSGQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.822442096 (594h44m53.693525141s) [noise=true , address=45.91.116.13:4153 , piece_id=AOOTF5AXYTVMUFEZDF5MFJ7T5SMRMXNDQKNNP5GWNWOJHMJ6EZMQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.822871664 (594h44m53.693954709s) [noise=true , address=145.79.227.45:4229 , piece_id=QN2DZNCTYR27642GLZJDTY3YS6KUGGC2K72NRZWW247AC6BD5AAA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.823497556 (594h44m53.694580591s) [noise=true , address=45.91.116.6:4973 , piece_id=BK5XIOIN3GADHKU7B6KIYULXT2X34WM7M5DNCPNL3YSKYQ2A254A, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.823993390 (594h44m53.695076435s) [noise=true , address=45.91.116.10:4082 , piece_id=PZR32XYQI46JTKJJVE2HQV2OEP6SMFULPXRQ6NDEXWCWQX7OJHFA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.825026287 (594h44m53.696109332s) [noise=true , address=45.91.116.9:4263 , piece_id=CBUEES5L3WJITPNS2MGHXH6SIDOSWCUGLCQJD73Q665ZTPVSBIEA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.825399136 (594h44m53.696482171s) [noise=true , address=145.79.227.32:4834 , piece_id=4JVXPA65M7NWOWQ3WWGF2M4NGLX7RQBVM3M57NEL5I2VMZ6H5SRA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.827149119 (594h44m53.698232164s) [noise=true , address=145.79.227.27:4691 , piece_id=LZEEY4P7X2HT3ZZVEQ5EW6NRVDYA43ETPJAIP2NEZAPL76O3SUWQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.829049918 (594h44m53.700132963s) [noise=true , address=45.91.116.7:4011 , piece_id=VRLS4Y6CBK2QB5L5O6MAAS2E6EQPGBWPHGISRCIXYDJRZKGYYB5A, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.830578379 (594h44m53.701661424s) [noise=true , address=77.90.31.21:4877 , piece_id=SAHJ4IELEIRLPU4LDU7MQ4VJWKLFIPY4EIWQE4FE5PB7E7IVUIRQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.832121418 (594h44m53.703204463s) [noise=true , address=145.79.227.35:4483 , piece_id=GIPIVSCSSSP2F7MEHTZ2DZXELSLBNLCJKOGEFN4M6E3A7P7GWNMA, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.833250657 (594h44m53.704333693s) [noise=true , address=145.79.227.7:4295 , piece_id=C3X76P2TIUUNENZIZDGE6G5GOABGRXMQ3IJ6DXUR5TYDDG6J447Q, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:53.833609420 (594h44m53.704692465s) [noise=true , address=45.91.116.11:4120 , piece_id=CMLGTEIV7I2MI3PNDX5LQA6WIRPUU4IEZMVHTUNRR55BJOMYCIJQ, seg_pos={0 0}, upload=521361]: Uploading piece... Dec 30 10:13:55.521281964 (594h44m55.392364999s) [upload=521361]: Next returned nil segment Dec 30 10:13:55.521296531 (594h44m55.392379576s) [upload=521361]: Waiting for error group managing segments... Dec 30 10:13:55.807258471 (594h44m55.678341516s) [noise=true , address=145.79.227.45:4229 , piece_id=QN2DZNCTYR27642GLZJDTY3YS6KUGGC2K72NRZWW247AC6BD5AAA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.813949893 (594h44m55.685032938s) [noise=true , address=145.79.227.15:4875 , piece_id=OOCYS4HUIF34KB4KZB5QTDNZHH2T4ZLKVVRGUT3YCTF5274GSQIA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.839729953 (594h44m55.710812988s) [noise=true , address=145.79.227.16:4910 , piece_id=SH3BYGBAELYAAA37RV4SMDPQZCZKDFHFJDZMORJJFFKQVEJHDSGQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.843776648 (594h44m55.714859683s) [noise=true , address=145.79.227.27:4691 , piece_id=LZEEY4P7X2HT3ZZVEQ5EW6NRVDYA43ETPJAIP2NEZAPL76O3SUWQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.844186248 (594h44m55.715269293s) [noise=true , address=145.79.227.7:4295 , piece_id=C3X76P2TIUUNENZIZDGE6G5GOABGRXMQ3IJ6DXUR5TYDDG6J447Q, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.856736325 (594h44m55.72781937s) [noise=true , address=145.79.227.47:4266 , piece_id=2NA5CVVDJ6Q5NPLT45S52WTJWUP2DY637543NIPDCRYIGVY77PSQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.859657648 (594h44m55.730740693s) [noise=true , address=145.79.227.32:4834 , piece_id=4JVXPA65M7NWOWQ3WWGF2M4NGLX7RQBVM3M57NEL5I2VMZ6H5SRA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:55.993136875 (594h44m55.86421991s) [noise=true , address=77.90.31.18:4805 , piece_id=M7GKCA4ADT4TPO7AN2IYVGM4IJ56U67CVGQSCAJY2UJPGO7C2YPA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.151840894 (594h44m56.022923929s) [noise=true , address=45.91.116.6:4973 , piece_id=BK5XIOIN3GADHKU7B6KIYULXT2X34WM7M5DNCPNL3YSKYQ2A254A, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.170265258 (594h44m56.041348303s) [noise=true , address=45.91.116.13:4153 , piece_id=AOOTF5AXYTVMUFEZDF5MFJ7T5SMRMXNDQKNNP5GWNWOJHMJ6EZMQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.180882705 (594h44m56.05196574s) [noise=true , address=45.91.116.7:4011 , piece_id=VRLS4Y6CBK2QB5L5O6MAAS2E6EQPGBWPHGISRCIXYDJRZKGYYB5A, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.228506578 (594h44m56.099589623s) [noise=true , address=45.91.116.11:4120 , piece_id=CMLGTEIV7I2MI3PNDX5LQA6WIRPUU4IEZMVHTUNRR55BJOMYCIJQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.457607426 (594h44m56.328690471s) [noise=true , address=45.91.116.9:4263 , piece_id=CBUEES5L3WJITPNS2MGHXH6SIDOSWCUGLCQJD73Q665ZTPVSBIEA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: <nil> Dec 30 10:13:56.457633785 (594h44m56.32871682s) [seg_pos={0 0}, upload=521361]: Segment reached optimal threshold of 13 pieces. Dec 30 10:13:56.457884362 (594h44m56.328967407s) [noise=true , address=45.91.116.10:4082 , piece_id=PZR32XYQI46JTKJJVE2HQV2OEP6SMFULPXRQ6NDEXWCWQX7OJHFA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 30 10:13:56.457895233 (594h44m56.328978278s) [noise=true , address=45.91.116.10:4082 , piece_id=PZR32XYQI46JTKJJVE2HQV2OEP6SMFULPXRQ6NDEXWCWQX7OJHFA, seg_pos={0 0}, upload=521361]: Long tail cancellation due to optimal threshold Dec 30 10:13:56.457943425 (594h44m56.32902647s) [noise=true , address=77.90.31.21:4877 , piece_id=SAHJ4IELEIRLPU4LDU7MQ4VJWKLFIPY4EIWQE4FE5PB7E7IVUIRQ, seg_pos={0 0}, upload=521361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:52166->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:52166->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52166->77.90.31.21:4877: use of closed network connection Dec 30 10:13:56.457960438 (594h44m56.329043483s) [noise=true , address=77.90.31.21:4877 , piece_id=SAHJ4IELEIRLPU4LDU7MQ4VJWKLFIPY4EIWQE4FE5PB7E7IVUIRQ, seg_pos={0 0}, upload=521361]: Long tail cancellation due to optimal threshold Dec 30 10:13:56.458037634 (594h44m56.329120679s) [noise=true , address=145.79.227.35:4483 , piece_id=GIPIVSCSSSP2F7MEHTZ2DZXELSLBNLCJKOGEFN4M6E3A7P7GWNMA, seg_pos={0 0}, upload=521361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 10:13:56.458045439 (594h44m56.329128484s) [noise=true , address=145.79.227.35:4483 , piece_id=GIPIVSCSSSP2F7MEHTZ2DZXELSLBNLCJKOGEFN4M6E3A7P7GWNMA, seg_pos={0 0}, upload=521361]: Long tail cancellation due to optimal threshold Dec 30 10:13:56.458081878 (594h44m56.329164913s) [seg_pos={0 0}, upload=521361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:13:56.458162552 (594h44m56.329245587s) [upload=521361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:13:56.458168083 (594h44m56.329251128s) [upload=521361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:13:56.748105272 (594h44m56.619188307s) [upload=521361]: Done uploading segments... Dec 30 10:13:56.748126652 (594h44m56.619209697s) [upload=521361]: Upload finished. err: <nil> Duration: 30.003718216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 41.71564492s trace: https://jaeger.dev.storj.io/trace/3195355f13f05d36 FAIL (2025-12-30 10:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.12:4608 , piece_id=JQ74XBBKJFVURSNSLTAFCOI5O254QZWMB3NFQ7WYJP6OJ2OWMSUA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.067019971 (594h19m17.938103006s) [noise=true , address=145.79.227.44:4213 , piece_id=YSD4J4VYC3YWB5GSSGY6LWELBAXX62VVWFEGDT7ROFV6MWYEDRRQ, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.067303912 (594h19m17.938386947s) [noise=true , address=77.90.31.17:4779 , piece_id=GLSAXG7T6S3JTFVS7HQDISHG3UDV7TUIYO5L2JOQSAVWOQYUQHQQ, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.068459432 (594h19m17.939542467s) [noise=true , address=145.79.227.29:4951 , piece_id=KAF5HJKOCR5CWF5NLYHF4AEMOHCXKCF5IB4AYSMKX5TVSJ56ATFA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.068997457 (594h19m17.940080502s) [noise=true , address=145.79.227.33:4877 , piece_id=QLY5EMHVDQYWKF4EDJTVDC2WI66VMY6EUU6BMDS4KB6R4ZEZT62Q, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.069543326 (594h19m17.940626371s) [noise=true , address=77.90.31.14:4674 , piece_id=6GZYDZAGRV5EE7C4ABVQI3FJHMNIMPKPN7MHF3W5CAG7YSDECU6Q, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.070650734 (594h19m17.941733779s) [noise=true , address=45.91.116.5:4308 , piece_id=JKYGMAAW5KR7MRDJI4HXRXO4IM3A6LH6TFYA5LNRSTRCQ3S6AG3A, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.070875393 (594h19m17.941958428s) [noise=true , address=64.49.13.169:4715 , piece_id=Y6TATIERO6IKFUFSX3XXIQ2BRDEROCNHNQJRYRJIXOG6W4QWZLVA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.071784003 (594h19m17.942867048s) [noise=true , address=77.90.31.10:4538 , piece_id=35EZE43JQLU3QS2V4ADSBVVSKY4ADW7L3HMZ6ATDKQVBM3LFDH5A, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.073711273 (594h19m17.944794318s) [noise=true , address=64.49.13.26:4946 , piece_id=VKIRJBY4ERVNZLWECAXQ5XBZSHQXNWLICXL2SH7L6YJTAOMQ4B5A, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.075235234 (594h19m17.946318279s) [noise=true , address=145.79.227.40:4053 , piece_id=JNLIJDJ6UVS3SXJMOH4QRGA2VA6B2AVKRMIB44NOORBMAE6GMUWA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.076829440 (594h19m17.947912515s) [noise=true , address=193.149.164.143:4699 , piece_id=P5SMN6VOYPZXI2YMBY6EMPUAGFE3ABPEJ3NSYYFYGID4MD4PDNQA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.078272018 (594h19m17.949355063s) [noise=true , address=145.79.227.34:4928 , piece_id=EIMXDAKABH73MGYGDOLEZ3NPMJ6DTIQMVENWBGZ4WWTCRMWEQZZA, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.078661900 (594h19m17.949744935s) [noise=true , address=77.90.31.19:4849 , piece_id=CPULLHYLFYY62ZMJWHDXF5TAXTGJYDXWZF4GQEHZBZTNSTFSNPTQ, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.080047989 (594h19m17.951131024s) [noise=true , address=77.90.31.21:4901 , piece_id=GND6INAI7OCC3F5YVVBDBUQ4MKP6NUNOP3GTNKYPRDKHDLZ3HB7Q, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:18.080417162 (594h19m17.951500207s) [noise=true , address=145.79.227.30:4765 , piece_id=UZK5TI3566QAEMMHWFRV2IEPZFAZXJTPCQ7YRS5KZVUVXFTTQNEQ, seg_pos={0 0}, upload=520994]: Uploading piece... Dec 30 09:48:19.524439219 (594h19m19.395522254s) [upload=520994]: Next returned nil segment Dec 30 09:48:19.524450460 (594h19m19.395533495s) [upload=520994]: Waiting for error group managing segments... Dec 30 09:48:19.918997407 (594h19m19.790080443s) [noise=true , address=145.79.227.13:4824 , piece_id=M3SJB242OYQKPPNRZQRYBJUY5IIW57IBIRNUYTL7ZGP72G4PDQYQ, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:19.996316117 (594h19m19.867399162s) [noise=true , address=64.49.13.26:4946 , piece_id=VKIRJBY4ERVNZLWECAXQ5XBZSHQXNWLICXL2SH7L6YJTAOMQ4B5A, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:19.998204242 (594h19m19.869287287s) [noise=true , address=145.79.227.29:4951 , piece_id=KAF5HJKOCR5CWF5NLYHF4AEMOHCXKCF5IB4AYSMKX5TVSJ56ATFA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.091699153 (594h19m19.962782198s) [noise=true , address=77.90.31.14:4674 , piece_id=6GZYDZAGRV5EE7C4ABVQI3FJHMNIMPKPN7MHF3W5CAG7YSDECU6Q, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.094776833 (594h19m19.965859868s) [noise=true , address=77.90.31.21:4901 , piece_id=GND6INAI7OCC3F5YVVBDBUQ4MKP6NUNOP3GTNKYPRDKHDLZ3HB7Q, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.125557174 (594h19m19.996640209s) [noise=true , address=77.90.31.18:4821 , piece_id=JJ2MBP4YE4ASKT34AS43HJH76O6ZS25QQM7E6JPDPYQ2HGVRXNRA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.190831969 (594h19m20.061915015s) [noise=true , address=77.90.31.19:4849 , piece_id=CPULLHYLFYY62ZMJWHDXF5TAXTGJYDXWZF4GQEHZBZTNSTFSNPTQ, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.202189506 (594h19m20.073272551s) [noise=true , address=77.90.31.17:4779 , piece_id=GLSAXG7T6S3JTFVS7HQDISHG3UDV7TUIYO5L2JOQSAVWOQYUQHQQ, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.256867624 (594h19m20.127950659s) [noise=true , address=77.90.31.10:4538 , piece_id=35EZE43JQLU3QS2V4ADSBVVSKY4ADW7L3HMZ6ATDKQVBM3LFDH5A, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.287664355 (594h19m20.15874739s) [noise=true , address=77.90.31.12:4608 , piece_id=JQ74XBBKJFVURSNSLTAFCOI5O254QZWMB3NFQ7WYJP6OJ2OWMSUA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.827809488 (594h19m20.698892523s) [noise=true , address=145.79.227.30:4765 , piece_id=UZK5TI3566QAEMMHWFRV2IEPZFAZXJTPCQ7YRS5KZVUVXFTTQNEQ, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.877420858 (594h19m20.748503903s) [noise=true , address=145.79.227.44:4213 , piece_id=YSD4J4VYC3YWB5GSSGY6LWELBAXX62VVWFEGDT7ROFV6MWYEDRRQ, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:20.990976158 (594h19m20.862059203s) [noise=true , address=193.149.164.143:4699 , piece_id=P5SMN6VOYPZXI2YMBY6EMPUAGFE3ABPEJ3NSYYFYGID4MD4PDNQA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:21.006244470 (594h19m20.877327506s) [noise=true , address=145.79.227.33:4877 , piece_id=QLY5EMHVDQYWKF4EDJTVDC2WI66VMY6EUU6BMDS4KB6R4ZEZT62Q, seg_pos={0 0}, upload=520994]: Done uploading piece. err: <nil> Dec 30 09:48:21.006262295 (594h19m20.87734533s) [seg_pos={0 0}, upload=520994]: Segment reached optimal threshold of 14 pieces. Dec 30 09:48:21.006858620 (594h19m20.877941655s) [noise=true , address=145.79.227.40:4053 , piece_id=JNLIJDJ6UVS3SXJMOH4QRGA2VA6B2AVKRMIB44NOORBMAE6GMUWA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 09:48:21.011274880 (594h19m20.882357925s) [noise=true , address=145.79.227.40:4053 , piece_id=JNLIJDJ6UVS3SXJMOH4QRGA2VA6B2AVKRMIB44NOORBMAE6GMUWA, seg_pos={0 0}, upload=520994]: Long tail cancellation due to optimal threshold Dec 30 09:48:21.011309486 (594h19m20.882392531s) [noise=true , address=145.79.227.34:4928 , piece_id=EIMXDAKABH73MGYGDOLEZ3NPMJ6DTIQMVENWBGZ4WWTCRMWEQZZA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B6ymfqNvqiDsAbkF8H4ExxEuYaYb7gYQ45Y5GF4PGAAwmjbyd): protocol: expected piece hash; context canceled; EOF Dec 30 09:48:21.011320236 (594h19m20.882403271s) [noise=true , address=145.79.227.34:4928 , piece_id=EIMXDAKABH73MGYGDOLEZ3NPMJ6DTIQMVENWBGZ4WWTCRMWEQZZA, seg_pos={0 0}, upload=520994]: Long tail cancellation due to optimal threshold Dec 30 09:48:21.011340335 (594h19m20.88242338s) [noise=true , address=45.91.116.5:4308 , piece_id=JKYGMAAW5KR7MRDJI4HXRXO4IM3A6LH6TFYA5LNRSTRCQ3S6AG3A, seg_pos={0 0}, upload=520994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 09:48:21.011349221 (594h19m20.882432266s) [noise=true , address=45.91.116.5:4308 , piece_id=JKYGMAAW5KR7MRDJI4HXRXO4IM3A6LH6TFYA5LNRSTRCQ3S6AG3A, seg_pos={0 0}, upload=520994]: Long tail cancellation due to optimal threshold Dec 30 09:48:21.011359190 (594h19m20.882442225s) [noise=true , address=64.49.13.169:4715 , piece_id=Y6TATIERO6IKFUFSX3XXIQ2BRDEROCNHNQJRYRJIXOG6W4QWZLVA, seg_pos={0 0}, upload=520994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:42790->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:42790->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42790->64.49.13.169:4715: use of closed network connection Dec 30 09:48:21.011406901 (594h19m20.882489946s) [noise=true , address=64.49.13.169:4715 , piece_id=Y6TATIERO6IKFUFSX3XXIQ2BRDEROCNHNQJRYRJIXOG6W4QWZLVA, seg_pos={0 0}, upload=520994]: Long tail cancellation due to optimal threshold Dec 30 09:48:21.011481854 (594h19m20.882564899s) [seg_pos={0 0}, upload=520994]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:48:21.011589038 (594h19m20.882672083s) [upload=520994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:48:21.011596151 (594h19m20.882679196s) [upload=520994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:48:21.309708250 (594h19m21.180791285s) [upload=520994]: Done uploading segments... Dec 30 09:48:21.309729050 (594h19m21.180812085s) [upload=520994]: Upload finished. err: <nil> Duration: 41.715674015s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)