Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 451944 reports (Page 392 of 9039)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXX] duration: 30.001306282s trace: https://jaeger.dev.storj.io/trace/304e1a84b0fad749 FAIL (2026-01-11 15:01:16 UTC) orbiter warning: context deadline exceeded Duration: 30.001351889s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...X............................X..X..................] duration: 5.737993891s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 15:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.738014329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 5.597669397s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.597685917s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 4.562016747s trace: https://jaeger.dev.storj.io/trace/6ef8a26f42e452a1 FAIL (2026-01-11 15:00:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.562035414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XX.XX...XX..X.XX..X.........XX...X....XX.X..X......X.] duration: 23.12559391s trace: https://jaeger.dev.storj.io/trace/7af1a61f6e1e4d30 FAIL (2026-01-11 15:00:39 UTC) orbiter warning: context deadline exceeded Duration: 23.125609651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XX] duration: 30.001615379s trace: https://jaeger.dev.storj.io/trace/2401c0c757176e9 FAIL (2026-01-11 15:00:36 UTC) orbiter warning: context deadline exceeded Duration: 30.001646239s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...X............................X..X.................] duration: 5.768727337s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 15:00:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.768750923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 5.877716404s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.877751501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [........XX.XX...XX..X.XX..X.........XX...X....XX.X..X......X] duration: 1m0.001693367s trace: https://jaeger.dev.storj.io/trace/7af1a61f6e1e4d30 FAIL (2026-01-11 15:00:16 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001718976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001071892s trace: https://jaeger.dev.storj.io/trace/6ef8a26f42e452a1 FAIL (2026-01-11 15:00:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001112249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 5.742217885s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:00:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.742248855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.X] duration: 30.001463836s trace: https://jaeger.dev.storj.io/trace/1953770fc1501377 FAIL (2026-01-11 14:59:54 UTC) orbiter warning: context deadline exceeded Duration: 30.001484846s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...X............................X..X................] duration: 5.794916583s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:59:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.794942053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 5.593109019s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:59:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.593139677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 3.806616036s trace: https://jaeger.dev.storj.io/trace/71f4b91cf0bdf01 FAIL (2026-01-11 14:59:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.806634213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.........XX.XX...XX..X.XX..X.........XX...X....XX.X..X......] duration: 26.787333341s trace: https://jaeger.dev.storj.io/trace/5dc23ee1d0613fa3 FAIL (2026-01-11 14:58:44 UTC) orbiter warning: context deadline exceeded Duration: 26.78735892s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.] duration: 26.014911169s trace: https://jaeger.dev.storj.io/trace/79c04c08ab785afa FAIL (2026-01-11 14:58:42 UTC) orbiter warning: context deadline exceeded Duration: 26.014935215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 6.121834433s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:58:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 6.121863147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000558174s trace: https://jaeger.dev.storj.io/trace/71f4b91cf0bdf01 FAIL (2026-01-11 14:58:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000569665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..........XX.XX...XX..X.XX..X.........XX...X....XX.X..X.....] duration: 30.529079284s trace: https://jaeger.dev.storj.io/trace/5dc23ee1d0613fa3 FAIL (2026-01-11 14:58:09 UTC) orbiter warning: context deadline exceeded Duration: 30.529111244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...X............................X..X...............] duration: 5.782790264s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:58:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.782818168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 6.402074495s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:58:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 6.402099575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...X............................X..X..............] duration: 5.911542759s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:57:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.911571845s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000756529s trace: https://jaeger.dev.storj.io/trace/5536b7b5a8e6e549 FAIL (2026-01-11 14:57:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000778139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX] duration: 30.001404736s trace: https://jaeger.dev.storj.io/trace/79c04c08ab785afa FAIL (2026-01-11 14:57:36 UTC) orbiter warning: context deadline exceeded Duration: 30.001420886s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001626966s trace: https://jaeger.dev.storj.io/trace/6490db7fb8f6a5f9 FAIL (2026-01-11 14:57:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001652234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X...X............................X..X.............] duration: 5.760051945s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.760100097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X..........XX.XX...XX..X.XX..X.........XX...X....XX.X..X....] duration: 33.639723304s trace: https://jaeger.dev.storj.io/trace/5dc23ee1d0613fa3 FAIL (2026-01-11 14:56:57 UTC) orbiter warning: context deadline exceeded Duration: 33.639749262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 6.027372115s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:56:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 6.027400408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X...X............................X..X............] duration: 5.840333303s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:56:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.840456127s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXX] duration: 30.001244694s trace: https://jaeger.dev.storj.io/trace/2d74ec9cb81226e4 FAIL (2026-01-11 14:56:24 UTC) orbiter warning: context deadline exceeded Duration: 30.001267918s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001156151s trace: https://jaeger.dev.storj.io/trace/3ce5dd47669b5b00 FAIL (2026-01-11 14:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001183142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.563292362s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:56:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.563321838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X..........XX.XX...XX..X.XX..X.........XX...X....XX.X..X...] duration: 31.10507448s trace: https://jaeger.dev.storj.io/trace/5dc23ee1d0613fa3 FAIL (2026-01-11 14:55:59 UTC) orbiter warning: context deadline exceeded Duration: 31.10509974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000602418s trace: https://jaeger.dev.storj.io/trace/15b3e7732560018d FAIL (2026-01-11 14:55:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000625091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXX] duration: 30.001722173s trace: https://jaeger.dev.storj.io/trace/9ea6831f4d35326 FAIL (2026-01-11 14:55:39 UTC) orbiter warning: context deadline exceeded Duration: 30.001732462s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X...X............................X..X...........] duration: 5.806710563s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:55:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.806735029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X...X............................X..X..........] duration: 5.864733045s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:55:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.86829085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [X..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXX] duration: 30.001204301s trace: https://jaeger.dev.storj.io/trace/343d23efa60fc1a3 FAIL (2026-01-11 14:55:09 UTC) orbiter warning: context deadline exceeded Duration: 30.001226482s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001830712s trace: https://jaeger.dev.storj.io/trace/f20bec861bbdfc1 FAIL (2026-01-11 14:55:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001854669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X...X............................X..X.........] duration: 5.875020964s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.875045591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.99835602s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.998379103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 12.778281466s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:54:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.778308907s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X..........XX.XX...XX..X.XX..X.........XX...X....XX.X..X..] duration: 24.873339896s trace: https://jaeger.dev.storj.io/trace/5dc23ee1d0613fa3 FAIL (2026-01-11 14:54:43 UTC) orbiter warning: context deadline exceeded Duration: 24.873364222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X...X............................X..X........] duration: 5.744751723s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:54:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.744764627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.852993198s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 14:54:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.853042794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 22.479489446s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:54:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.479513272s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000733776s trace: https://jaeger.dev.storj.io/trace/62af489911fa056c FAIL (2026-01-11 14:54:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000755356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X...X............................X..X.......] duration: 5.821635895s trace: https://jaeger.dev.storj.io/trace/3d7950d1a686ac04 FAIL (2026-01-11 14:53:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 14:49:13.739112613 (887h20m13.610195658s) [upload=776500]: Starting upload Jan 11 14:49:13.739196893 (887h20m13.610279928s) [upload=776500]: Done starting upload Jan 11 14:49:13.739293567 (887h20m13.610376602s) [upload=776500]: Uploading segments... Jan 11 14:49:13.739425709 (887h20m13.610508754s) [upload=776500]: Got next segment. Inline: false Jan 11 14:49:13.739474071 (887h20m13.610557106s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:49:13.739479831 (887h20m13.610562877s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:49:13.983075730 (887h20m13.854158775s) [seg_pos={0 0}, upload=776500]: Begin upload segment... Jan 11 14:49:13.983160221 (887h20m13.854243266s) [seg_pos={0 0}, upload=776500]: Done begin upload segment. Jan 11 14:49:13.983204275 (887h20m13.854287311s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983221207 (887h20m13.854304252s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983269298 (887h20m13.854352333s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.983710649 (887h20m13.854793694s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984149916 (887h20m13.855232951s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.984571469 (887h20m13.855654514s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985266502 (887h20m13.856349547s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.985994669 (887h20m13.857077704s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.986847152 (887h20m13.857930197s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.987552695 (887h20m13.85863573s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.988247488 (887h20m13.859330533s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.989145678 (887h20m13.860228713s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.990503694 (887h20m13.861586729s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.992246222 (887h20m13.863329267s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.994388842 (887h20m13.865471887s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:13.999326654 (887h20m13.870409689s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Uploading piece... Jan 11 14:49:16.784920925 (887h20m16.65600396s) [upload=776500]: Next returned nil segment Jan 11 14:49:16.784937556 (887h20m16.656020591s) [upload=776500]: Waiting for error group managing segments... Jan 11 14:49:17.245630364 (887h20m17.116713399s) [noise=true , address=145.79.227.23:4545 , piece_id=ZWQ4R4O4SQ2KLLQCAF6NTUGXYNGTNORFENU23ZCDFFDNSQHSHI4A, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:17.356936736 (887h20m17.228019781s) [noise=true , address=145.79.227.6:4259 , piece_id=6DLHR2QRLIOPTBBJ3GI2KEUQQHT5YZJRXMG2UDHF37YWUVCVVKPQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.193080625 (887h20m18.06416367s) [noise=true , address=145.79.227.32:4836 , piece_id=GLQ6XMSNCCU52S73DGUJZTRMRTYW43RUJUC677DGWWYE3WRKRRCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:18.407801010 (887h20m18.278884056s) [noise=true , address=145.79.227.20:4439 , piece_id=HFRTB6S7XWTOSGPV3OJCQH74E773JXGS67FU22QGGLURUL6XKWXA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.252346869 (887h20m19.123429914s) [noise=true , address=145.79.227.17:4972 , piece_id=CZ3FOWREYXVDHMGVANZ6FF5OZV2CPA3IV7OC6IV7AX2LERGJUONA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:19.431324035 (887h20m19.30240707s) [noise=true , address=145.79.227.9:4365 , piece_id=X3NNIF6OOBI53ACZB4VB4UBUGRYPPPPGVKLH5ITFNRFU53O2AYSA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:20.580368786 (887h20m20.451451821s) [noise=true , address=145.79.227.45:4231 , piece_id=VGBDIVIBPKF2KVNI23IN6EAYQ3POEB43AQNO4BD2OQHXHMEGQPEQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:21.905000122 (887h20m21.776083157s) [noise=true , address=145.79.227.29:4943 , piece_id=4KMYFGFZHADUXXWWKQIFGQ62ONGDUSWVKUUSV3MXUYI2SK3JD6WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:23.340341679 (887h20m23.211424714s) [noise=true , address=145.79.227.41:4086 , piece_id=T3PF2I33OJULW4NL4FOVTTFCD4WSHZBHEK4CQJW666OX6ZCAF5XA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:26.312610568 (887h20m26.183693603s) [noise=true , address=145.79.227.5:4221 , piece_id=YIPH4XQF523IMSDIIUP6ZCHR7XKFRQTXAEVQRRNHKP3EA423XQCA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:30.961077935 (887h20m30.83216097s) [noise=true , address=145.79.227.52:4447 , piece_id=UZT4G3KM5BJN5VCJ7V2YLKPIKDO2BD5GODU3XQR2TOLGISSYPDUQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:31.334562579 (887h20m31.205645624s) [noise=true , address=45.91.116.5:4300 , piece_id=EDVQFBUVFMH32X7VSD6MLXBS2BBDGCTIRRQ653AY327ZIY3TNZCQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996134480 (887h20m38.867217515s) [noise=true , address=145.79.227.33:4870 , piece_id=YXA5K7JKY7LJWU3MJLV4WYFL5KFAHWDYX4H3ICPLCFFXEF5KFIJA, seg_pos={0 0}, upload=776500]: Done uploading piece. err: <nil> Jan 11 14:49:38.996155710 (887h20m38.867238745s) [seg_pos={0 0}, upload=776500]: Segment reached optimal threshold of 13 pieces. Jan 11 14:49:38.996465410 (887h20m38.867548455s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: read tcp 188.166.225.170:41168->145.79.227.48:4303: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:38.996474557 (887h20m38.867557602s) [noise=true , address=145.79.227.48:4303 , piece_id=JFIRCJ44KA6GKSA3IYGMA3MYL6ADRSE2VJDIRDQLUAWVESFQC3JQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996497441 (887h20m38.867580476s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:38.996505085 (887h20m38.86758812s) [noise=true , address=145.79.227.36:4517 , piece_id=L7YE4XLUN3J624O7TWNRLZMTFQFDNY5RU7ISJDIAB4K7SBCI37WQ, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996780419 (887h20m38.867863455s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: context canceled; context canceled Jan 11 14:49:38.996791381 (887h20m38.867874426s) [noise=true , address=145.79.227.42:4122 , piece_id=CLPAG6SJRTS5KDFPDX2COJ76DTAKCVDWQEBFM3PDZ2PQEZPO6X4Q, seg_pos={0 0}, upload=776500]: Long tail cancellation due to optimal threshold Jan 11 14:49:38.996897563 (887h20m38.867980608s) [seg_pos={0 0}, upload=776500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:49:38.996980651 (887h20m38.868063696s) [upload=776500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:38.996987374 (887h20m38.868070419s) [upload=776500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:39.287134662 (887h20m39.158217707s) [upload=776500]: Done uploading segments... Jan 11 14:49:39.287158889 (887h20m39.158241934s) [upload=776500]: Upload finished. err: <nil> Duration: 5.821656805s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXX] duration: 30.001962874s trace: https://jaeger.dev.storj.io/trace/47484fcdca5fde00 FAIL (2026-01-11 14:53:56 UTC) orbiter warning: context deadline exceeded Duration: 30.001972913s edge gateway us-select-1

1
...
390
391
392
393
394
...
9039