Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.] duration: 20.117194293s trace: https://jaeger.dev.storj.io/trace/196f5381a23dc1f FAIL (2025-12-29 07:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 29 07:13:33.917928894 (567h44m33.789011929s) [upload=497756]: Starting upload Dec 29 07:13:33.917980142 (567h44m33.789063177s) [upload=497756]: Done starting upload Dec 29 07:13:33.918085582 (567h44m33.789168617s) [upload=497756]: Uploading segments... Dec 29 07:13:33.918223515 (567h44m33.78930655s) [upload=497756]: Got next segment. Inline: false Dec 29 07:13:33.918269562 (567h44m33.789352597s) [upload=497756]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:13:33.918275433 (567h44m33.789358479s) [upload=497756]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:13:34.164020195 (567h44m34.035103231s) [seg_pos={0 0}, upload=497756]: Begin upload segment... Dec 29 07:13:34.164072965 (567h44m34.03515601s) [seg_pos={0 0}, upload=497756]: Done begin upload segment. Dec 29 07:13:34.164106329 (567h44m34.035189364s) [noise=true , address=145.79.227.20:4437 , piece_id=ZIT6LOL52ZDRXZB7RMZS7PT5G7TOOVM7WOGLIVOQBQJEFD3KZULQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.164270332 (567h44m34.035353377s) [noise=true , address=145.79.227.38:4979 , piece_id=OEYIBNCFBCSTB3XRJ43CO52IVEZNUXVREZPJSXVZO6VU3AKWRCOQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.164759102 (567h44m34.035842137s) [noise=true , address=145.79.227.26:4654 , piece_id=Q6HSEEH2LGWYUCOGVS34IW2JLQYYNVIJ2PXEOATO54G4EBB5OZNA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165138485 (567h44m34.03622153s) [noise=true , address=145.79.227.47:4267 , piece_id=DATTF6B5NXANYYVYVYNHVXFQTOTY5AW6T63ZBTBWV3B47MDF3HQA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165444969 (567h44m34.036528014s) [noise=true , address=145.79.227.15:4875 , piece_id=2N3VN3ZCEMX2VWS7TJR27TX3URLAWNNDILI5O5TNRTLW4GTQHG4A, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165760579 (567h44m34.036843625s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165965279 (567h44m34.037048314s) [noise=true , address=45.91.116.11:4118 , piece_id=GS7QW27NHIQYASI4UIVVLOC4PUT3Y7S4LVYJSQX6KOFTXXWF3NIA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166182463 (567h44m34.037265498s) [noise=true , address=145.79.227.50:4375 , piece_id=B7C3G3MWOQJJUBF3QTVHNUTFXRZC35G4UQCJJEFTXTI5J6GCHXPQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166401950 (567h44m34.037484985s) [noise=true , address=45.91.116.5:4297 , piece_id=HJ2BBQKCAUCXDFXSTC47I56BC6JYAJML4J3F4FISP6ARYJW7VZZA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166505768 (567h44m34.037588803s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166794107 (567h44m34.037877152s) [noise=true , address=145.79.227.52:4445 , piece_id=CQK53CVBZN7TTTDU7H4LFL5SUWBCPH72FFH7MBTTBFLIMVGRCOZQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166873459 (567h44m34.037956504s) [noise=true , address=145.79.227.46:4602 , piece_id=M67262HGYHIDE7WRYLO4UCND5T67YFTJTFJ5B6LLL2GZZLHGAYVA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.167251447 (567h44m34.038334493s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.168309383 (567h44m34.039392418s) [noise=true , address=45.91.116.14:4192 , piece_id=SO5EMCLJELG72O7NTS4TZBBJA7IWSEERD4JXSCX6PWES5T7EHR3Q, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.170606667 (567h44m34.041689702s) [noise=true , address=145.79.227.19:4077 , piece_id=FO4IZIQSKNNKFHKZ6JW73J6WNZHPU5AR3SFTDUM7WT3KTF6BNOEQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.170841775 (567h44m34.04192481s) [noise=true , address=145.79.227.16:4910 , piece_id=OJE66GV7LB2JYLNQ5JNJ4E4QA4XQUIXPL2IGUXPUWWW7CH4RQBIQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:35.869278605 (567h44m35.74036164s) [upload=497756]: Next returned nil segment Dec 29 07:13:35.869324353 (567h44m35.740407408s) [upload=497756]: Waiting for error group managing segments... Dec 29 07:13:36.164946630 (567h44m36.036029665s) [noise=true , address=145.79.227.46:4602 , piece_id=M67262HGYHIDE7WRYLO4UCND5T67YFTJTFJ5B6LLL2GZZLHGAYVA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.189954898 (567h44m36.061037944s) [noise=true , address=145.79.227.38:4979 , piece_id=OEYIBNCFBCSTB3XRJ43CO52IVEZNUXVREZPJSXVZO6VU3AKWRCOQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.191989123 (567h44m36.063072158s) [noise=true , address=145.79.227.19:4077 , piece_id=FO4IZIQSKNNKFHKZ6JW73J6WNZHPU5AR3SFTDUM7WT3KTF6BNOEQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.192346222 (567h44m36.063429257s) [noise=true , address=145.79.227.20:4437 , piece_id=ZIT6LOL52ZDRXZB7RMZS7PT5G7TOOVM7WOGLIVOQBQJEFD3KZULQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.192705947 (567h44m36.063788992s) [noise=true , address=145.79.227.15:4875 , piece_id=2N3VN3ZCEMX2VWS7TJR27TX3URLAWNNDILI5O5TNRTLW4GTQHG4A, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193096270 (567h44m36.064179305s) [noise=true , address=145.79.227.16:4910 , piece_id=OJE66GV7LB2JYLNQ5JNJ4E4QA4XQUIXPL2IGUXPUWWW7CH4RQBIQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193431489 (567h44m36.064514524s) [noise=true , address=145.79.227.26:4654 , piece_id=Q6HSEEH2LGWYUCOGVS34IW2JLQYYNVIJ2PXEOATO54G4EBB5OZNA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193853753 (567h44m36.064936798s) [noise=true , address=145.79.227.50:4375 , piece_id=B7C3G3MWOQJJUBF3QTVHNUTFXRZC35G4UQCJJEFTXTI5J6GCHXPQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193949916 (567h44m36.065032951s) [noise=true , address=145.79.227.47:4267 , piece_id=DATTF6B5NXANYYVYVYNHVXFQTOTY5AW6T63ZBTBWV3B47MDF3HQA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.195957168 (567h44m36.067040203s) [noise=true , address=145.79.227.52:4445 , piece_id=CQK53CVBZN7TTTDU7H4LFL5SUWBCPH72FFH7MBTTBFLIMVGRCOZQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.387883211 (567h44m36.258966256s) [noise=true , address=45.91.116.14:4192 , piece_id=SO5EMCLJELG72O7NTS4TZBBJA7IWSEERD4JXSCX6PWES5T7EHR3Q, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.438305823 (567h44m36.309388858s) [noise=true , address=45.91.116.11:4118 , piece_id=GS7QW27NHIQYASI4UIVVLOC4PUT3Y7S4LVYJSQX6KOFTXXWF3NIA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.490720718 (567h44m36.361803764s) [noise=true , address=45.91.116.5:4297 , piece_id=HJ2BBQKCAUCXDFXSTC47I56BC6JYAJML4J3F4FISP6ARYJW7VZZA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.490759973 (567h44m36.361843008s) [seg_pos={0 0}, upload=497756]: Segment reached optimal threshold of 13 pieces. Dec 29 07:13:36.490969793 (567h44m36.362052838s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection Dec 29 07:13:36.490981054 (567h44m36.362064109s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.490991344 (567h44m36.362074389s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 07:13:36.490996624 (567h44m36.362079669s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.491290002 (567h44m36.362373037s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 07:13:36.491300061 (567h44m36.362383116s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.491320771 (567h44m36.362403806s) [seg_pos={0 0}, upload=497756]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:13:36.491402807 (567h44m36.362485852s) [upload=497756]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:13:36.491409370 (567h44m36.362492405s) [upload=497756]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:13:36.789735223 (567h44m36.660818268s) [upload=497756]: Done uploading segments... Dec 29 07:13:36.789758778 (567h44m36.660841823s) [upload=497756]: Upload finished. err: <nil> Duration: 20.117230432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 15.648658722s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:14:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 15.648675453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 42.547289725s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:14:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 42.547315382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.......X...................] duration: 3.466484002s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:14:07 UTC) orbiter warning: context deadline exceeded Duration: 3.46651408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X] duration: 30.003589849s trace: https://jaeger.dev.storj.io/trace/196f5381a23dc1f FAIL (2025-12-29 07:14:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 29 07:13:33.917928894 (567h44m33.789011929s) [upload=497756]: Starting upload Dec 29 07:13:33.917980142 (567h44m33.789063177s) [upload=497756]: Done starting upload Dec 29 07:13:33.918085582 (567h44m33.789168617s) [upload=497756]: Uploading segments... Dec 29 07:13:33.918223515 (567h44m33.78930655s) [upload=497756]: Got next segment. Inline: false Dec 29 07:13:33.918269562 (567h44m33.789352597s) [upload=497756]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:13:33.918275433 (567h44m33.789358479s) [upload=497756]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:13:34.164020195 (567h44m34.035103231s) [seg_pos={0 0}, upload=497756]: Begin upload segment... Dec 29 07:13:34.164072965 (567h44m34.03515601s) [seg_pos={0 0}, upload=497756]: Done begin upload segment. Dec 29 07:13:34.164106329 (567h44m34.035189364s) [noise=true , address=145.79.227.20:4437 , piece_id=ZIT6LOL52ZDRXZB7RMZS7PT5G7TOOVM7WOGLIVOQBQJEFD3KZULQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.164270332 (567h44m34.035353377s) [noise=true , address=145.79.227.38:4979 , piece_id=OEYIBNCFBCSTB3XRJ43CO52IVEZNUXVREZPJSXVZO6VU3AKWRCOQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.164759102 (567h44m34.035842137s) [noise=true , address=145.79.227.26:4654 , piece_id=Q6HSEEH2LGWYUCOGVS34IW2JLQYYNVIJ2PXEOATO54G4EBB5OZNA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165138485 (567h44m34.03622153s) [noise=true , address=145.79.227.47:4267 , piece_id=DATTF6B5NXANYYVYVYNHVXFQTOTY5AW6T63ZBTBWV3B47MDF3HQA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165444969 (567h44m34.036528014s) [noise=true , address=145.79.227.15:4875 , piece_id=2N3VN3ZCEMX2VWS7TJR27TX3URLAWNNDILI5O5TNRTLW4GTQHG4A, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165760579 (567h44m34.036843625s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.165965279 (567h44m34.037048314s) [noise=true , address=45.91.116.11:4118 , piece_id=GS7QW27NHIQYASI4UIVVLOC4PUT3Y7S4LVYJSQX6KOFTXXWF3NIA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166182463 (567h44m34.037265498s) [noise=true , address=145.79.227.50:4375 , piece_id=B7C3G3MWOQJJUBF3QTVHNUTFXRZC35G4UQCJJEFTXTI5J6GCHXPQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166401950 (567h44m34.037484985s) [noise=true , address=45.91.116.5:4297 , piece_id=HJ2BBQKCAUCXDFXSTC47I56BC6JYAJML4J3F4FISP6ARYJW7VZZA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166505768 (567h44m34.037588803s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166794107 (567h44m34.037877152s) [noise=true , address=145.79.227.52:4445 , piece_id=CQK53CVBZN7TTTDU7H4LFL5SUWBCPH72FFH7MBTTBFLIMVGRCOZQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.166873459 (567h44m34.037956504s) [noise=true , address=145.79.227.46:4602 , piece_id=M67262HGYHIDE7WRYLO4UCND5T67YFTJTFJ5B6LLL2GZZLHGAYVA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.167251447 (567h44m34.038334493s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.168309383 (567h44m34.039392418s) [noise=true , address=45.91.116.14:4192 , piece_id=SO5EMCLJELG72O7NTS4TZBBJA7IWSEERD4JXSCX6PWES5T7EHR3Q, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.170606667 (567h44m34.041689702s) [noise=true , address=145.79.227.19:4077 , piece_id=FO4IZIQSKNNKFHKZ6JW73J6WNZHPU5AR3SFTDUM7WT3KTF6BNOEQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:34.170841775 (567h44m34.04192481s) [noise=true , address=145.79.227.16:4910 , piece_id=OJE66GV7LB2JYLNQ5JNJ4E4QA4XQUIXPL2IGUXPUWWW7CH4RQBIQ, seg_pos={0 0}, upload=497756]: Uploading piece... Dec 29 07:13:35.869278605 (567h44m35.74036164s) [upload=497756]: Next returned nil segment Dec 29 07:13:35.869324353 (567h44m35.740407408s) [upload=497756]: Waiting for error group managing segments... Dec 29 07:13:36.164946630 (567h44m36.036029665s) [noise=true , address=145.79.227.46:4602 , piece_id=M67262HGYHIDE7WRYLO4UCND5T67YFTJTFJ5B6LLL2GZZLHGAYVA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.189954898 (567h44m36.061037944s) [noise=true , address=145.79.227.38:4979 , piece_id=OEYIBNCFBCSTB3XRJ43CO52IVEZNUXVREZPJSXVZO6VU3AKWRCOQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.191989123 (567h44m36.063072158s) [noise=true , address=145.79.227.19:4077 , piece_id=FO4IZIQSKNNKFHKZ6JW73J6WNZHPU5AR3SFTDUM7WT3KTF6BNOEQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.192346222 (567h44m36.063429257s) [noise=true , address=145.79.227.20:4437 , piece_id=ZIT6LOL52ZDRXZB7RMZS7PT5G7TOOVM7WOGLIVOQBQJEFD3KZULQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.192705947 (567h44m36.063788992s) [noise=true , address=145.79.227.15:4875 , piece_id=2N3VN3ZCEMX2VWS7TJR27TX3URLAWNNDILI5O5TNRTLW4GTQHG4A, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193096270 (567h44m36.064179305s) [noise=true , address=145.79.227.16:4910 , piece_id=OJE66GV7LB2JYLNQ5JNJ4E4QA4XQUIXPL2IGUXPUWWW7CH4RQBIQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193431489 (567h44m36.064514524s) [noise=true , address=145.79.227.26:4654 , piece_id=Q6HSEEH2LGWYUCOGVS34IW2JLQYYNVIJ2PXEOATO54G4EBB5OZNA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193853753 (567h44m36.064936798s) [noise=true , address=145.79.227.50:4375 , piece_id=B7C3G3MWOQJJUBF3QTVHNUTFXRZC35G4UQCJJEFTXTI5J6GCHXPQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.193949916 (567h44m36.065032951s) [noise=true , address=145.79.227.47:4267 , piece_id=DATTF6B5NXANYYVYVYNHVXFQTOTY5AW6T63ZBTBWV3B47MDF3HQA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.195957168 (567h44m36.067040203s) [noise=true , address=145.79.227.52:4445 , piece_id=CQK53CVBZN7TTTDU7H4LFL5SUWBCPH72FFH7MBTTBFLIMVGRCOZQ, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.387883211 (567h44m36.258966256s) [noise=true , address=45.91.116.14:4192 , piece_id=SO5EMCLJELG72O7NTS4TZBBJA7IWSEERD4JXSCX6PWES5T7EHR3Q, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.438305823 (567h44m36.309388858s) [noise=true , address=45.91.116.11:4118 , piece_id=GS7QW27NHIQYASI4UIVVLOC4PUT3Y7S4LVYJSQX6KOFTXXWF3NIA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.490720718 (567h44m36.361803764s) [noise=true , address=45.91.116.5:4297 , piece_id=HJ2BBQKCAUCXDFXSTC47I56BC6JYAJML4J3F4FISP6ARYJW7VZZA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: <nil> Dec 29 07:13:36.490759973 (567h44m36.361843008s) [seg_pos={0 0}, upload=497756]: Segment reached optimal threshold of 13 pieces. Dec 29 07:13:36.490969793 (567h44m36.362052838s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55894->77.90.31.21:4880: use of closed network connection Dec 29 07:13:36.490981054 (567h44m36.362064109s) [noise=true , address=77.90.31.21:4880 , piece_id=RN7WFYKCOCQEUQYRMEF7CTDRTV36DKMWMOF33WZYAHHSUFTZWM6A, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.490991344 (567h44m36.362074389s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 07:13:36.490996624 (567h44m36.362079669s) [noise=true , address=145.79.227.40:4049 , piece_id=YOCONDXXXXAHRVRP2Z6GTJS6MWPFVEQNLRLGGIXRTBWXBZBWIXLA, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.491290002 (567h44m36.362373037s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 07:13:36.491300061 (567h44m36.362383116s) [noise=true , address=145.79.227.35:4482 , piece_id=C2XEHFLHK4KESN7O5LMU63OYUGWQUMLOJE44DFQXFP7GQ5QYAHIA, seg_pos={0 0}, upload=497756]: Long tail cancellation due to optimal threshold Dec 29 07:13:36.491320771 (567h44m36.362403806s) [seg_pos={0 0}, upload=497756]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:13:36.491402807 (567h44m36.362485852s) [upload=497756]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:13:36.491409370 (567h44m36.362492405s) [upload=497756]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:13:36.789735223 (567h44m36.660818268s) [upload=497756]: Done uploading segments... Dec 29 07:13:36.789758778 (567h44m36.660841823s) [upload=497756]: Upload finished. err: <nil> Duration: 30.003605188s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.......X..................] duration: 3.414227577s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:13:41 UTC) orbiter warning: context deadline exceeded Duration: 3.414257355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.] duration: 6.032522219s trace: https://jaeger.dev.storj.io/trace/19ff7652e4349d35 FAIL (2025-12-29 07:13:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:12:19.493320371 (567h43m19.364403417s) [upload=497739]: Starting upload Dec 29 07:12:19.493402958 (567h43m19.364486003s) [upload=497739]: Done starting upload Dec 29 07:12:19.493670378 (567h43m19.364753423s) [upload=497739]: Uploading segments... Dec 29 07:12:19.493730432 (567h43m19.364813507s) [upload=497739]: Got next segment. Inline: false Dec 29 07:12:19.493771861 (567h43m19.364854916s) [upload=497739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:12:19.493778013 (567h43m19.364861058s) [upload=497739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:12:19.740115724 (567h43m19.611198769s) [seg_pos={0 0}, upload=497739]: Begin upload segment... Dec 29 07:12:19.740164677 (567h43m19.611247712s) [seg_pos={0 0}, upload=497739]: Done begin upload segment. Dec 29 07:12:19.740198010 (567h43m19.611281045s) [noise=true , address=145.79.227.14:4840 , piece_id=DXQRV74MQ54UJHS4JJ5VXO6S2ZIOBZPSLULXSSDB4JICFVVNXUIQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740250020 (567h43m19.611333055s) [noise=true , address=77.90.31.16:4736 , piece_id=3Z6TMWR4KD5WG2NWA3GFXUDL5JJM5LUJRG4ZBCLEGUHRIRV6BRNQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740706980 (567h43m19.611790015s) [noise=true , address=145.79.227.37:4566 , piece_id=JHZNTUJNUQX67CS3PZXMMOSN2YKYZJC27PRI7F7NPBRQ7CPSQECA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740985169 (567h43m19.612068204s) [noise=true , address=145.79.227.12:4767 , piece_id=RDFCI5E62MFRPHNQL3IWRSJ2BE42VRO4JL5OHWUJISXPPZNY6EOQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741271084 (567h43m19.612354119s) [noise=true , address=45.91.116.5:4299 , piece_id=755Z5LOQVIMDJUUDLCQIQF4IQKFL3CUXTTPSEEIJGRBW5N7FPYJQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741541349 (567h43m19.612624384s) [noise=true , address=145.79.227.47:4267 , piece_id=ODPOJMO5MKQYSRMSDYFNVDVNFXFJS77QJ27OAOSJYERDNYNZFVZQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741824357 (567h43m19.612907392s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.742083180 (567h43m19.613166215s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.742346882 (567h43m19.613429917s) [noise=true , address=45.91.116.10:4083 , piece_id=EOEJABZQVMTOYPEVZY3MRC2EPS4P2R73YAYHHD65VSDTTNSYRIZA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.743891794 (567h43m19.614974829s) [noise=true , address=77.90.31.10:4518 , piece_id=BLSQSSQDSHDXBWKNNMGZ3NVTOO46RTNOHUCXROZVPE2YGV4YPJZA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.745252244 (567h43m19.616335279s) [noise=true , address=145.79.227.18:4007 , piece_id=EM22TFZAP4NOFM2ICTZPLUIPERJWDDXSU4FT2HVYNHACMF2DQA7Q, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.748026557 (567h43m19.619109602s) [noise=true , address=45.91.116.11:4117 , piece_id=3SNEQWTZE33P4FZ6NRPKYAXO2VT5VQLYVQOAPWCCLCGVAFKPGQHA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.749211403 (567h43m19.620294438s) [noise=true , address=145.79.227.21:4473 , piece_id=G4USU4KKCBBGF36B4KINDEEY5OKW6RHH4SCWUEVF6EEMIMZNYBMQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.750285098 (567h43m19.621368133s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.751510461 (567h43m19.622593506s) [noise=true , address=45.91.116.9:4264 , piece_id=QSIJZYHQER6RPWCTXM2ZTL7GGHDNGZIXGVIOYFEDKG7OAEXIOGMQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.752528209 (567h43m19.623611244s) [noise=true , address=77.90.31.14:4661 , piece_id=KGY4FLFRZV5AU2AZ2RYJGI7G5VAM4AXV7Z5RJXXOCAUF7CIPY42A, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:21.444749127 (567h43m21.315832162s) [upload=497739]: Next returned nil segment Dec 29 07:12:21.444762191 (567h43m21.315845246s) [upload=497739]: Waiting for error group managing segments... Dec 29 07:12:21.736913248 (567h43m21.607996303s) [noise=true , address=145.79.227.18:4007 , piece_id=EM22TFZAP4NOFM2ICTZPLUIPERJWDDXSU4FT2HVYNHACMF2DQA7Q, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.741559855 (567h43m21.6126429s) [noise=true , address=145.79.227.47:4267 , piece_id=ODPOJMO5MKQYSRMSDYFNVDVNFXFJS77QJ27OAOSJYERDNYNZFVZQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.741826072 (567h43m21.612909117s) [noise=true , address=145.79.227.12:4767 , piece_id=RDFCI5E62MFRPHNQL3IWRSJ2BE42VRO4JL5OHWUJISXPPZNY6EOQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.747854712 (567h43m21.618937747s) [noise=true , address=145.79.227.37:4566 , piece_id=JHZNTUJNUQX67CS3PZXMMOSN2YKYZJC27PRI7F7NPBRQ7CPSQECA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.763305623 (567h43m21.634388668s) [noise=true , address=145.79.227.21:4473 , piece_id=G4USU4KKCBBGF36B4KINDEEY5OKW6RHH4SCWUEVF6EEMIMZNYBMQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.769309425 (567h43m21.64039247s) [noise=true , address=145.79.227.14:4840 , piece_id=DXQRV74MQ54UJHS4JJ5VXO6S2ZIOBZPSLULXSSDB4JICFVVNXUIQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.932866420 (567h43m21.803949455s) [noise=true , address=77.90.31.10:4518 , piece_id=BLSQSSQDSHDXBWKNNMGZ3NVTOO46RTNOHUCXROZVPE2YGV4YPJZA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.941431168 (567h43m21.812514213s) [noise=true , address=77.90.31.14:4661 , piece_id=KGY4FLFRZV5AU2AZ2RYJGI7G5VAM4AXV7Z5RJXXOCAUF7CIPY42A, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.941975996 (567h43m21.813059031s) [noise=true , address=77.90.31.16:4736 , piece_id=3Z6TMWR4KD5WG2NWA3GFXUDL5JJM5LUJRG4ZBCLEGUHRIRV6BRNQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.987549825 (567h43m21.85863286s) [noise=true , address=45.91.116.9:4264 , piece_id=QSIJZYHQER6RPWCTXM2ZTL7GGHDNGZIXGVIOYFEDKG7OAEXIOGMQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.009213194 (567h43m21.88029624s) [noise=true , address=45.91.116.10:4083 , piece_id=EOEJABZQVMTOYPEVZY3MRC2EPS4P2R73YAYHHD65VSDTTNSYRIZA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.142121020 (567h43m22.013204065s) [noise=true , address=45.91.116.11:4117 , piece_id=3SNEQWTZE33P4FZ6NRPKYAXO2VT5VQLYVQOAPWCCLCGVAFKPGQHA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.833606916 (567h43m22.704689951s) [noise=true , address=45.91.116.5:4299 , piece_id=755Z5LOQVIMDJUUDLCQIQF4IQKFL3CUXTTPSEEIJGRBW5N7FPYJQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.833626082 (567h43m22.704709117s) [seg_pos={0 0}, upload=497739]: Segment reached optimal threshold of 13 pieces. Dec 29 07:12:22.833930141 (567h43m22.705013177s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.833937325 (567h43m22.70502036s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834113961 (567h43m22.705197006s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.834136615 (567h43m22.70521965s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834194275 (567h43m22.70527731s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.834201518 (567h43m22.705284563s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834218610 (567h43m22.705301646s) [seg_pos={0 0}, upload=497739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:12:22.834293113 (567h43m22.705376148s) [upload=497739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:12:22.834300547 (567h43m22.705383582s) [upload=497739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:12:23.133092409 (567h43m23.004175444s) [upload=497739]: Done uploading segments... Dec 29 07:12:23.133111346 (567h43m23.004194381s) [upload=497739]: Upload finished. err: <nil> Duration: 6.032552797s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X.......X.................] duration: 3.666790261s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:13:20 UTC) orbiter warning: context deadline exceeded Duration: 3.666813564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X......X........] duration: 25.651626777s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 25.651743088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X] duration: 30.002932817s trace: https://jaeger.dev.storj.io/trace/19ff7652e4349d35 FAIL (2025-12-29 07:12:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:12:19.493320371 (567h43m19.364403417s) [upload=497739]: Starting upload Dec 29 07:12:19.493402958 (567h43m19.364486003s) [upload=497739]: Done starting upload Dec 29 07:12:19.493670378 (567h43m19.364753423s) [upload=497739]: Uploading segments... Dec 29 07:12:19.493730432 (567h43m19.364813507s) [upload=497739]: Got next segment. Inline: false Dec 29 07:12:19.493771861 (567h43m19.364854916s) [upload=497739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:12:19.493778013 (567h43m19.364861058s) [upload=497739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:12:19.740115724 (567h43m19.611198769s) [seg_pos={0 0}, upload=497739]: Begin upload segment... Dec 29 07:12:19.740164677 (567h43m19.611247712s) [seg_pos={0 0}, upload=497739]: Done begin upload segment. Dec 29 07:12:19.740198010 (567h43m19.611281045s) [noise=true , address=145.79.227.14:4840 , piece_id=DXQRV74MQ54UJHS4JJ5VXO6S2ZIOBZPSLULXSSDB4JICFVVNXUIQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740250020 (567h43m19.611333055s) [noise=true , address=77.90.31.16:4736 , piece_id=3Z6TMWR4KD5WG2NWA3GFXUDL5JJM5LUJRG4ZBCLEGUHRIRV6BRNQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740706980 (567h43m19.611790015s) [noise=true , address=145.79.227.37:4566 , piece_id=JHZNTUJNUQX67CS3PZXMMOSN2YKYZJC27PRI7F7NPBRQ7CPSQECA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.740985169 (567h43m19.612068204s) [noise=true , address=145.79.227.12:4767 , piece_id=RDFCI5E62MFRPHNQL3IWRSJ2BE42VRO4JL5OHWUJISXPPZNY6EOQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741271084 (567h43m19.612354119s) [noise=true , address=45.91.116.5:4299 , piece_id=755Z5LOQVIMDJUUDLCQIQF4IQKFL3CUXTTPSEEIJGRBW5N7FPYJQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741541349 (567h43m19.612624384s) [noise=true , address=145.79.227.47:4267 , piece_id=ODPOJMO5MKQYSRMSDYFNVDVNFXFJS77QJ27OAOSJYERDNYNZFVZQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.741824357 (567h43m19.612907392s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.742083180 (567h43m19.613166215s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.742346882 (567h43m19.613429917s) [noise=true , address=45.91.116.10:4083 , piece_id=EOEJABZQVMTOYPEVZY3MRC2EPS4P2R73YAYHHD65VSDTTNSYRIZA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.743891794 (567h43m19.614974829s) [noise=true , address=77.90.31.10:4518 , piece_id=BLSQSSQDSHDXBWKNNMGZ3NVTOO46RTNOHUCXROZVPE2YGV4YPJZA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.745252244 (567h43m19.616335279s) [noise=true , address=145.79.227.18:4007 , piece_id=EM22TFZAP4NOFM2ICTZPLUIPERJWDDXSU4FT2HVYNHACMF2DQA7Q, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.748026557 (567h43m19.619109602s) [noise=true , address=45.91.116.11:4117 , piece_id=3SNEQWTZE33P4FZ6NRPKYAXO2VT5VQLYVQOAPWCCLCGVAFKPGQHA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.749211403 (567h43m19.620294438s) [noise=true , address=145.79.227.21:4473 , piece_id=G4USU4KKCBBGF36B4KINDEEY5OKW6RHH4SCWUEVF6EEMIMZNYBMQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.750285098 (567h43m19.621368133s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.751510461 (567h43m19.622593506s) [noise=true , address=45.91.116.9:4264 , piece_id=QSIJZYHQER6RPWCTXM2ZTL7GGHDNGZIXGVIOYFEDKG7OAEXIOGMQ, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:19.752528209 (567h43m19.623611244s) [noise=true , address=77.90.31.14:4661 , piece_id=KGY4FLFRZV5AU2AZ2RYJGI7G5VAM4AXV7Z5RJXXOCAUF7CIPY42A, seg_pos={0 0}, upload=497739]: Uploading piece... Dec 29 07:12:21.444749127 (567h43m21.315832162s) [upload=497739]: Next returned nil segment Dec 29 07:12:21.444762191 (567h43m21.315845246s) [upload=497739]: Waiting for error group managing segments... Dec 29 07:12:21.736913248 (567h43m21.607996303s) [noise=true , address=145.79.227.18:4007 , piece_id=EM22TFZAP4NOFM2ICTZPLUIPERJWDDXSU4FT2HVYNHACMF2DQA7Q, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.741559855 (567h43m21.6126429s) [noise=true , address=145.79.227.47:4267 , piece_id=ODPOJMO5MKQYSRMSDYFNVDVNFXFJS77QJ27OAOSJYERDNYNZFVZQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.741826072 (567h43m21.612909117s) [noise=true , address=145.79.227.12:4767 , piece_id=RDFCI5E62MFRPHNQL3IWRSJ2BE42VRO4JL5OHWUJISXPPZNY6EOQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.747854712 (567h43m21.618937747s) [noise=true , address=145.79.227.37:4566 , piece_id=JHZNTUJNUQX67CS3PZXMMOSN2YKYZJC27PRI7F7NPBRQ7CPSQECA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.763305623 (567h43m21.634388668s) [noise=true , address=145.79.227.21:4473 , piece_id=G4USU4KKCBBGF36B4KINDEEY5OKW6RHH4SCWUEVF6EEMIMZNYBMQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.769309425 (567h43m21.64039247s) [noise=true , address=145.79.227.14:4840 , piece_id=DXQRV74MQ54UJHS4JJ5VXO6S2ZIOBZPSLULXSSDB4JICFVVNXUIQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.932866420 (567h43m21.803949455s) [noise=true , address=77.90.31.10:4518 , piece_id=BLSQSSQDSHDXBWKNNMGZ3NVTOO46RTNOHUCXROZVPE2YGV4YPJZA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.941431168 (567h43m21.812514213s) [noise=true , address=77.90.31.14:4661 , piece_id=KGY4FLFRZV5AU2AZ2RYJGI7G5VAM4AXV7Z5RJXXOCAUF7CIPY42A, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.941975996 (567h43m21.813059031s) [noise=true , address=77.90.31.16:4736 , piece_id=3Z6TMWR4KD5WG2NWA3GFXUDL5JJM5LUJRG4ZBCLEGUHRIRV6BRNQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:21.987549825 (567h43m21.85863286s) [noise=true , address=45.91.116.9:4264 , piece_id=QSIJZYHQER6RPWCTXM2ZTL7GGHDNGZIXGVIOYFEDKG7OAEXIOGMQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.009213194 (567h43m21.88029624s) [noise=true , address=45.91.116.10:4083 , piece_id=EOEJABZQVMTOYPEVZY3MRC2EPS4P2R73YAYHHD65VSDTTNSYRIZA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.142121020 (567h43m22.013204065s) [noise=true , address=45.91.116.11:4117 , piece_id=3SNEQWTZE33P4FZ6NRPKYAXO2VT5VQLYVQOAPWCCLCGVAFKPGQHA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.833606916 (567h43m22.704689951s) [noise=true , address=45.91.116.5:4299 , piece_id=755Z5LOQVIMDJUUDLCQIQF4IQKFL3CUXTTPSEEIJGRBW5N7FPYJQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: <nil> Dec 29 07:12:22.833626082 (567h43m22.704709117s) [seg_pos={0 0}, upload=497739]: Segment reached optimal threshold of 13 pieces. Dec 29 07:12:22.833930141 (567h43m22.705013177s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.833937325 (567h43m22.70502036s) [noise=true , address=77.90.31.17:4770 , piece_id=JTEUMPERJJSJ3YPGT32O7BSPR73KVXDFAIRZOCA57SA4TKZK2ZYQ, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834113961 (567h43m22.705197006s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.834136615 (567h43m22.70521965s) [noise=true , address=45.91.116.7:4009 , piece_id=MC235EXMT3PKJAQX2VGNGTJFKA4ROM4SSKLI4PM772KJUVYUXBIA, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834194275 (567h43m22.70527731s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 07:12:22.834201518 (567h43m22.705284563s) [noise=true , address=145.79.227.40:4049 , piece_id=GL4D7OEBI562RJOHUTRYIETZL2EHZXX5B6FX5DOWEFGZBYTQES7Q, seg_pos={0 0}, upload=497739]: Long tail cancellation due to optimal threshold Dec 29 07:12:22.834218610 (567h43m22.705301646s) [seg_pos={0 0}, upload=497739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:12:22.834293113 (567h43m22.705376148s) [upload=497739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:12:22.834300547 (567h43m22.705383582s) [upload=497739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:12:23.133092409 (567h43m23.004175444s) [upload=497739]: Done uploading segments... Dec 29 07:12:23.133111346 (567h43m23.004194381s) [upload=497739]: Upload finished. err: <nil> Duration: 30.002952444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X......X.......] duration: 46.699308784s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 46.699403014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..................................X.......X................] duration: 5.001000155s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:12:09 UTC) orbiter warning: context deadline exceeded Duration: 5.00105978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 35.375967261s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:11:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 35.375991627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X..................................X.......X...............] duration: 5.001211152s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:11:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001237342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......] duration: 5.626895457s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:11:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.626924713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X......] duration: 5.946207909s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.946233157s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X..................................X.......X..............] duration: 3.397445222s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:10:28 UTC) orbiter warning: context deadline exceeded Duration: 3.397845213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 35.873973496s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 35.874039021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.....] duration: 5.914078416s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:10:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.91410699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X..................................X.......X.............] duration: 3.437726735s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:09:51 UTC) orbiter warning: context deadline exceeded Duration: 3.437760429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 21.38558597s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:09:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 21.385617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X....] duration: 5.666200387s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.66622856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X..................................X.......X............] duration: 3.558206248s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:09:07 UTC) orbiter warning: context deadline exceeded Duration: 3.558251434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X......X...] duration: 34.729531769s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:08:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 34.729545966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X...] duration: 5.651028123s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:08:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.6510522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 44.536214931s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 44.536234108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X..................................X.......X...........] duration: 3.469256711s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:07:52 UTC) orbiter warning: context deadline exceeded Duration: 3.469285976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X..] duration: 6.058894825s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:07:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 6.058921355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 33.627448591s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:07:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 33.627489459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X..................................X.......X..........] duration: 3.446950177s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:06:50 UTC) orbiter warning: context deadline exceeded Duration: 3.446978351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 1m0.004012477s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:06:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 1m0.004054297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.] duration: 5.718290091s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:06:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 5.718317011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X] duration: 30.003145186s trace: https://jaeger.dev.storj.io/trace/2478eb89eff596c4 FAIL (2025-12-29 07:05:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:05:21.455242567 (567h36m21.326325602s) [upload=497630]: Starting upload Dec 29 07:05:21.455297061 (567h36m21.326380096s) [upload=497630]: Done starting upload Dec 29 07:05:21.455443620 (567h36m21.326526655s) [upload=497630]: Uploading segments... Dec 29 07:05:21.456002113 (567h36m21.327085148s) [upload=497630]: Got next segment. Inline: false Dec 29 07:05:21.456035767 (567h36m21.327118812s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:05:21.456040396 (567h36m21.327123441s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:05:21.705676368 (567h36m21.576759413s) [seg_pos={0 0}, upload=497630]: Begin upload segment... Dec 29 07:05:21.705733998 (567h36m21.576817033s) [seg_pos={0 0}, upload=497630]: Done begin upload segment. Dec 29 07:05:21.705764155 (567h36m21.57684719s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.705875637 (567h36m21.576958682s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.706528340 (567h36m21.577611385s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707083968 (567h36m21.578167003s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707483399 (567h36m21.578566434s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.707907145 (567h36m21.57899018s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708387099 (567h36m21.579470134s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.708798132 (567h36m21.579881167s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.709240365 (567h36m21.5803234s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.710515272 (567h36m21.581598317s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.711956346 (567h36m21.583039381s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.714016359 (567h36m21.585099394s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.724893120 (567h36m21.595976195s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.727995307 (567h36m21.599078352s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.728792825 (567h36m21.59987587s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:21.729724098 (567h36m21.600807153s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Uploading piece... Dec 29 07:05:23.414877717 (567h36m23.285960752s) [upload=497630]: Next returned nil segment Dec 29 07:05:23.414892566 (567h36m23.285975601s) [upload=497630]: Waiting for error group managing segments... Dec 29 07:05:23.696513913 (567h36m23.567596958s) [noise=true , address=145.79.227.48:4303 , piece_id=T25DXRKGGUOJUJPAO6OZAWPL47S5JEOBRZJR4NRLKBDBSJATMEDA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.700290844 (567h36m23.571373879s) [noise=true , address=145.79.227.14:4839 , piece_id=HEE4EDR2RASRVJGBYZFUYZON66A5VHBWECLULF4SKP4AJ4G4XPAA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717837346 (567h36m23.588920381s) [noise=true , address=145.79.227.21:4473 , piece_id=A6GQOVSCPRDYPWSYIN5R26B6I3JQQKEY4K65ED2FUT3OT4DJZ7BA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.717885798 (567h36m23.588968843s) [noise=true , address=145.79.227.25:4618 , piece_id=C43AWUKFT46YGWWOMNKLA3IYIYOVJTOGJIXJ26NF4PSYFYJ4IM3A, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.720688714 (567h36m23.591771749s) [noise=true , address=145.79.227.16:4912 , piece_id=HR3LGZNQ6Q3RJOVQD5DZ3A5Y6V3UXT55NHDU4M7PRI624LFTWUFA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.743347741 (567h36m23.614430776s) [noise=true , address=145.79.227.42:4122 , piece_id=6ANJRUQBSHCDDTJBK3EOW2N4LJQNCTIDBRGSASL36WULBH7NQVSQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.748896376 (567h36m23.619979421s) [noise=true , address=145.79.227.39:4013 , piece_id=FFPDH26UQOS5Y7DRR3SZVB74TDN3NJIBUZQFNBQCTQIJWTIRYSGQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881291345 (567h36m23.75237442s) [noise=true , address=77.90.31.21:4878 , piece_id=KVKFV67TUDU4VKGI3XPQCTOHYQQIOXSN3MJPNFKPFBXCOONJENZQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.881943106 (567h36m23.753026151s) [noise=true , address=77.90.31.13:4626 , piece_id=CZ32HE3R7SS45MH6DC3CHITZWKH3RRGESI5HUWZNL6A2YJEFE7LA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:23.980258372 (567h36m23.851341427s) [noise=true , address=45.91.116.11:4120 , piece_id=NS5DM6REV4BSDEOBT2DUYW7FUGMWVTEIUOK5BAKUDAEKIX7MTH6Q, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.001892968 (567h36m23.872976013s) [noise=true , address=45.91.116.4:4939 , piece_id=7K22YKR4C5FAL6IMSX5MFWSZHPJ7XSYRMFUE5NN4KINLK4C2WHKQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.009645259 (567h36m23.880728304s) [noise=true , address=77.90.31.11:4555 , piece_id=EMBTT4YCJEROVJV63VT7LOGUJHPYTFLADB27CJNSJOBBU37HYVDQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051960933 (567h36m23.923043978s) [noise=true , address=45.91.116.7:4010 , piece_id=EWV3ZLBSR5HWNF244XPD23N4PVST72G4IWCI6OWH62X4FX7NPMTQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: <nil> Dec 29 07:05:24.051989969 (567h36m23.923073024s) [seg_pos={0 0}, upload=497630]: Segment reached optimal threshold of 13 pieces. Dec 29 07:05:24.052220248 (567h36m23.923303293s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052232230 (567h36m23.923315265s) [noise=true , address=45.91.116.14:4191 , piece_id=WVUR3BUDVM6G76EI2OMPETE3AP44SYC5AYQLHUNJTGAV4ARVDCWQ, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052255294 (567h36m23.92333834s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:24.052264983 (567h36m23.923348018s) [noise=true , address=45.91.116.10:4084 , piece_id=IKPGRLBVZZ7SG5DOU6XHZ665IESN2LRQIQZDWE3RJEYQUPX7ATXA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052403116 (567h36m23.923486151s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40526->45.91.116.9:4261: use of closed network connection Dec 29 07:05:24.052412032 (567h36m23.923495077s) [noise=true , address=45.91.116.9:4261 , piece_id=5NUOUFROJK7JVNLAYOQJME75XELIMPGJKLYM7I5BF2TAYVS4MKNA, seg_pos={0 0}, upload=497630]: Long tail cancellation due to optimal threshold Dec 29 07:05:24.052463651 (567h36m23.923546686s) [seg_pos={0 0}, upload=497630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:05:24.052533975 (567h36m23.92361702s) [upload=497630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:24.052539576 (567h36m23.923622621s) [upload=497630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:24.340718819 (567h36m24.211801894s) [upload=497630]: Done uploading segments... Dec 29 07:05:24.340747203 (567h36m24.211830238s) [upload=497630]: Upload finished. err: <nil> Duration: 30.003171296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X..................................X.......X.........] duration: 3.518161554s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:05:43 UTC) orbiter warning: context deadline exceeded Duration: 3.518193605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 37.510051884s trace: https://jaeger.dev.storj.io/trace/5afe9fdd46a8e54f FAIL (2025-12-29 07:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 122dviPhVkb4TfDr3dSuQ5uXYr7mqYZGyawsfaiDa5CPtyEFi5D, Piece ID: DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ) context deadline exceeded; piecestore: (Node ID: 12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, Piece ID: AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA) context deadline exceeded ======LOGS====== 4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.470770769 (567h29m44.341853814s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471234462 (567h29m44.342317497s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471653239 (567h29m44.342736274s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472101613 (567h29m44.343184648s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472582339 (567h29m44.343665374s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473069336 (567h29m44.344152371s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473596440 (567h29m44.344679485s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.474789441 (567h29m44.345872476s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.476765935 (567h29m44.34784897s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.480795248 (567h29m44.351878293s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.484071686 (567h29m44.355154731s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.485106818 (567h29m44.356189863s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.488197412 (567h29m44.359280457s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.489260917 (567h29m44.360343962s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:45.821734925 (567h29m45.69281797s) [upload=497538]: Next returned nil segment Dec 29 06:58:45.821749823 (567h29m45.692832868s) [upload=497538]: Waiting for error group managing segments... Dec 29 06:58:46.321656276 (567h29m46.192739321s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.322482269 (567h29m46.193565304s) [noise=true , address=145.79.227.7:4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.333140875 (567h29m46.20422391s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.341854527 (567h29m46.212937562s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.344155919 (567h29m46.215238954s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.347603513 (567h29m46.218686559s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.372664974 (567h29m46.243748009s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.447122322 (567h29m46.318205357s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.502282514 (567h29m46.373365549s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.791207519 (567h29m46.662290564s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.234402275 (567h29m47.10548531s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.252070127 (567h29m47.123153172s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.319572090 (567h29m47.190655135s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383438812 (567h29m47.254521847s) [noise=true , address=77.90.31.17:4782 , piece_id=AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383457116 (567h29m47.254540161s) [seg_pos={0 0}, upload=497538]: Segment reached optimal threshold of 14 pieces. Dec 29 06:58:47.383670413 (567h29m47.254753448s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383680091 (567h29m47.254763126s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383696432 (567h29m47.254779467s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection Dec 29 06:58:47.383704077 (567h29m47.254787112s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383780052 (567h29m47.254863087s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection Dec 29 06:58:47.383784981 (567h29m47.254868016s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383932652 (567h29m47.255015687s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383940787 (567h29m47.255023832s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.384012954 (567h29m47.255095989s) [seg_pos={0 0}, upload=497538]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:58:47.384097746 (567h29m47.255180781s) [upload=497538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:58:47.384102636 (567h29m47.255185671s) [upload=497538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:58:47.684099632 (567h29m47.555182677s) [upload=497538]: Done uploading segments... Dec 29 06:58:47.684123357 (567h29m47.555206412s) [upload=497538]: Upload finished. err: <nil> Duration: 37.510090267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X..] duration: 5.874096984s trace: https://jaeger.dev.storj.io/trace/5dc50d5607afcb75 FAIL (2025-12-29 07:05:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ) context deadline exceeded ======LOGS====== Dec 29 07:03:26.605630546 (567h34m26.476713581s) [upload=497603]: Starting upload Dec 29 07:03:26.605706462 (567h34m26.476789507s) [upload=497603]: Done starting upload Dec 29 07:03:26.605876175 (567h34m26.47695922s) [upload=497603]: Uploading segments... Dec 29 07:03:26.606091605 (567h34m26.47717465s) [upload=497603]: Got next segment. Inline: false Dec 29 07:03:26.606171276 (567h34m26.477254331s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:03:26.606179582 (567h34m26.477262637s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:03:26.850554898 (567h34m26.721637943s) [seg_pos={0 0}, upload=497603]: Begin upload segment... Dec 29 07:03:26.850617657 (567h34m26.721700702s) [seg_pos={0 0}, upload=497603]: Done begin upload segment. Dec 29 07:03:26.850662312 (567h34m26.721745347s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.850725072 (567h34m26.721808117s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851302050 (567h34m26.722385085s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851635215 (567h34m26.72271825s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851743330 (567h34m26.722826376s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851962297 (567h34m26.723045332s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852356187 (567h34m26.723439232s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852674503 (567h34m26.723757538s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852692187 (567h34m26.723775232s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853005614 (567h34m26.724088649s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853047333 (567h34m26.724130368s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853400706 (567h34m26.724483751s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853416126 (567h34m26.724499341s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853729032 (567h34m26.724812067s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854159972 (567h34m26.725243007s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854607775 (567h34m26.72569081s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:28.556562875 (567h34m28.42764592s) [upload=497603]: Next returned nil segment Dec 29 07:03:28.556581762 (567h34m28.427664797s) [upload=497603]: Waiting for error group managing segments... Dec 29 07:03:28.836878777 (567h34m28.707961822s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.840256138 (567h34m28.711339203s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.846293083 (567h34m28.717376118s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.849048189 (567h34m28.720131224s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859491585 (567h34m28.73057463s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859857912 (567h34m28.730940947s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.866828044 (567h34m28.737911089s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867339358 (567h34m28.738422403s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867874507 (567h34m28.738957543s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.874538456 (567h34m28.745621501s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.045623807 (567h34m28.916706852s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.100033961 (567h34m28.971116996s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138577784 (567h34m29.009660819s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138610136 (567h34m29.009693181s) [seg_pos={0 0}, upload=497603]: Segment reached optimal threshold of 13 pieces. Dec 29 07:03:29.138860522 (567h34m29.009943587s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.138870822 (567h34m29.009953878s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.138936277 (567h34m29.010019322s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139054352 (567h34m29.010137407s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139338372 (567h34m29.010421417s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139348541 (567h34m29.010431606s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139373309 (567h34m29.010456354s) [seg_pos={0 0}, upload=497603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:03:29.139477447 (567h34m29.010560492s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:03:29.139484731 (567h34m29.010567786s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:03:29.433346344 (567h34m29.304429399s) [upload=497603]: Done uploading segments... Dec 29 07:03:29.433369609 (567h34m29.304452654s) [upload=497603]: Upload finished. err: <nil> Duration: 5.874121761s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X..................................X.......X........] duration: 3.658063817s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:04:33 UTC) orbiter warning: context deadline exceeded Duration: 3.658086489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X.] duration: 5.7074255s trace: https://jaeger.dev.storj.io/trace/5dc50d5607afcb75 FAIL (2025-12-29 07:04:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ) context deadline exceeded ======LOGS====== Dec 29 07:03:26.605630546 (567h34m26.476713581s) [upload=497603]: Starting upload Dec 29 07:03:26.605706462 (567h34m26.476789507s) [upload=497603]: Done starting upload Dec 29 07:03:26.605876175 (567h34m26.47695922s) [upload=497603]: Uploading segments... Dec 29 07:03:26.606091605 (567h34m26.47717465s) [upload=497603]: Got next segment. Inline: false Dec 29 07:03:26.606171276 (567h34m26.477254331s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:03:26.606179582 (567h34m26.477262637s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:03:26.850554898 (567h34m26.721637943s) [seg_pos={0 0}, upload=497603]: Begin upload segment... Dec 29 07:03:26.850617657 (567h34m26.721700702s) [seg_pos={0 0}, upload=497603]: Done begin upload segment. Dec 29 07:03:26.850662312 (567h34m26.721745347s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.850725072 (567h34m26.721808117s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851302050 (567h34m26.722385085s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851635215 (567h34m26.72271825s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851743330 (567h34m26.722826376s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851962297 (567h34m26.723045332s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852356187 (567h34m26.723439232s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852674503 (567h34m26.723757538s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852692187 (567h34m26.723775232s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853005614 (567h34m26.724088649s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853047333 (567h34m26.724130368s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853400706 (567h34m26.724483751s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853416126 (567h34m26.724499341s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853729032 (567h34m26.724812067s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854159972 (567h34m26.725243007s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854607775 (567h34m26.72569081s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:28.556562875 (567h34m28.42764592s) [upload=497603]: Next returned nil segment Dec 29 07:03:28.556581762 (567h34m28.427664797s) [upload=497603]: Waiting for error group managing segments... Dec 29 07:03:28.836878777 (567h34m28.707961822s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.840256138 (567h34m28.711339203s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.846293083 (567h34m28.717376118s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.849048189 (567h34m28.720131224s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859491585 (567h34m28.73057463s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859857912 (567h34m28.730940947s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.866828044 (567h34m28.737911089s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867339358 (567h34m28.738422403s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867874507 (567h34m28.738957543s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.874538456 (567h34m28.745621501s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.045623807 (567h34m28.916706852s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.100033961 (567h34m28.971116996s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138577784 (567h34m29.009660819s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138610136 (567h34m29.009693181s) [seg_pos={0 0}, upload=497603]: Segment reached optimal threshold of 13 pieces. Dec 29 07:03:29.138860522 (567h34m29.009943587s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.138870822 (567h34m29.009953878s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.138936277 (567h34m29.010019322s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139054352 (567h34m29.010137407s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139338372 (567h34m29.010421417s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139348541 (567h34m29.010431606s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139373309 (567h34m29.010456354s) [seg_pos={0 0}, upload=497603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:03:29.139477447 (567h34m29.010560492s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:03:29.139484731 (567h34m29.010567786s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:03:29.433346344 (567h34m29.304429399s) [upload=497603]: Done uploading segments... Dec 29 07:03:29.433369609 (567h34m29.304452654s) [upload=497603]: Upload finished. err: <nil> Duration: 5.707471888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 25.912504868s trace: https://jaeger.dev.storj.io/trace/5afe9fdd46a8e54f FAIL (2025-12-29 07:03:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 122dviPhVkb4TfDr3dSuQ5uXYr7mqYZGyawsfaiDa5CPtyEFi5D, Piece ID: DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ) context deadline exceeded; piecestore: (Node ID: 12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, Piece ID: AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA) context deadline exceeded ======LOGS====== 4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.470770769 (567h29m44.341853814s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471234462 (567h29m44.342317497s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471653239 (567h29m44.342736274s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472101613 (567h29m44.343184648s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472582339 (567h29m44.343665374s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473069336 (567h29m44.344152371s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473596440 (567h29m44.344679485s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.474789441 (567h29m44.345872476s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.476765935 (567h29m44.34784897s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.480795248 (567h29m44.351878293s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.484071686 (567h29m44.355154731s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.485106818 (567h29m44.356189863s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.488197412 (567h29m44.359280457s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.489260917 (567h29m44.360343962s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:45.821734925 (567h29m45.69281797s) [upload=497538]: Next returned nil segment Dec 29 06:58:45.821749823 (567h29m45.692832868s) [upload=497538]: Waiting for error group managing segments... Dec 29 06:58:46.321656276 (567h29m46.192739321s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.322482269 (567h29m46.193565304s) [noise=true , address=145.79.227.7:4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.333140875 (567h29m46.20422391s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.341854527 (567h29m46.212937562s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.344155919 (567h29m46.215238954s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.347603513 (567h29m46.218686559s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.372664974 (567h29m46.243748009s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.447122322 (567h29m46.318205357s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.502282514 (567h29m46.373365549s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.791207519 (567h29m46.662290564s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.234402275 (567h29m47.10548531s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.252070127 (567h29m47.123153172s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.319572090 (567h29m47.190655135s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383438812 (567h29m47.254521847s) [noise=true , address=77.90.31.17:4782 , piece_id=AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383457116 (567h29m47.254540161s) [seg_pos={0 0}, upload=497538]: Segment reached optimal threshold of 14 pieces. Dec 29 06:58:47.383670413 (567h29m47.254753448s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383680091 (567h29m47.254763126s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383696432 (567h29m47.254779467s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection Dec 29 06:58:47.383704077 (567h29m47.254787112s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383780052 (567h29m47.254863087s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection Dec 29 06:58:47.383784981 (567h29m47.254868016s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383932652 (567h29m47.255015687s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383940787 (567h29m47.255023832s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.384012954 (567h29m47.255095989s) [seg_pos={0 0}, upload=497538]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:58:47.384097746 (567h29m47.255180781s) [upload=497538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:58:47.384102636 (567h29m47.255185671s) [upload=497538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:58:47.684099632 (567h29m47.555182677s) [upload=497538]: Done uploading segments... Dec 29 06:58:47.684123357 (567h29m47.555206412s) [upload=497538]: Upload finished. err: <nil> Duration: 25.912518985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X..................................X.......X.......] duration: 4.7544293s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:03:56 UTC) orbiter warning: context deadline exceeded Duration: 4.754464008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..X] duration: 30.002841053s trace: https://jaeger.dev.storj.io/trace/5dc50d5607afcb75 FAIL (2025-12-29 07:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ) context deadline exceeded ======LOGS====== Dec 29 07:03:26.605630546 (567h34m26.476713581s) [upload=497603]: Starting upload Dec 29 07:03:26.605706462 (567h34m26.476789507s) [upload=497603]: Done starting upload Dec 29 07:03:26.605876175 (567h34m26.47695922s) [upload=497603]: Uploading segments... Dec 29 07:03:26.606091605 (567h34m26.47717465s) [upload=497603]: Got next segment. Inline: false Dec 29 07:03:26.606171276 (567h34m26.477254331s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:03:26.606179582 (567h34m26.477262637s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:03:26.850554898 (567h34m26.721637943s) [seg_pos={0 0}, upload=497603]: Begin upload segment... Dec 29 07:03:26.850617657 (567h34m26.721700702s) [seg_pos={0 0}, upload=497603]: Done begin upload segment. Dec 29 07:03:26.850662312 (567h34m26.721745347s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.850725072 (567h34m26.721808117s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851302050 (567h34m26.722385085s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851635215 (567h34m26.72271825s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851743330 (567h34m26.722826376s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.851962297 (567h34m26.723045332s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852356187 (567h34m26.723439232s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852674503 (567h34m26.723757538s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.852692187 (567h34m26.723775232s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853005614 (567h34m26.724088649s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853047333 (567h34m26.724130368s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853400706 (567h34m26.724483751s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853416126 (567h34m26.724499341s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.853729032 (567h34m26.724812067s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854159972 (567h34m26.725243007s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:26.854607775 (567h34m26.72569081s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Uploading piece... Dec 29 07:03:28.556562875 (567h34m28.42764592s) [upload=497603]: Next returned nil segment Dec 29 07:03:28.556581762 (567h34m28.427664797s) [upload=497603]: Waiting for error group managing segments... Dec 29 07:03:28.836878777 (567h34m28.707961822s) [noise=true , address=145.79.227.2:4113 , piece_id=7TGFQTOFEOGVBF54ABWQJ7UAVI6NRWJRHZ2DNTLYAGLZUJYRPDIA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.840256138 (567h34m28.711339203s) [noise=true , address=145.79.227.15:4874 , piece_id=IXNQ742PLDIZOO6MLQVY6YZZGST737XM4CTM4JNQS6ZF5EMPMPFA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.846293083 (567h34m28.717376118s) [noise=true , address=145.79.227.17:4972 , piece_id=MATB7U4PBXZAVDHJIFLFRHQ5YGMUWSQN7U74VR3I5BHGVVAFYJDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.849048189 (567h34m28.720131224s) [noise=true , address=145.79.227.4:4185 , piece_id=3SAW3TFFCN23TDE7CXTMO2KDYUNSUSTBSTFG6K6OCIE7CSKBOTXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859491585 (567h34m28.73057463s) [noise=true , address=145.79.227.19:4078 , piece_id=ED7RAKKDE676LKHU5VKSLPZV6R7K3EU6RJ3BOYEN4EEB4W3GNDBQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.859857912 (567h34m28.730940947s) [noise=true , address=145.79.227.46:4603 , piece_id=WS7NCJBVMQ2XGY5QYCYXI7UFUK6QAWZC7FDKXZY4GEBIMQA4TEDA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.866828044 (567h34m28.737911089s) [noise=true , address=145.79.227.47:4267 , piece_id=J42ACAPR6ZNOVBU6CI6T3KDQY7EYJ5DJIXDHUNTPJ3U7S6SH6JLA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867339358 (567h34m28.738422403s) [noise=true , address=145.79.227.16:4909 , piece_id=ZNTCVEQUISFPMW3VHWDGFGKMEEKKHRGSJKA3WWZLZI5QGSOSNBNA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.867874507 (567h34m28.738957543s) [noise=true , address=145.79.227.21:4473 , piece_id=WNH4HJOJR6SLCANUMFEEWRRGOZNNI3LUUTW6B2VGUA5XKULVDAIQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:28.874538456 (567h34m28.745621501s) [noise=true , address=145.79.227.38:4979 , piece_id=2BYZHSY7I4GI5S5W2WRWQDTZBXAHTAUISOTDZSVAH7IMSD6PNUKA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.045623807 (567h34m28.916706852s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDVOJEEYFK3UFZ44YO4HJAPVGPWKVVTEMASKRN4KXGS74IK3SJXA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.100033961 (567h34m28.971116996s) [noise=true , address=45.91.116.13:4155 , piece_id=IUPPVLDAPK6UDQPCBQC2PPQFHEWAYGU4NU5IZHFSP2PT5I3KJPNQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138577784 (567h34m29.009660819s) [noise=true , address=45.91.116.12:4227 , piece_id=4LTBPTREYNHI27P72PVJURHK7TRGQL7LZM7FA6KBE6234KSBBBJA, seg_pos={0 0}, upload=497603]: Done uploading piece. err: <nil> Dec 29 07:03:29.138610136 (567h34m29.009693181s) [seg_pos={0 0}, upload=497603]: Segment reached optimal threshold of 13 pieces. Dec 29 07:03:29.138860522 (567h34m29.009943587s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.138870822 (567h34m29.009953878s) [noise=true , address=45.91.116.11:4118 , piece_id=ZXMSKFQOUFRMPKWSXFCOMXXTIXS2IECJ6SOBSPODZTEKAQLPGJ3A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.138936277 (567h34m29.010019322s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139054352 (567h34m29.010137407s) [noise=true , address=45.91.116.9:4264 , piece_id=IRM5CXJOF4GYXDKTONDYKIMO4YCKXBYVPPGOAWHSOMFGCUMLAZ6A, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139338372 (567h34m29.010421417s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 29 07:03:29.139348541 (567h34m29.010431606s) [noise=true , address=77.90.31.13:4629 , piece_id=DLD6Y7CJLGTGYQPH2J36DC2ZFZXO5XEVXNMQPI4WXKF32F3PQDXQ, seg_pos={0 0}, upload=497603]: Long tail cancellation due to optimal threshold Dec 29 07:03:29.139373309 (567h34m29.010456354s) [seg_pos={0 0}, upload=497603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:03:29.139477447 (567h34m29.010560492s) [upload=497603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:03:29.139484731 (567h34m29.010567786s) [upload=497603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:03:29.433346344 (567h34m29.304429399s) [upload=497603]: Done uploading segments... Dec 29 07:03:29.433369609 (567h34m29.304452654s) [upload=497603]: Upload finished. err: <nil> Duration: 30.002866511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X..................................X.......X......] duration: 3.400432651s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:03:34 UTC) orbiter warning: context deadline exceeded Duration: 3.400478519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 31.166767241s trace: https://jaeger.dev.storj.io/trace/5afe9fdd46a8e54f FAIL (2025-12-29 07:03:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 122dviPhVkb4TfDr3dSuQ5uXYr7mqYZGyawsfaiDa5CPtyEFi5D, Piece ID: DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ) context deadline exceeded; piecestore: (Node ID: 12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, Piece ID: AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA) context deadline exceeded ======LOGS====== 4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.470770769 (567h29m44.341853814s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471234462 (567h29m44.342317497s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471653239 (567h29m44.342736274s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472101613 (567h29m44.343184648s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472582339 (567h29m44.343665374s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473069336 (567h29m44.344152371s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473596440 (567h29m44.344679485s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.474789441 (567h29m44.345872476s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.476765935 (567h29m44.34784897s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.480795248 (567h29m44.351878293s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.484071686 (567h29m44.355154731s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.485106818 (567h29m44.356189863s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.488197412 (567h29m44.359280457s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.489260917 (567h29m44.360343962s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:45.821734925 (567h29m45.69281797s) [upload=497538]: Next returned nil segment Dec 29 06:58:45.821749823 (567h29m45.692832868s) [upload=497538]: Waiting for error group managing segments... Dec 29 06:58:46.321656276 (567h29m46.192739321s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.322482269 (567h29m46.193565304s) [noise=true , address=145.79.227.7:4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.333140875 (567h29m46.20422391s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.341854527 (567h29m46.212937562s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.344155919 (567h29m46.215238954s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.347603513 (567h29m46.218686559s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.372664974 (567h29m46.243748009s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.447122322 (567h29m46.318205357s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.502282514 (567h29m46.373365549s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.791207519 (567h29m46.662290564s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.234402275 (567h29m47.10548531s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.252070127 (567h29m47.123153172s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.319572090 (567h29m47.190655135s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383438812 (567h29m47.254521847s) [noise=true , address=77.90.31.17:4782 , piece_id=AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383457116 (567h29m47.254540161s) [seg_pos={0 0}, upload=497538]: Segment reached optimal threshold of 14 pieces. Dec 29 06:58:47.383670413 (567h29m47.254753448s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383680091 (567h29m47.254763126s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383696432 (567h29m47.254779467s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection Dec 29 06:58:47.383704077 (567h29m47.254787112s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383780052 (567h29m47.254863087s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection Dec 29 06:58:47.383784981 (567h29m47.254868016s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383932652 (567h29m47.255015687s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383940787 (567h29m47.255023832s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.384012954 (567h29m47.255095989s) [seg_pos={0 0}, upload=497538]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:58:47.384097746 (567h29m47.255180781s) [upload=497538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:58:47.384102636 (567h29m47.255185671s) [upload=497538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:58:47.684099632 (567h29m47.555182677s) [upload=497538]: Done uploading segments... Dec 29 06:58:47.684123357 (567h29m47.555206412s) [upload=497538]: Upload finished. err: <nil> Duration: 31.166796908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X..] duration: 5.676137891s trace: https://jaeger.dev.storj.io/trace/39063344efa1fe43 FAIL (2025-12-29 07:03:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: 5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA) context deadline exceeded ======LOGS====== Dec 29 07:01:21.627607260 (567h32m21.498690305s) [upload=497577]: Starting upload Dec 29 07:01:21.627669218 (567h32m21.498752254s) [upload=497577]: Done starting upload Dec 29 07:01:21.627918313 (567h32m21.499001358s) [upload=497577]: Uploading segments... Dec 29 07:01:21.627966033 (567h32m21.499049068s) [upload=497577]: Got next segment. Inline: false Dec 29 07:01:21.628033001 (567h32m21.499116046s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:01:21.628041277 (567h32m21.499124322s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:01:21.873866993 (567h32m21.744950048s) [seg_pos={0 0}, upload=497577]: Begin upload segment... Dec 29 07:01:21.873940152 (567h32m21.745023197s) [seg_pos={0 0}, upload=497577]: Done begin upload segment. Dec 29 07:01:21.873952355 (567h32m21.74503539s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.873981040 (567h32m21.745064085s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874098583 (567h32m21.745181619s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874718755 (567h32m21.7458018s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875011863 (567h32m21.746094908s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875353905 (567h32m21.74643694s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875660157 (567h32m21.746743202s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876010384 (567h32m21.747093419s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876309444 (567h32m21.747392479s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876638921 (567h32m21.747721956s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877003816 (567h32m21.748086851s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877426761 (567h32m21.748509796s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877741881 (567h32m21.748824926s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878092970 (567h32m21.749176015s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878103781 (567h32m21.749186826s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878139289 (567h32m21.749222334s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:23.577717824 (567h32m23.448800869s) [upload=497577]: Next returned nil segment Dec 29 07:01:23.577770464 (567h32m23.448853509s) [upload=497577]: Waiting for error group managing segments... Dec 29 07:01:23.849612580 (567h32m23.720695615s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.865977682 (567h32m23.737060717s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.870108908 (567h32m23.741191943s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.872800162 (567h32m23.743883197s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873364106 (567h32m23.744447151s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873718091 (567h32m23.744801136s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.875620563 (567h32m23.746703608s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.878594156 (567h32m23.749677191s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.879766487 (567h32m23.750849532s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.882545519 (567h32m23.753628564s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.921023717 (567h32m23.792106852s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.073714381 (567h32m23.944797436s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119411736 (567h32m23.990494781s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119515644 (567h32m23.990598679s) [seg_pos={0 0}, upload=497577]: Segment reached optimal threshold of 13 pieces. Dec 29 07:01:24.119891138 (567h32m23.990974213s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119901248 (567h32m23.990984283s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.119934782 (567h32m23.991017827s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119970009 (567h32m23.991053054s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120247357 (567h32m23.991330402s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.120264349 (567h32m23.991347414s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120313652 (567h32m23.991396687s) [seg_pos={0 0}, upload=497577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:01:24.120425005 (567h32m23.99150804s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:01:24.120431407 (567h32m23.991514452s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:01:24.417375912 (567h32m24.288458967s) [upload=497577]: Done uploading segments... Dec 29 07:01:24.417393174 (567h32m24.288476219s) [upload=497577]: Upload finished. err: <nil> Duration: 5.676169122s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X..................................X.......X.....] duration: 4.393964813s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:02:57 UTC) orbiter warning: context deadline exceeded Duration: 4.393989881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 45.381130245s trace: https://jaeger.dev.storj.io/trace/5afe9fdd46a8e54f FAIL (2025-12-29 07:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 122dviPhVkb4TfDr3dSuQ5uXYr7mqYZGyawsfaiDa5CPtyEFi5D, Piece ID: DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ) context deadline exceeded; piecestore: (Node ID: 12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, Piece ID: AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA) context deadline exceeded ======LOGS====== 4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.470770769 (567h29m44.341853814s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471234462 (567h29m44.342317497s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.471653239 (567h29m44.342736274s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472101613 (567h29m44.343184648s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.472582339 (567h29m44.343665374s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473069336 (567h29m44.344152371s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.473596440 (567h29m44.344679485s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.474789441 (567h29m44.345872476s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.476765935 (567h29m44.34784897s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.480795248 (567h29m44.351878293s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.484071686 (567h29m44.355154731s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.485106818 (567h29m44.356189863s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.488197412 (567h29m44.359280457s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:44.489260917 (567h29m44.360343962s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Uploading piece... Dec 29 06:58:45.821734925 (567h29m45.69281797s) [upload=497538]: Next returned nil segment Dec 29 06:58:45.821749823 (567h29m45.692832868s) [upload=497538]: Waiting for error group managing segments... Dec 29 06:58:46.321656276 (567h29m46.192739321s) [noise=true , address=145.79.227.2:4139 , piece_id=327T2EJ5XH4UNXW44XNOILMEZPB6LI3JQPZWZVB52KSPJJCB64PA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.322482269 (567h29m46.193565304s) [noise=true , address=145.79.227.7:4303 , piece_id=IIVVMOLPPOIYXGRPWL3CEEYWNCID7CCSWYZO4H5PYPJYNCJRMAWQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.333140875 (567h29m46.20422391s) [noise=true , address=145.79.227.10:4425 , piece_id=HS43T34MAJR25XICXZXYJCRSCQTK32AWQK6SFBYNF4HHQN46WI3A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.341854527 (567h29m46.212937562s) [noise=true , address=145.79.227.9:4392 , piece_id=R3VHQVF3PT2426YBY4XXZXYISAG4WAMNQV4JAZTCMOTLVHBC4WZA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.344155919 (567h29m46.215238954s) [noise=true , address=145.79.227.25:4637 , piece_id=NLSMZE5VKURCEH5WFRSB5LEQJFVEKCLLQIJHTTZNR3WMS6YFVMHA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.347603513 (567h29m46.218686559s) [noise=true , address=145.79.227.46:4624 , piece_id=V3ZEHZ4XTQB3WEMWYYKXQSSKW2QB6HES2E3A6O7HPLFVYG7L4SUA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.372664974 (567h29m46.243748009s) [noise=true , address=145.79.227.43:4160 , piece_id=JWGJNIOMZH4QE346BW4FL7HBOW2WOWEODJN3GIJ6MWSU7QFV4HRQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.447122322 (567h29m46.318205357s) [noise=true , address=64.49.13.140:4516 , piece_id=DZPI2XXVMCH5P6GCMB5EL66CCUOFH3XEBK6YW66NTLOJYZC7WONQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.502282514 (567h29m46.373365549s) [noise=true , address=77.90.31.14:4665 , piece_id=FKIHGVDI3AGWL2L46OQQTUTZR3CIN27D4TNKQBDHC44F6GDMU4OA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:46.791207519 (567h29m46.662290564s) [noise=true , address=45.91.116.14:4211 , piece_id=SIMJOIJKI25AXRZ5UO343EESV4DHRV5YJJJQUTQGTTDJEEM3HVPA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.234402275 (567h29m47.10548531s) [noise=true , address=145.79.227.44:4197 , piece_id=FKL2VPQCODFPVAH3YGRW2DQYLCQ77RMQBHGGRYDZ3MFPKC5XZ6XQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.252070127 (567h29m47.123153172s) [noise=true , address=145.79.227.30:4791 , piece_id=C4PBTLWUXPVTP4TGJKRUTKMH5XE7ZJKSAVSOOIUPQZQHCIYUGJIQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.319572090 (567h29m47.190655135s) [noise=true , address=64.49.13.26:4946 , piece_id=7JHXHYKKTI7W3KDCEVJQHESVAMH4OHI46ISSXLDKO2V44BB34QBA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383438812 (567h29m47.254521847s) [noise=true , address=77.90.31.17:4782 , piece_id=AV5CISNU43JLWGQJOWQFFXTBZPV2Z5JYCDL3HVK7OY6DYROF5AYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: <nil> Dec 29 06:58:47.383457116 (567h29m47.254540161s) [seg_pos={0 0}, upload=497538]: Segment reached optimal threshold of 14 pieces. Dec 29 06:58:47.383670413 (567h29m47.254753448s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383680091 (567h29m47.254763126s) [noise=true , address=193.149.164.143:4703 , piece_id=QXV42K7GWSXXATUDPIFMAXB6UID66EZJHABF5F3LQO4OYHA7RHYA, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383696432 (567h29m47.254779467s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58072->64.49.13.169:4715: use of closed network connection Dec 29 06:58:47.383704077 (567h29m47.254787112s) [noise=true , address=64.49.13.169:4715 , piece_id=3YMXYSIRGDHTYGO475OFE54LYTPLCFBNTW2Y6KQQRQXHQU4KJ64A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383780052 (567h29m47.254863087s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57184->77.90.31.18:4826: use of closed network connection Dec 29 06:58:47.383784981 (567h29m47.254868016s) [noise=true , address=77.90.31.18:4826 , piece_id=XTLKOYTIRIBWR7IMR3NZFXAV5VNGQ4XU2NPISUGM3NV5LVJTVNXQ, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.383932652 (567h29m47.255015687s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 29 06:58:47.383940787 (567h29m47.255023832s) [noise=true , address=145.79.227.34:4916 , piece_id=4LEMJLDIIKHO5TCO2WVVMTMZKHQ6XFGE4JT7543D6IV5WGQIV37A, seg_pos={0 0}, upload=497538]: Long tail cancellation due to optimal threshold Dec 29 06:58:47.384012954 (567h29m47.255095989s) [seg_pos={0 0}, upload=497538]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:58:47.384097746 (567h29m47.255180781s) [upload=497538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:58:47.384102636 (567h29m47.255185671s) [upload=497538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:58:47.684099632 (567h29m47.555182677s) [upload=497538]: Done uploading segments... Dec 29 06:58:47.684123357 (567h29m47.555206412s) [upload=497538]: Upload finished. err: <nil> Duration: 45.38115874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X.] duration: 6.120646581s trace: https://jaeger.dev.storj.io/trace/39063344efa1fe43 FAIL (2025-12-29 07:02:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: 5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA) context deadline exceeded ======LOGS====== Dec 29 07:01:21.627607260 (567h32m21.498690305s) [upload=497577]: Starting upload Dec 29 07:01:21.627669218 (567h32m21.498752254s) [upload=497577]: Done starting upload Dec 29 07:01:21.627918313 (567h32m21.499001358s) [upload=497577]: Uploading segments... Dec 29 07:01:21.627966033 (567h32m21.499049068s) [upload=497577]: Got next segment. Inline: false Dec 29 07:01:21.628033001 (567h32m21.499116046s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:01:21.628041277 (567h32m21.499124322s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:01:21.873866993 (567h32m21.744950048s) [seg_pos={0 0}, upload=497577]: Begin upload segment... Dec 29 07:01:21.873940152 (567h32m21.745023197s) [seg_pos={0 0}, upload=497577]: Done begin upload segment. Dec 29 07:01:21.873952355 (567h32m21.74503539s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.873981040 (567h32m21.745064085s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874098583 (567h32m21.745181619s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874718755 (567h32m21.7458018s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875011863 (567h32m21.746094908s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875353905 (567h32m21.74643694s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875660157 (567h32m21.746743202s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876010384 (567h32m21.747093419s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876309444 (567h32m21.747392479s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876638921 (567h32m21.747721956s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877003816 (567h32m21.748086851s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877426761 (567h32m21.748509796s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877741881 (567h32m21.748824926s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878092970 (567h32m21.749176015s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878103781 (567h32m21.749186826s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878139289 (567h32m21.749222334s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:23.577717824 (567h32m23.448800869s) [upload=497577]: Next returned nil segment Dec 29 07:01:23.577770464 (567h32m23.448853509s) [upload=497577]: Waiting for error group managing segments... Dec 29 07:01:23.849612580 (567h32m23.720695615s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.865977682 (567h32m23.737060717s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.870108908 (567h32m23.741191943s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.872800162 (567h32m23.743883197s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873364106 (567h32m23.744447151s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873718091 (567h32m23.744801136s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.875620563 (567h32m23.746703608s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.878594156 (567h32m23.749677191s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.879766487 (567h32m23.750849532s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.882545519 (567h32m23.753628564s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.921023717 (567h32m23.792106852s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.073714381 (567h32m23.944797436s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119411736 (567h32m23.990494781s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119515644 (567h32m23.990598679s) [seg_pos={0 0}, upload=497577]: Segment reached optimal threshold of 13 pieces. Dec 29 07:01:24.119891138 (567h32m23.990974213s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119901248 (567h32m23.990984283s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.119934782 (567h32m23.991017827s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119970009 (567h32m23.991053054s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120247357 (567h32m23.991330402s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.120264349 (567h32m23.991347414s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120313652 (567h32m23.991396687s) [seg_pos={0 0}, upload=497577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:01:24.120425005 (567h32m23.99150804s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:01:24.120431407 (567h32m23.991514452s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:01:24.417375912 (567h32m24.288458967s) [upload=497577]: Done uploading segments... Dec 29 07:01:24.417393174 (567h32m24.288476219s) [upload=497577]: Upload finished. err: <nil> Duration: 6.120677069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X..................................X.......X....] duration: 4.439450515s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:02:09 UTC) orbiter warning: context deadline exceeded Duration: 4.439475673s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.....X.X.X.X..X.X..X.....XX..X.X.X.X.XXXXX.X....XXX...X.XX.X] duration: 30.004011721s trace: https://jaeger.dev.storj.io/trace/39063344efa1fe43 FAIL (2025-12-29 07:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: 5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA) context deadline exceeded ======LOGS====== Dec 29 07:01:21.627607260 (567h32m21.498690305s) [upload=497577]: Starting upload Dec 29 07:01:21.627669218 (567h32m21.498752254s) [upload=497577]: Done starting upload Dec 29 07:01:21.627918313 (567h32m21.499001358s) [upload=497577]: Uploading segments... Dec 29 07:01:21.627966033 (567h32m21.499049068s) [upload=497577]: Got next segment. Inline: false Dec 29 07:01:21.628033001 (567h32m21.499116046s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:01:21.628041277 (567h32m21.499124322s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:01:21.873866993 (567h32m21.744950048s) [seg_pos={0 0}, upload=497577]: Begin upload segment... Dec 29 07:01:21.873940152 (567h32m21.745023197s) [seg_pos={0 0}, upload=497577]: Done begin upload segment. Dec 29 07:01:21.873952355 (567h32m21.74503539s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.873981040 (567h32m21.745064085s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874098583 (567h32m21.745181619s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.874718755 (567h32m21.7458018s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875011863 (567h32m21.746094908s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875353905 (567h32m21.74643694s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.875660157 (567h32m21.746743202s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876010384 (567h32m21.747093419s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876309444 (567h32m21.747392479s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.876638921 (567h32m21.747721956s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877003816 (567h32m21.748086851s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877426761 (567h32m21.748509796s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.877741881 (567h32m21.748824926s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878092970 (567h32m21.749176015s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878103781 (567h32m21.749186826s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:21.878139289 (567h32m21.749222334s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Uploading piece... Dec 29 07:01:23.577717824 (567h32m23.448800869s) [upload=497577]: Next returned nil segment Dec 29 07:01:23.577770464 (567h32m23.448853509s) [upload=497577]: Waiting for error group managing segments... Dec 29 07:01:23.849612580 (567h32m23.720695615s) [noise=true , address=145.79.227.6:4259 , piece_id=DF3JKYQ7ZSELBL5VNK4AKPTIZV2GYLV4CNRGMHFW7GLKCHLTKODA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.865977682 (567h32m23.737060717s) [noise=true , address=145.79.227.26:4654 , piece_id=YW42O7YJVV76ZPJ4GO7UUMV2PSUFC2RO25HFTDDEMR2SDL6UEZZQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.870108908 (567h32m23.741191943s) [noise=true , address=145.79.227.20:4437 , piece_id=MWLHGO5AXTDGLAZ26ZOJ4ZZO4A4ZP7IUSDY5WOHQ4GSHMCCTPQYQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.872800162 (567h32m23.743883197s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZVZR5BLLOMCQQSZVQUXM4WQ4QX57WL7S5BCC5KNN7RMUZOILIQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873364106 (567h32m23.744447151s) [noise=true , address=145.79.227.13:4802 , piece_id=KPLMFWUJ5ZJDQCRV5R6K43QYRWHCVGVHE5YEMN25MUGHNYM2CBGQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.873718091 (567h32m23.744801136s) [noise=true , address=145.79.227.50:4375 , piece_id=VMQIGWF5IQVBFVBWVIXXPWCRVL6TRC4U7UDIQOVER2AUBYST3H5Q, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.875620563 (567h32m23.746703608s) [noise=true , address=145.79.227.19:4077 , piece_id=IDDEWT3MHNWIGGT3KFBQX6NKSVLFN4PCB4NZDV6FMY5KA36MOZMA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.878594156 (567h32m23.749677191s) [noise=true , address=145.79.227.14:4840 , piece_id=PLSIWZ4SDNK4LLFXZF6EGJTOHJD64QN2BNIIHP6GRNW3QK76VUVQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.879766487 (567h32m23.750849532s) [noise=true , address=145.79.227.46:4601 , piece_id=5RC6LD2GWDA7WLZNUNNBPLBM47STRZRIMNNG7M7G2BITCJJVHMPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.882545519 (567h32m23.753628564s) [noise=true , address=145.79.227.41:4085 , piece_id=TTQGSESR4BH4VMWIHB2X7KOF3GSJ35FFVZ7L4HO5FRWVN73P35HQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:23.921023717 (567h32m23.792106852s) [noise=true , address=145.79.227.16:4910 , piece_id=IBWE25PNL5AKJT5ZNDOMTGIUUPWQHGQPBVSQBZ5QMQGNVJL2ZJPA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.073714381 (567h32m23.944797436s) [noise=true , address=77.90.31.10:4520 , piece_id=5DRVEPSSHV5HPZP2H3JEJ2Q7TXWEDKSN7TH2ZWHUG2QPMSBU4ERA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119411736 (567h32m23.990494781s) [noise=true , address=45.91.116.10:4084 , piece_id=DXSPNPYWJWO7LGUE4RIWFP7XAQQAASRWHFPU3IEBIXPI3EYJKXSA, seg_pos={0 0}, upload=497577]: Done uploading piece. err: <nil> Dec 29 07:01:24.119515644 (567h32m23.990598679s) [seg_pos={0 0}, upload=497577]: Segment reached optimal threshold of 13 pieces. Dec 29 07:01:24.119891138 (567h32m23.990974213s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119901248 (567h32m23.990984283s) [noise=true , address=77.90.31.18:4808 , piece_id=MI3A2DQV3NQQBAEJWC2GMNU7E5PAYATR6WVZ2DQ7V2LBFYGQBDNQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.119934782 (567h32m23.991017827s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.119970009 (567h32m23.991053054s) [noise=true , address=45.91.116.4:4937 , piece_id=KV65NTNWZ32OQ3CHZBE5DOW3UELOHR5ZJW6OO26KVSUFGHEF352A, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120247357 (567h32m23.991330402s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 07:01:24.120264349 (567h32m23.991347414s) [noise=true , address=45.91.116.6:4973 , piece_id=5FSBYS36YP7FA7VSWQ5LZCCWMN57HNDOBWXRJV5CALT3G6ZNOMAQ, seg_pos={0 0}, upload=497577]: Long tail cancellation due to optimal threshold Dec 29 07:01:24.120313652 (567h32m23.991396687s) [seg_pos={0 0}, upload=497577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:01:24.120425005 (567h32m23.99150804s) [upload=497577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:01:24.120431407 (567h32m23.991514452s) [upload=497577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:01:24.417375912 (567h32m24.288458967s) [upload=497577]: Done uploading segments... Dec 29 07:01:24.417393174 (567h32m24.288476219s) [upload=497577]: Upload finished. err: <nil> Duration: 30.004068508s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X..................................X.......X...] duration: 3.790956866s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:01:33 UTC) orbiter warning: context deadline exceeded Duration: 3.790988024s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)